Testing CountReads

Introduction

One of the best ways to test a new GUI program you've written is to ask a friend or colleague to try it out while you observe and take notes. This gives you a chance to detect usability or functional problems before you deploy your program to a larger audience.

By observing a friend or colleague interact with your software, you'll often get ideas for how to make it more user-friendly or even new features to add that would make it better. And of course, you'll likely find bugs you might have missed during your own testing process.

During class, pair up with a partner (your choice!) and work with them to try out and test their CountReads program. Test each other's programs using the following recipe. Take notes as you watch your friend use the software.

When you're done, you'll use what you learned to implement at least one improvement to your software - bug fix or usability enhancement. On Friday, you'll demonstrate the new and improved version to your partner.

Write a short report (< 2 pages) containing your notes and description of how you used your partner's observations to improve your software. Send this to your partner and to Dr. Loraine and Wei Song for full credit.

0. Get oriented.

Explain the purpose of this exercise and thank the tester for their participation. Explain your goal is to find potential errors or usability problems in the software before releasing it to larger audience.

Set a time limit before you begin - for this exercise, spend no more than 30 minutes per session. Be sure to reserve time for each person to play both roles - tester and developer.

1. Demonstrate program.

Developer demonstrates how to use the program. Explain what the program is for and how to use it. Answer any questions.

2. Tester takes over.

Developer passes keyboard and mouse to the tester.

3. Enter good input.

Tester gets started with the program. Check that it gives the correct answer when given proper input. In the case of the CountReads program, you could use samtools to confirm that it gives the correct answer.

4. Enter bad input.

How does the program react when you give it bad data? Test each text entry field:

Enter non-numerical values in start and end fields.
Enter negative integers in start and end fields.
Enter string data in start and end fields.
Enter a chromosome name that is not in the BAM File.
Choose a file of the wrong type and attempt to run a query.
Attempt to run a query without having chosen a file.
Enter start and end but not sequence.
And so on...

When given bad input, a program could respond in a variety of ways. These could include:

The program may crash.
It might fail without notifying you that it failed.
It might give a wrong answer or continue to display the answer to the last correctly entered query.
It might display an error message (e.g., open a dialog box).

How does your partner's program handle bad input? If it displayed warning messages, did the warning messages explain what you did wrong?

5. Evaluate ease of use.

In general, fewer operations (mouse clicks, typing) are better - users have to do less work to use your program.

What happens when you try to open a second BAM file in the same directory? Does the program remember the location of the previously opened file or do you have to click through lots of folders to find what you're looking for?

Programs that require user input (like CountReads) often provide reasonable defaults. For example, if the user just wants to know how many reads overlap on chromosome 1, can you run the query without also enterring the start and end position of the chromosome's first and last base? To test, enter:

Enter just chromosome.
Enter chromosome and start position only.

What happens?

6. Wrap up.

Developer thanks the tester for the help, advice, and feedback.

7. Switch roles.

Change roles and repeat steps 1-6.

8. Follow up.

Using your partner's observations as a starting point, implement at least one improvement to your program. In Friday's class, you'll demonstrate the new and improved version of the software to your partner.