Our latest blog will discuss the Test Run. For today’s purpose, NVP considers a Test Run to be one, single execution of a testcase. This could mean that the testcase ran to completion and the expected AND actual results were identical, or that the case the testcase did not have actual results that equalled the expected. We have stayed away from the words ‘successful’ and ‘unsuccessful’ since some may feel a testcase is only successful if it uncovers a problem and is unsuccessful if it does not.

We are interested in this statistic of test runs for a number of reasons:

  1. It helps in estimation
  2. It helps justify the time taken to test
  3. It provides a measure of code stability

Estimation

Knowing the number of Runs of a testcase helps determines how long the cycles and the whole test effort will take next time. If we know we had to run each testcase an average of 5 or 6 times before it ran to completion without raising an issue then we know how many times we may need to run it next time. Note that unsuccessful runs may include attempts that lead to fixing the testcase or relevant test data. Once we have ‘debugged’ the testcase, these runs may not recur.

Justification

If we only report the count of completed testcases with actual results equalling expected results, then each testcase might only show a single execution. This would hide a lot of work and effort and make the testers appear very unproductive. Showing that each testcase was executed 6 or 7 times before we were satisfied gives a much better idea of the effort involved.

Code Stability

If a testcase is run a dozen times and only on the last time does it run to completion with Expected Results equal to Actual Results, then we may have a concern with code stability or whether that final run was really correct. Something that fails a dozen times and then is successful is highly suspect. Maybe the conditions changed, maybe we missed something, maybe the issue was finally fixed. Whatever the case, we are not sure of the stability.

Discussion Questions

  1. Do you have defined test Runs?
  2. What is the worst case for number of times they had to be run?
  3. What is your least number of runs

Next Week: Process Improvement