One of the problems QA people have is deciding how much proof is enough.
Proof is expensive and time-consuming to generate. Even with the current test tools taking screen images and storing them, you may have to annotate the proofs and combine them with other items to send to various places. You may want to weed out items you don’t want and add things that are missed. Having done all of that and made a nice package, you then wonder if anyone ever looks at it. We have had clients look and some clients need it for regulatory purposes. However, a lot want to know it is there and they could look at it if they had a need but they don’t ever open up the package and look.
Of course it also gets superseded with new executions of the test case, new test runs or simply new code. Sometimes the proofs accumulate in the test tool or a directory until they overwhelm storage and get put onto backup just in case they are every needed. Eventually either they can no longer be read or someone cleans up all the old backups.
We have two major questions:
- Are the proofs necessary?
- How long should they be kept?
The answer to the first question depends on risk, and the ability to learn and use items from the past. There is no doubt that higher risk items need more proof and it needs to be kept. But the second part is even more critical, can we use the value in the proofs to inform future testing efforts in the software. Often a second review will identify other items that might have been missed either in the existing proof or in test coverage.
The answer to the second question is actually a process that should exist for all documentation. There is an absolute upper limit in terms of years after which, if the proof is that old and the testcase has not been re-executed in all that time, then it is not likely it is going to come back. Alternatively, if the testcase has been executed multiple times since the the original proof was stored and all the subsequent proofs have been retained, then we will not be going back and it is probably safe to discard the older proofs.
Either way, we do not keep everything for ever.
If you have input on the above, you might want to consider our survey.