Question

Michael Bolton posted what I think is quite a helpful list of heuristics here: http://www.developsense.com/2009/09/when-do-we-stop-test.html

They include:

1. The Time's Up! Heuristic. 
2. The Piñata Heuristic.
3. The Dead Horse Heuristic. 
4. The Mission Accomplished Heuristic. 
5. The Mission Revoked Heuristic. 
6. The I Feel Stuck! Heuristic.
7. The Pause That Refreshes Heuristic. 
8. The Flatline Heuristic. 
9. The Customary Conclusion Heuristic. 
10. No more interesting questions. 
11. The Avoidance/Indifference Heuristic.

Anyone else have any good guidelines? Thoughts to add?

Justin

Answer 1

Actually, I think these are two separate questions.

The only time I have seen testing really stop is only when a commercial product is permanantly retired, and there is no additional support. I am sure there is "ship it and forget it type software" but, n products with a longer shelf life and/or multiple versions or iterations testing continues well after the product ships and throughout the maintenance and support period.

Of course, I suspect an external consultant's perspective of when testing stops is quite different. I am not suggesting this as a bad thing, I think it is just a different perspective.

How much testing is enough is an even harder question to answer. One could easily say when we aren't finding any more bugs, or when our tests aren't providing any further information of significant value. But, honestly I don't think we have a really good handle on how or when to make that call. Your a stats guy Justin, so I think you know that today many test teams don't have a really good hand on the sample size of either valid or invalid test data being exercised in our tests. I would suspect the answer to how much testing is enough has more to do with the point the test team exhausts their ability to design additional tests capable of revealing new information.

But, I will also add these questions are without context and my answers are generalities based on my experience with complex commercial software. I am sure there are one-off solutions where someone could say "in such and such case 'x' amount of testing is sufficient" or "on our team the testers are the gate keepers."

Answer 2

I use to paraphrase Paul Valery, who said "A poem is never finished, only abandoned".
Same happens with tests - one would go forever unless context stopped us.

The list by Michael is a great summary, and the heuristics are flexible enough that any other reason can be matched to one of them.
For example, the "No more money" heuristics is part of the "Mission Revoked Heuristic" in his list. But it may useful to think of it separately, so add it to the list:

• The No More Money Heuristic (goes with "5."?)
• The People Can't Cope with Another Bug Heuristic (can be included in point "11.")
• The Customer is Satisfied and Says We can Stop (can be included in "2.", "3." or "10.")

Answer 3

In reply to BJ's remark on testing stopping, note that these heuristics might apply to stopping a single test, stopping a test cycle, or stopping testing as the product approaches shipment, in addition to the notion of testing stopping when a project is sunsetted (which is typically a case of mission abandoned, by they way).

In reply to BJ's remark on external consultants, note I lived this stuff while a program manager for a commercial product which, at the time, was at the top of PC Magazine's best-seller lists. Above Windows.

For a consultant, part of the job is to recognize how many different of people decide to stop many different kinds of tests, and for many different kinds of contexts. For example, people may continue to test software even after it's retired. On a project a couple of years back, the product that we were working on was replacing a product that was 35 years old. (Yes, 35.) The only artifact of the old development effort was the running software itself; we had no documentation and no source code for it. To determine consistency with the new application, we had to test the old application.

Note also that there's a twelfth heuristic, contributed by Cem Kaner: The Mission Rejected heuristic, in which testers decide not to test because of ethical concerns; because they're being paid too little; or paid too much, presumably to do something trivial.

---Michael B.