OK, first things first here:
I know this is, in the words of Michael Bolton, a "unicorn question" and also that the challenge in providing a meaningful answer to this question is compounded because, e.g., the definitions of "what is a 'Systems Test'?" will vary significantly between companies, the cost of fixing defects in production will vary tremendously based on whether the application is web-based or not, etc. Even so, I am interested in: what rules of thumb do you use?
My rationale for asking the question: it is a practical question that managers should think through in order to make useful decisions about, e.g., "how much of our testing effort (and budget) should be focused on finding defects earlier as opposed to later?" "When should we stop testing?" etc.
Quick, high-level "gut-feel" rules of thumb would be interesting as data points.
Answers that dodge providing specific figures but instead say, "I've addressed the rationale behind the question a few times in different contexts and have found the following ways of thinking about the topics are useful..." would be welcome.
Ratios (as opposed to dollar figures) would be totally fine.
Different definitions of different phases of testing would be totally fine.
Answers citing actual case studies and/or methods used to gather the data would be especially interesting.
Answers referring to factors that impact the costs and/or ratios would be superb and "above and beyond" the call of duty. For a great example of how this was done, please see how Cem Kaner et al answered another unicorn question (scroll down to see Michael Bolton's answer).
Thanks in advance.
Justin
