Question

Those familiar with the business of New Media will be aware project lifespans are short, deadlines never seem to match time required, specifications are often lacking (to the point that only a few concepts may be provided), and projects have to be constantly juggled for time on a daily basis.

Given the lack of repetition it is hard to justify spending the time on automation and automation will result in a narrower coverage of potential usage paths and potential oversight of issues if a set of tests are created around a single instance where coverage may normally be achieved by spreading single user paths / scenarios across the spectrum of what's available (with pairing and matrix styled ideas, but not necessarily documented).

Obviously organisation skills play a key role in management of such QA, but I am interested in others take / perspective how they believe they would approach managing the QA of such work. So I would like to hear others approaches or ideas with dealing with QA in such an environment?

Answer 1

Hi,

I've been in this same situation a few times and I actually find that a certain level of basic automation smoke tests is very valuable. As for a proper testing strategy; well, not really worth it unless the product is going to have some form of longevity.

The ROI is what matters and sometimes some awesome exploratory testing, like Alan mentioned, is all that is needed.

With the automation I had some high level guidance tests which I wrote as I explored meaning I could cover the same ground again if I needed to. I also managed my exploratory testing through simple .txt files which got checked in to source control.

Reporting was done through a low tech dashboard which gave a basic overview. Detailed metrics aren't really required on an agile project with the emphasis been on complete stories.

I'd be careful to assume that New Media and agile go together. New Media is an industry. Agile is used in many industries. Agile also doesn't mean lacking specification and mismatching deadlines. But it sounds like your approach is very iterative based so would suit some automation and exploratory testing. Just worth pointing out not to confuse agile testing with ad-hoc/reactionary testing.

Hope this helps.

Rob.. http://thesocialtester.posterous.com/

Answer 2

I debated giving my answer to this, but since nobody else has, I'll chime in.

You're right - in this situation, it probably doesn't make sense to automate - or at least that much. I may, however (and depending on the app), write some automation - particularly model-based tests that can double up as lightweight regression tests and stress tests (basically as a smart monkey test).

I'd focus a lot on exploratory testing. Tours as described by James Whittaker would work well. One key thing to cover is the major scenarios. Work with whoever owns the customer voice to define a set of end to end scenarios the cover the expected use, and test the heck out of those, including variations, shortcuts, etc. to ensure that at the very least, the "happy path" will always work. As time allows, branch out into other areas of the application.

Finally, don't forget that there are plenty of other tools in a testers belt beyond automation. Use tools to view memory usage, processor time, registry usage, etc. The tools from sysinternals.com are great for this.