The QA Paradox
Or How thorough QA can lead to lower quality ?
Indeed this is a paradox but it appeared very clearly to me after observing a change in the QA style in our team. Before this realisation I would have happily argued that thorough QA is the best thing that can happen to a project. The paradox unfolds in three parts.
Firstly it makes it harder (and more importantly longer) to finish stories. I understand that it also avoids accepting low quality work. However the QA perspective is slightly different (and this is good) from an actual user’s. So thorough QA, uncovers plenty of minor problems that do not significantly impact user experience or that would be minor production issues. This generates frivolous work that delays the development of more important stories, bug fixes or refactoring, thereby reducing the overall quality of the product.
Second, after a while the QA style modifies the developer’s attitude. When QA is thorough there is a sense of false confidence in the QA process that generates sub par unit and integration tests. This in turns leads to more bugs. Also, when QA is too picky before accepting a story, it removes one of the incentives to write well tested code because the difference in terms of the time it takes to close a story with good tests or not is marginally different. This effect in general is all about removing quality ownership from the development team and transfer it to the QA function.
Lastly the combined effect of the first two is that velocity falls, the project gets behind, people (well mostly managers) get nervous and start sending the message that quality is not important. The first signal is to encourage QA to log a bug but pass the story. The second signal is to de-prioritise existing bugs, effectively ignoring them (as we all know bugs underneath a given critically are never fixed and fall into the priority blackhole).
And there you have the QA paradox. Does this resonate with your own experience ?
July 22nd, 2008 at 3:35 pm
In my experience, thorough QA is always good. We may run into issues with it only if we do not process its output (basically, categorize the bugs) properly. Let me attempt to address each of the three points raised above.
I agree with the first point that it will take longer to finish a task. When I started my career in software development, I had the naive notion to strive for perfection, both in development and QA. As I get older (and hopefully wiser), I realize that we should strive for “good enough” instead. We should also bear in mind that oftentimes fixing a bug introduces more bugs. The famous story is that of astronauts carrying bulky binders of known bugs when boarding the Space Shuttle because NASA deems the bugs minor enough and does not want to risk introducing more unknowns. For this to work, managers must be able to accurately classify bugs at different levels.
I do not agree with the second point above. My experience has been that, when QA is thorough, developers tend to be more careful in their own testing, to avoid the embarrassment of their stories failing QA. At any rate, if QA’s testing is automated, then I personally would not make much distinction between developers’ unit/integration tests and QA’s. Again, I have seen that developers’ attempt to stay respected among their peers will keep them diligent. Managers should also monitor whether individual developers tend to produce code that often fail QA.
The last point can only be addressed with more experienced managers. Again, the signal should be “quality is important at certain levels”. Someone will have to make a call to decide how bugs are categorized no matter what. How this can be done effectively probably should be addressed in another blog posting.
July 27th, 2008 at 4:48 pm
I would not contend that thorough QA always causes this paradoxical effect.
I only point out that there are circumstances when it happens. Granted, when this happens there are other problems in the project and in the team that need to be addressed. It is still a fact that a more thorough QA practice can further destabilise the development process and ultimately paralyse delivery.
Your comment also makes it clear that the style of QA applied is important. Automation to test validated behaviour will never lead to this effect. I think this effect is a result of a tougher acceptance (i.e. at a level of details that appears excessive) and exploratory testing on cases that would not be relevant to the user (because it is not a valid business scenario of because it is a rare circumstance easily worked around in practice).
Note that this is very much in the realms of perceptions/communications. It may also mean there is a disconnect between an agile development practice and a waterfall QA practice (with all the quotes and asterisks that apply
).
July 28th, 2008 at 2:39 pm
I am curious about what would constitute agile and waterfall QA processes and their main differences. Can you elaborate?
August 4th, 2008 at 2:29 am
I think QA for waterfall is very much about verifying the delivered software matches the detailed requirements. Moreover the waterfall process would have clearly defined the data space.
In an Agile practice, verification should be fairly light. On the other hand, exploratory testing would be more important to uncover unexpected or possibly unwanted behavior. I think in a very true way a lot of what would be bugs in a waterfall process would become new requirements in an agile process.