Regression Testing
I had a discussion with a colleague the other day about the necessity of regression testing in fast release cycles.
I argued that manual regression is not necessary then, aside from the fact that it is typically in direct contradiction with the idea of having fast release cycles. Manual regression simply takes too much time for a system of significant size to allow monthly releases.
I believe firmly in automated regression. As a matter of fact we have three suites of regression test: unit tests, developer written integration tests and QA written silk tests. I also believe that the purpose of regression tests is to give the development team some confidence that no major regressions were introduced since the previous release. The word “major” in the previous sentence is very important, as is the expression “some confidence”. In other words the level of effort that goes into the regression testing is highly dependent on the appetite for risk the team has.
There are three good arguments for short release cycles and therefore limited manual regression.
First manually regressing a system is inefficient and should be limited to parts that cannot be automated (due for instance to the necessity to co-ordinate with other systems). Not automating the regression tests that can be costs development time (because feedback comes late in the process as bugs) and QA time. It obviously limits the ability to release in short cycles (our manual regressions currently take around three weeks for the entire QA team, leaving a week to test an entire month of development).
Second if the release cycle is short enough, most of the areas touched during the development cycle will have been tested as part of testing the new features. This may not be very thorough as far as regression goes but should uncover most “major” regressions and as a result improve the existing regression test suites.
Third having fast release cycles, including partially or fully automated deployment processes limits the risk created by regressions by providing an easy way to address them quickly, as they are discovered in the production system.
Should I underline that none of this is acceptable for critical systems. That is why the thoroughness of regression is dependent on the risk appetite of the system’s owners. An obvious follow-up question is “How do I know how much risk I am taking ?”.