Or How thorough QA can lead to lower quality ?

Indeed this is a paradox but it appeared very clearly to me after observing a change in the QA style in our team. Before this realisation I would have happily argued that thorough QA is the best thing that can happen to a project. The paradox unfolds in three parts.

Firstly it makes it harder (and more importantly longer) to finish stories. I understand that it also avoids accepting low quality work. However the QA perspective is slightly different (and this is good) from an actual user’s. So thorough QA, uncovers plenty of minor problems that do not significantly impact user experience or that would be minor production issues. This generates frivolous work that delays the development of more important stories, bug fixes or refactoring, thereby reducing the overall quality of the product.

Second, after a while the QA style modifies the developer’s attitude. When QA is thorough there is a sense of false confidence in the QA process that generates sub par unit and integration tests. This in turns leads to more bugs. Also, when QA is too picky before accepting a story, it removes one of the incentives to write well tested code because the difference in terms of the time it takes to close a story with good tests or not is marginally different. This effect in general is all about removing quality ownership from the development team and transfer it to the QA function.

Lastly the combined effect of the first two is that velocity falls, the project gets behind, people (well mostly managers) get nervous and start sending the message that quality is not important. The first signal is to encourage QA to log a bug but pass the story. The second signal is to de-prioritise existing bugs, effectively ignoring them (as we all know bugs underneath a given critically are never fixed and fall into the priority blackhole).

And there you have the QA paradox. Does this resonate with your own experience ?

In agile literature, stories are sometimes described as a “place holder for a conversation”. The thinking behind it is that the client and the development team (developer and QA when there is one) will sit together and flesh out the details when the story is ready to play. This turns out to not be the entire story. First the story has to go through the estimation process. In order to give an adequate estimate (not necessarily a precise one) some details need to be available already. The story is enriched by this first conversation. Second because during the development phase, the specifics of the story may change due to technical difficulties, new ideas or the discovery of new use cases. This simply means the story evolves and changes and gets more precise as it approaches completion.

Unfortunately, in the team I currently work on, this introduces friction. The big question is how detailed should the story be for QA ? And even if we agree on this, how are discussions and changes recorded in the story ? Even more importantly, who records those changes ?

Enters BDD story writing.

I recently shared with the team an article describing a recommended story format introduced by BDD. I hope this can be used to address the different changes in the story. From the short narrative and user scenarii written by the client to the use cases discovered during the development and QA phases, the BDD format enables to easily enrich the story “just in time”. It enables to keep track of the state of a changing specification. It leaves most of the non critical specifics out and it does not record the full discussion but simply its results. It is also a format that is easily accessible to technical and non technical people alike, both for reading and writing. I will be sure to report on our progress using this technique.

All this helped me realise a few things about Agile Storytelling. As the title implies it is akin to the communication technique called storytelling. The story facilitates the conversation between people with very different backgrounds. It sets the common vocabulary. It also builds a common vision of the story’s results and in the long term of the product. A key difference is that the story is not a one way message, it is written by multiple people putting their own point of view in a language that is understandable by all.

Did I mention that an agile project without well written stories and a clear definition of “done” is bound to fail ?

Don’t miss Jay Fields’ excellent article about story estimation on infoq.

One thing that I think is particularly important: the accuracy of the estimation is not important, consistency is. The planning is done by combining velocity and estimations. The velocity already contains estimation errors so as long as your error is consistent everything is fine.

Since I started exploring the world of agile methodologies I have come to understand that software engineering is all about communication. Even typing code is all about communicating your desire to the computer and communicate your intent to fellow developers who will come after you. Every time a communication break down happens, things will be painful. The lucky ones will be gone from the project before the pain becomes noticeable.

This in turn led me to appreciate the meaning of ‘people over process’ found in the agile manifesto. Sure advocates a number of important and useful practices. Most of them are aimed at improving communication at one level or the other (Stand-ups, pair programming, tests, continuous integration, stories, show and tells, retrospectives to name a few). None of that will be useful unless the people involved in the project actually make it work. This is not specific to Agile but to any process that is not mechanised.

So it takes a particular type of people (from stake-holders to developers) to make an agile project work, to get the process running and working. What I am starting to think is that regardless of the process initially chosen, these people would make it succeed anyway, unless they were strongly constrained by red tape and armed guards.

So is it the agile methodology that makes the project succeed or is it that the people who like agile environments are better at delivering ? In other words is the agile process immanent or transcendent ?

César touched on hiring and promoting a while back. This came in conjunction with a number of other things that I have heard or lived in the recent past. It is all about the “right” person.

What is the “right” person ? Is she the person that satisfies all your criteria ? Will your criteria become obsolete as the environment evolves ? Or even worse you will discover that you had the wrong criteria ? What if the job you envisioned for the person has changed before they could actually start ? And if the “right” person does not want to work with you, will you settle for a “wrong” one ?

Hiring is all about compromise. The firm needs to make a compromise between diversity of opinions and common values. It needs both to be innovative. Compromise because a job always needs a mix of qualities a fine balance between expertise, character and ethics. There is also an important compromise between immediate results and long term contributions by the new hire. For instance, do you prefer to hire a cheap graduate that you will have to train but you will also get to grow or a seasoned veteran that has done the same thing ten times and will do it again for you ?

Earlier in my career I tended to interview people as a reflection of myself rather than as separate contributors. It turns out no one like me wanted to work with me. As I have aged I now realise that a job definition will always be stereotypical. The person hired to fill it will largely influence its actual shape. And the social group (team, department, company) is the result of the co-evolution of all the persons that compose it. Therefore the “right” person would be right in the situation as it was before you hired her. There is no telling what the situation will be after.

So as a conclusion, if you are deterministic, the person you hire is necessary the right person. For the rest of us there is no such thing as the “right: person. In other words nobody is perfect and right implies a contextual perfection that does not exist. More importantly, looking for the “right” person clouds your judgement as a recruiter. You have to be agile as a recruiter as well.

César over at non linear enterprise is spot on about feedback loops.

That is what so-called reenforcing practices are about. Pairing introduces a very fast feedback loop on code writing. Automated regression tests provide a fast feedback loop on development. Retrospectives are a quick feedback loop on the project.

All this would tend to indicate that Agile is actually a more tightly controlled process than many other development practices. Even more interesting the control is exercised at the time and on the rythm that is most appropriate to the controlled process. Finally most of this control happens without external intervention (which will insecure many managers) which removes the controller single point of failure. It also makes it possible for the process to adapt and self repair.

But beware that feedback loops can also be a factor of instability…

I finally had the revelation, the answer to that puzzling fact that, no matter how much effort and faith a company puts into its transition to Agile methods, it fails. Well it is a bit blunt. Some actually succeed. And others manage to maintain a pretence of success for years.

I have been reading Peopleware recently and found the answer there. I am sure it exists in other places as well. It is not as if it was a hidden treasure of wisdom or a lost manuscript by aristotle. It is the well known fact that change is difficult. So difficult actually that the authors of Peopleware advise to effect only one change at a time ! And here is my realisation. To transition to Agile takes more than a single change. You have to change how people work together collectively, how they work individually, how information flows inside and out of the team and even what type of communication is needed. Any of these changes is challenging by itself. These categories may even hide multiple changes (switching to Test Driven Development and Pair Programming for instance).

Interestingly, this realisation ties perfectly with an open space discussion that happened at the BayAPLN a while back where we discussed the strategies to effect a successful transition to Agile. We came to the conclusion that it was easier (necessary ?) to foster change agents (positive deviants). Another successful approach was to sandbox agile practices. Agile sand boxing is essentially an attempt to transition small isolated (geographically or functionally) groups that are willing to become agile. For obvious reasons, groups that are not sand boxed are likely to be crushed by the rest of the company. The sand boxed group is freer to experiment with practices and to be allowed to progressively install new standards. The rate of possible change is also arguably improved.

So now I think I have linked the reason for failure and the path to success. Or am I missing something ?

I recently got to feel the weight of social conditioning in the enterprise. I was asked to temporary (three weeks to be precise) fill in in a somewhat managerial position. I started with the idea that I have great confidence in my team mates and that therefore I would not need to do much in this interim period aside from fielding a few e-mails and showing up in a few meetings (I was even spared the administrative burden !).

I soon discovered however that I was trapped in the conventional thinking that is continuously hammered into us that people will not do what they should and that, as a result, if you are delegating the work, you should also verify it very thoroughly. Even knowing it is not good (see Peopleware for convincing arguments) I still felt compelled to do it. Worse, I had to refrain myself from over doing it and try to be more supportive than prescriptive !

The awful thing is that it is an attitude I don’t like as a subordinate, it is something that does not happen when the relationship is truly a peer to peer one. I came to realise the reason I was trapped in this attitude is that I felt singled out as the individual responsible for the outcome. It did not feel like a team responsibility but my responsibility.

That is what I would call the management trap. The illusion that things are at stake for you personally rather than the group. It is the social trap that we use everyday to ensure a control structure rather than a collaborative one.

I have been pondering this post for a while now and thought that a visit to the Land of the Rising Sun was an appropriate incentive for me to write.The team I am part of has attempted to work in an Agile manner for the better part of two years with a few ups and mostly downs on that ground. This is however one of the big successes we had in terms of work organization.From the inception the team had QA members for which a test environment needed to be maintained. By itself, deploying a QA environment every couple of days is a sizable task. Even though that task was, over the months, considerably trimmed down, it is disruptive for a developer. More so when QA needs to chase a developer to do the work.As a further disruption, our environment is connected to a number (5 or 6) other systems that occasionally and regularly failed. We therefore needed someone to investigate and resolve (or get someone else to do so) those issues.Finally we had an automated integration test suite running on a fixed schedule. That suite was two long to execute after each check-in as our regular regression test suite does. And the schedule was long enough that the blame was typically difficult to place at first glance. When it was not a false alert due to some other system failing.All those activities quickly prove rather expensive in terms of developers time because, being good citizens, two or three persons would jump on the problem to attempt a quick resolution. Or on the contrary, everyone felt too busy to bother, assuming someone would pick up the flag, wasting precious QA hours. So the obvious solution, as you may have guessed was to designate someone to perform those ungrateful tasks. Two weeks would the term. That person would be called the Ronin.It quickly became apparent that the Ronin plays a release engineering role and so it was assigned the responsibility to maintain and enhance the build system. It also became apparent that in many cases the Ronin was not fully occupied and had room for short, lonely tasks. And he became the teams’ designated bug killer during his term.Maybe Yojimbo would have been a better word.

Miki, over at Leadership Turn, tagged me and asked what I will do with my tax refund. This obviously has nothing to do with my usual topics. Although, strangely enough the next meeting of the Bay APLN has been titled “Taxing Agile”.

Let me start with two notes. First this refund only applies to US tax payers. In most other countries, don’t expect to get any money back. Second, whatever you with your refund, given the poor shape of the US economy, it will be good for the country and quite possibly for the world. Paying down debts and saving are good as it frees some cash for banks to stay in business. Spend and it will help businesses stay afloat.

To link this back to agile subjects, let’s remember that, in a well designed tax scheme, only those who can contribute are taxed. Moreover they are only taxed according to the level they can contribute to but only at a level that encourages them to be able to contribute more. In such a well established tax system contributions grow as well as the well being of contributors. Now I think about it, it really is similar to how companies work. Typical teams try to over tax their members until they have nothing more to give. Agile teams tax their members less to encourage a continuously growing return.

I kind of wonder what a refund means in that case ? Is you were too taxed ? Or that you did not produce as much as anticipated ? I will let my valued readers answer this.

Meanwhile I will answer the original question. As far as I am concerned, it will go into the savings category, most probably under the college fund header.

Alas, as I do not know any american blogger well enough to tag them, I am unable to fulfil the Meme. Oh well… I need to network more !