This post is prompted by a recent (or fairly old by modern speed standards) entry by Jay Fields. My own experience over the past 3 years has been exactly what he describes, dead ends, changes, improvements which leave the test code base in a state fit for archeological research.

The biggest pitfalls I fell into (and made other people fall into, but they did not have anything better to offer): too many assertions per test, too many mock expectations, too much set up, too much coupling with the implementation. A lot of those problems stem from the overall quality of our design at the object level which in turn is explained partly by our immaturity in using tests to shape the low level interactions between objects.

I think we are doing a bit better now and I want to share what I try to do now.

Here is an example for a unit test:

@Test
public void putShouldStoreValue() {
        givenAPropertyName();
        givenAPropertyValue();

        BaseMutableDataStore testedContainer = new BaseMutableDataStore();
        testedContainer.put(propertyName(), propertyValue());

        assertSame(propertyValue(),testedContainer.get(propertyName()));
}

This uses primarily the builder idea that Jay presented to remove the technical details of stubbing and set-ups. The idea, taken from BDD frameworks and rspec/cucumber in particular is to focus the text of the test on the assumptions and the assertions. Nothing here is new. But I had I read an example like this 3 years ago, I would have saved myself and my team-mates a lot of trouble.

I am currently trying to expand the same presentation to integration tests that our QA could review and help improve and refer to as part of their testing plans. Hopefully more on this in a future post.

Also, as I am keen on continuously improving, please do not hesitate to help me get better (my special thanks to Jay for his posts already).

Requirements Come Second

Brilliant article on agile transition.

In particular:

Why is it that doing the wrong thing well is so much better than doing the right thing badly? I think the explanation lies in our old friend the waterfall.

Organizations which attempt to do the right thing naturally focus on requirements. This leads them to take a requirements-first approach. This starts the waterfall, and since 80% of these requirements maybe unnecessary these organization create the demand that pushes up costs.

In a previous post, I was proposing to stop the line when discovering a defect that is clearly worth fixing.

Stop the line is an expression coming from the Toyota production system where, because there is not inventory to use as buffer, a defective piece results in an interruption of the production line. The advantage is that it forces problems resolution. In the case of defect management this means that instead of recording a defect we know we will have to spend time on, and store that time in our inventory of bugs to fix, we do it right then and there.

This obviously assumes that the team can make a resolute and quick decision of what needs to be fixed or not. It also means that it is that much less effort spent into tracking the defect’s life from discovery to investigation to resolution to verification. There is no need to record that defect as the record will become obsolete very quickly. This is already what happens for severe defects found in a production environment. An extra benefit happens when the defect can be directly traced to the originator (which requires a fast feedback loop) as the originator (rather than another person) will have to sink the time to fix the defect, very visibly so, introducing a very strong incentive to deliver quality work all the time.

Adopting this attitude also means that defects on record are the ones which may be fixed in the future. No need to really prioritise them anymore as if the defect was to become more important it would be re-raised in a stop the line manner. Defects on records would then become a sign of “code smells” or highlight areas that may need significant re-factoring that the team can take on at a convenient time.

When playing with this idea after David Anderson’s talk at the Bay APLN and pondering the Iron Chef presentation that happened there too, someone objected that there are cases when defects would require a serious rework and take a lot of time and should therefore be tracked. And I agree. However, either that rework should already be counted in the defect originator’s work and then it is already tracked or it is likely the defects were there for a long time but were never discovered before because the function was not used/tested. In this later case, it should be considered as rework to upgrade the existing functions and be prioritised and tracked with the rest of the work. It may be that it becomes client valued work (pretty similar in my mind to the upgrade work you have to do in a house when the construction code changes).

What do you think of the stop the line philosophy ? Have you practised it ? Did it work for you ? How did the process look like ?

In a previous post, I was proposing to avoid recording things that the team knows for sure it will never attempt to fix. Things that seem so unimportant as to be trivia and distraction.

The reason I am proposing this is that investigating a defect, finding a way to reproduce it in a controlled situation, prioritising and maintaining it all take time. If the defect is never fixed all that effort is wasted. This should be made very clear by reducing the number of priorities used for defects. We could simply stop recording them. Note that things reported by users would never go into this ‘refuse’ category.

Now I understand that some people are more comfortable with recording everything to keep track of what they did and just in case it becomes important some day. I content that this is the reason why we need five priority levels (in our case) and that recording things when they become important (rather than in case they do) is usually enough to ensure user satisfaction.

What is your take ? Is there real benefit in keeping an exhaustive record of findings ? If you already reduced the number of recorded defects, how did you benefit ?

My thanks to Simon Baker at Agile in Action to make me discover Bob Sutton.

There is plenty of good things in his blog but let me point out these Peter’s Prognosis and Peter’s Paralysis: Timeless Wisdom from The Peter Principle and Andy Grove Tells The Truth About What Great Leaders Do

And do not forget to take the asshole rating self-exam. I am probably in complete denial as I scored 3.

In a previous post, I was proposing to reduce the number of defect priorities to three. In the project I work on we currently have five. Elements in the last two gather dust (so to speak) because no one is looking at them. We know that things that stand in these priorities will never be fixed.

It made me wonder why we picked five priorities. I suppose it was an attempt at having fine grained prioritisation. In practice however we only use three priorities: must fixed in the next day because something really important and central is broken, must fixed before next release because the users cannot live with it and the fix when you have time/opportunity defects.

That makes three priority levels. And it ensures none of the reported defects falls into oblivion. I am under the impression that this combined with a defect time value and putting defects reported by users in the first two priorities would help manage defects in a sensible manner.

What is your practice ? How many priority levels do you use ? Is it that we have too many defects, period ?

In a previous post, I was proposing to automatically assign a very high priority to a defect reported by a user (user in the released product). Arguably many of these defects are not critical and for the critical ones arguments against this idea will be short lived.

Why assign a high priority to non critical defects reported by a user ? I have at least two good reasons for this.

First, because a user took the time to report such a defect, it had to be very visible and quiet annoying at the time he discovered it. That should be enough to put it ahead of anything that was found in the rest of the software development process that was not scheduled immediately for a fix.

Second, it is a very strong signal from the team to the user that his feedback is valued and acted upon. This in turn will encourage users to report not only cosmetic defects but also inefficiencies in the way their workflow is implemented. It is actually something that I have noticed in many cases, until the cosmetic noise is gone, many users do not focus on the actual workflow and usability.

In many cases, our appreciation of what is important is different from our users’. Putting a high priority on defects that are reported by the user makes sure we do not substitute our priorities to his.

What do you think ? Are defect reports from your users useful ? Is it easy for them to report defects ? Would this introduce too much noise in the defect tracking system ?

Joanna Zweig and César Idrovo have a second article in the agile journal about building common purpose.

Let me only reinforce that organisations that wish to involve everyone in the decision making process at all levels should remember to introduce it progressively. Tocqueville noted in his Democracy in America that when people are not used to make the small decisions (or participate in making), you cannot expect them to be able to participate usefully in the big ones.

In a previous post, I was proposing to assign a time value to defects and have them expire automatically after a while. As a disclaimer, obviously it would be useful to extend the life of a defect if desired. The rationale behind this falls on two points.

First if a defect stayed untouched for a long time (and assuming it still exists at that time) it is likely to be benign and somewhat irrelevant to the end users of the system. They would otherwise have raised the visibility of this issue (assuming that communication channels are available for them to do so, but that is a different problem).

Second if the code base is being modified actively, it is likely that some of the analysis, the steps to reproduce the defect and how it manifests itself have changed. It could also happen that the defect has become completely invalid, either because it was fixed, belonged to an area that has been removed or thoroughly re-factored or that the focus of the product has changed (remember we are in an Agile environment).

We should let our defect tracking system be our memory. Let our defect tracking be forgetful of unnecessary details to make sure it provides us access to the important things.

What do you think ? Do you often go through older bug reports ? What are you using them for ?

Funny how things take a new light when you measure them. Take bugs for instance. When I started on measuring the time our current, reported, bugs have been opened I was expecting something like this:

For full disclosure, we use five level of priorities and we have a policy to fix all outstanding P1 and P2 bugs before shipping a release. What I found was this:

So effectively, we are not fixing anything that is below P2 in a reasonable amount of time. P5 are unlikely to be fixed anytime. Still we spend time managing that inventory of bugs (trying to avoid duplicate bugs, reviewing the priorities from time to time). In pure waste.

A second thing came to my mind looking at the mean time bugs stayed open. Given the amount of changes the code base undergoes in 6 months, what is the validity of a bug that was recorded at that time and was not considered important enough to be fixed ?

I will throw in a few ideas here that could lead to improvement in this area. One thing I would love is to be able to time expire defects.
Anything older than a coupe of releases should disappear automatically. Assign a high priority to defects reported from the production as they are truly visible and of value to the users who reported them. Reduce the number of priorities to three. Do not bother to record things that are evaluated below P3 in our current scale. Stop the line for anything
that we know we want to fix at the time of discovery. Each of these ideas will require a dedicated post (coming soon).

What are your thoughts on defect management ? Are these ideas outlandish ? How do you manage defects efficiently in your projects ?