This is the first of a three parts series.

I first stumbled upon widespread caching of objects in 2001 when I joined Calypso. At the time the idea was mostly to save time on the relational to object mapping. I did not realize at the time all the consequences of that on the design of the software. Since then, companies have flourished selling distributed caching and I discovered that I was not the only one that lacked this knowledge. So I will try to summarize my accumulated experience on this subject in the first two parts and I will try to finish wih a more speculative solution (a solution that I have not had the occasion to test personnaly).

The most important thing to remember when using caches is that objects that are cached should not have a direct reference to other cached objects. I first understood that in 2002. The reason is that if you have a reference to an instance in cache; when that instance is replaced by an updated version of the object the referencer will be holding a stale version. This obviously quickly becomes a design headache. There are two basic solutions to this (I reserver the fourth one for my speculative part).

The first one (I tried it in 2002-2003 and it works fine for a small project with few cached objects) is to actually reference the instance and never discard it. Instead, when an update comes in the existing (cached) instance is fully updated. This is a bit cumbersome to implement. In java it has the advantage of making use of garbage collection properties with very long lived cache objects and very short lived update objects.

The second one is to modify your object model to use only the identifier of the cached objects. This in tuen implies to provide access to the caches from almost everywhere in the code (whereas in the first solution few parts of the code, apart from the update mechanism, need to know about the cache). I am sure that dozens of solutions can be found to handle this. Ideally, the solution does not imply to change the interfaces (accessors) and most of the code, aside from direct referencers can ignore the mechanism. Also note that collections can become a bit trickier to handle. This is the solution the project I currently work on uses. It requires a fair amount of code to have a well polished solution.

Now there is at least third solution that I can think of. It was rejected in my current project because the team did not have a high level of comfort with it. That solution is therefore entirely speculative on my part. I will present it in part III of this series.

Next: Of the impact of caching on your design (II)

This article is long overdue but, trying to be agile, I had to adapt to events and threads that happened since I my previous entry in this series.

One of the important points in the graphic design book I read is that graphical design is iterative. Each iteration on design helps refine it by eliminating noise and keeping only the relevant information.

As Agile proponents know very well, software development and software design are iterative processes. However it is often seen from the perspective of adaptation to change and short time to market. It is undeniable that this is true. But it is also true that underlying all the software development processes I know is the iterative refinement of design. In waterfall, even if we skip over the original misunderstanding that made it an explicitly iterative process, each part of the process is iterative (from requirements to specifications to development). Formal methods use an iterative process to refine and precise the specification until it is translatable into code. Refactoring is a common name for refining code design.

The result of these refinement cycles is to make the software more elegant, easy to navigate and enhance it’s ability to accommodate user requirements.

Which leads me back to design in general as Richard Gabriel was presenting it at QCon: human design is canalized. We use patterns to accelerate our design and we refine our original design until it is aesthetically satisfactory. If you look around all our activities are driven in this way. Even scientific theories work in this way. And it takes specially inspired people to break existing patterns and find new ones, new ways of doing things. We call them geniuses.

Back to the point, the only way to achieve good software is to enable and support successive polishing of the initial design (formal methods use demonstrations for that, non formal methods use refactoring). More importantly, as software lives in constrained and changing environments, it is constantly bent and scratched and customized. These in turn results in software decay that will eventually make it useless, unless the proper polish is constantly (and lovingly) applied.

Last week-end Agile in Action asked whether we (as professionals) should you always do what customer wants (and don’t skip the comments at least one of those is great, will give you pre canned arguments in such a discussion and its not mine) ? It concluded that if a customer wants to compromise quality the author would walk away. I would agree with him, when it is possible because it is very much the sign of a barbarian culture on the clients part.

The way I understand it,  quality is a measure of how well the software product finally delivered implements the desired properties (or qualities) and in particular the desired functionalities. In that context, compromising quality means that you are prepared to deliver a product that may not be fit for its intended purpose. Which generally really means they you want to go to the market as fast as possible, let the customer test (discovering in the process what they really use and how) and then fix the problems as fast as possible. In short a barbarian approach. I believe that stating it this way is actually acceptable because it means the client understands the implications of the choice to be as fast as possible (and it may even be ok to execute on this in very specific circumstances).

Obviously, this is exactly the kind of situations that Agile methodologies address best. They enable to discover the functionalities the client really cares about as well as how they should work. They enable to make the other -ilities emerge though refactoring when they become desirable attributes of the product. They enable to bring features to market as soon as they work (as opposed to before they are tested). They are more satisfying for both the client and the implementor.

In conclusion, rather than compromising quality for the things you need, drop the things you don’t. Unless your client wants to consciously sabotage the product and specify the bugs, truly making them features.

This is not a summary of Jay Fields’ talk but mostly the thoughts it inspired me.

To make it clear, in Martin Fowler’s nomenclature, Business Natural Languages are external Domain Specific Languages. That being said, they tie pretty neatly with the major principle of software development that you should establish a common language between the parties involved in a project (which, in the context of QCon is a tenet of Domain Driven Design).

Anyway, the basic principle is that you create a language that is close to natural language but applies only in a specific context in the business and you create a user interface from it. This is the grand rediscovery of text based interfaces, which many of us know are much more efficient than graphical user interfaces in many cases. The example that specifically spring to my mind is Summit System’s quick entry for some trades that applied that exact principle by enabling the users to use their terse business language, the very same one they use on the phone all the time.

I would go a step further. Defining that language probably is a very useful step in the design of a graphical user interface. Because to be natural and easy to use, the graphical user interface should embody that business’ language.

I can see a few obvious examples of when this kind of textual interface can be very efficient. One case:

• A part of the business that would lead to a fairly complex dialog box with plenty of optional fields,
• That is not constantly and repeatedly used (until there are efficient embeddable language workbenches).

The other (as illustrated in the previous example):

• A terse syntax already exists in the business that would make even a simple GUI clunky (for instance because selection from list is too long).

It could also be very useful to program query in a less formal setting, like processing e-mail business queries in a partially automated fashion.

Jay Fields used regular expressions to create his BNL languages. However, outside the context of his projects, I feel that using formal grammars and tools (like lex/yacc, antlr or similar) would make the language easier to maintain and enrich.

A final point that was stressed during the presentation is that such languages should be very simple and specific to an area of the business. Remember that computer understanding of natural languages in general is a very complex problem.

This is by no means a summary of Eric Evans’ talk, only my main take away and some thoughts it triggered.

I have to admit that the term Domain Driven Design was a bit fuzzy and this talk did a good job at clarifying what it means. Essentially it means that when you build software you effectively build a domain model as well (whether implicit or explicit) and that you should pay attention to that.

In particular, the domain model you build is partial on the domain itself. It reflects some of the views you (and hopefully) your users, take on the domain. It serves the specific purpose of building a shared representation of the domain with your users. All that circled back neatly with agile and embedding the customer and all those good things.

Also I cannot pass over the row boat analogy through which Eric Evans helps explain why agility works more easily on small teams (he noted that boats have a maximum of eight rowers, in part because rowing requires perfect synchronization and that a single miss by one of the rowers throws everybody off rhythm). I would add a few notes to that. It is interested to note that boats of two and four are self organized whereas for eight someone needs to call the cadence. I would also like to reflect that it does not mean that boats of more than eight cannot exist (see the antiquity’s galleys). Just that to make them faster than eight will require a lot of training and that it may be difficult to assemble the individuals that fit together and that it may require more non rowing team members (who thought dead weight ?).

There was another greatly useful analogy with athletic relays which illustrated one pitfall of over engineering system interfaces but it would be too long to explain here.  The core of it is that you should engineer an interface that the other system will be able to handle reliably. If it is a csv file exchange then so be it.

Anyway, a final point before the presentation was interrupted by a fire alarm, was that there will always be multiple models of the same domain. That these models are more or less compatible and that you may need to translate between them. As it is, I have first hand experience in that with FpML, Calypso and others and I will add this, if you need to translate your model into another, make sure they are not too different or it will cost you a lot of time and effort which is exactly what you are trying to reduce by defining your domain specific model.

What is the link with strategic design in all that ? Well the premise that not all of a large system will be well designed. Therefore make your domain model explicit helps focus on the core of the system. If peripheral parts of the system are not as well designed, it is not so important.

Once again, this is not in any way a summary of Richard Gabriel’s talk but mainly some thoughts it triggered (both positive and negative although I traditionally have an easier time with negative).

The talk was basically about how will we design the future ultra large software systems. Specifically, how do we design things that are essentially too large and complex for humans to understand.

It started with a few definitions of the word design. All of which somehow failed to stress that design process is a purely human (although maybe not uniquely human)  activity whose product (the design)  is destined to other humans for them to understand the intent (see White Space in my Software).

Also, the speaker made very interesting analogies with biology or utility networks, although unfortunately he failed to point that neither were designed (utility networks have been designed in successive parts). Also I would have loved to see him draw a parallel with how western economies were built over the past three centuries. Utility networks and western economies had parts locally designed (although locally in our understanding probably was globally for the designer).

I was also puzzled by the expression evolutionary design. Which seems like a complete contradiction in terms, for which I will have to seek clarification from Richard Gabriel (because I was too shy to do so at the conference). I am saying this because he correctly pointed out that humans have a specific way to explore a solution space (canalized which is exactly what design is). To me evolution is another way of exploring the solution space. Random is another one.

Which should yield the conclusion that we don’t design such systems. We build them with intent but without design.

Well not the ones you may think about. It’s been years since I manipulated an actual pointer in my code.

I am still reflecting on that graphic design book I was talking about here where another important thought was that one of the goals of graphic design is to lead the reader efficiently through the document.

And I have been wondering since then if and how I could apply this to my software. After all, it is meant to be read, if only by a maintainer. So how do I make sure that a maintainer immediately captures the ideas and the intent that I coded ? How do I let a new developer understand what I intended to do and why I designed it in a deliberate way ? Or the way I left a complete mess  sometimes.

Well it really starts with software development 101: sensible names for classes, methods, variables, members, functions and what not. Don’t let your reader guess what this thing is doing. It also about the single point of return and the verification of parameters at the beginning of method. A mix of conventions, explicit names and familiar patterns (yes design patterns) that reduces the noise. It is also about concision and simplicity: short text, do a single thing at a time. Keep things simple (you know all those static code metrics that PMD calculates for you ? keep an eye on them).

But all that does not provide any insight on the structure of the software. So you have to reach for higher ground. First through the source code layout. The choice of modules, the dependencies described in your build system (do I think Maven ?) give valuable insight design of your software.

Final piece in the puzzle: the tests. Specifically, for this purpose integration and functional tests as they provide examples of how the sub systems are assembled and work together. Once again, naming is crucial. If you use an ?unit framework, don’t be shy of long long long names that fully describe what is tested (remember, you will never have to type these names again). Or even better, use BDD tools, like rspec. An important point on the tests: they should be written as elegantly and as well as the so-called production code to be useful for that purpose.

Okay you think, that’s nice and good but what about documentation ? Documentation ? What is that I say ? Well seriously, documentation is great for end users of tools, not as good for the people who will actually change the code. On the other hand it is sometimes very useful to highlight a particularly difficult point. To document extension points, nothing beats a short example. Also a complete and well maintained documentation is absolutely mandatory for the configuration files. Especially if they are written in XML because it is almost impossible to make an xml document easy and nice to read !

So here you have it, my ideas to make it easier for people to get an understanding of your software. I strongly suspect that it would be possible to research code patterns that make it even easier to get into someone else’s code. But overall, my feeling is, if you think about it while coding, and leave plenty of white spaces, the world will be a better place.

Next post: Polish on my software.

And camel case in my title. Sort of.

Anyway, did I mention how much I loved this book ? Well, it lead me to think about the analogy between software design and graphical design. Granted, the analogies may work better with architecture or civil engineering but I just read a book on graphical design. The different thoughts that reading this book inspired me won’t fit in a single post, so I will transform this in a series. Obviously, as with most analogies, it is only a trigger for reflections, not a perfect parallel of activities.

My reflections stem from four remarks central to that book:

• empty space has a form given by the shapes it contains,
• the design is meant to efficiently lead the reader to meaning (actually to meaning relevant to the reader himself),
• the design is the result of successive refinements that aim at eliminating noise (or useless information),
• the text layout must be adapted to the human brain’s capabilities.

After this lengthy intoduction, this first post will entirely focus on the empty space, hence the title.

Thinking about the empty space means making a conscious decision of what to include and exclude in the design rather than throwing every possible idea in the design. I believe it applies at each stage of the software design process. It applies when selecting the features (hence the feel of quality of many of Apple’s softwares). It applies to the definition of classes (when you define a clear and simple interface). It applies to the user interfaces. The only place where it happens almost naturally is the code itself (who adds useless statements honestly ?). All the other places require conscious and sometimes painful choices.

Ever noticed how early versions of a framework are simple and a breeze to use but then, as years pass and user base extends, they get bloated and a pain ? Ever noticed the same of softwares (office anyone) ? All that to say that I believe that simplicity and clarity are great design qualities in software and that a good way to approach it is to keep thinking about the empty space you are filling with each new feature, new button and new method.

I also believe that both agile methodologies and TDD help keep the design simple and readable. Agile methodologies by putting in plain view each additional feature, allowing to remove the unecessary ones before they are integrated in the design. TDD (or BDD) because it appeals to the lazyness of the developper to make simplicity a top priority.

Unfortunately, in software as in life, white space tends to get filled if you are not careful and constantly prune. Which could bring a useful gardening analogy… another day maybe.

Next post: Pointers in my Software.