Why Agility Works

Why do agile methodologies work? More importantly, why do they work better than traditional methodologies?

I'll single out three reasons. (Martin Fowler's excellent article provides many more.) Agile methodologies work because:

(a) They provide a new answer to an old problem

(b) They recognise, and emphasise, creativity

A New Answer to An Old Problem

Software is hard to change. For instance, it's cheaper to make a change early in a project, when you are gathering requirements, than to make the same change later, after the code is written.

Traditionally, we have minimised the cost of change with careful planning and extensible designs. We emphasise comprehensive requirements definition up-front. Then we plan for change when designing the software, so that future changes can be made with minimal impact on existing code.

To summarise, the "traditional" view is that:

Changing software is hard - so we'll avoid doing so

(We will follow an exhaustive design process and we'll build "extensibility hooks" so we can add things later without changing existing code.)

The agile community offers an alternative:

Changing software is hard - so we'll make it easier!

Agile processes aim to produce simpler software. It is well-designed, but it makes no guesses about what "someone" might need "later". Simpler software is easier to change.

Some agile methodologies also include automated regression testing. Automated tests are important because when we say, "it's hard to change software", we really mean is that, "it's hard to change software without breaking it". It's relatively easy to change existing code, but it's hard to be sure whether or not you've broken anything. With automated tests, you can be sure (one way or the other!).

It's worth noting the differences between agile methodologies. Extreme Programming (XP) is firmly in the "make change easy" camp, with relatively little emphasis on up-front planning. In most other other agile methodologies, ease-of-change supplements up-front design but does not replace it.

Recognising Creativity

For many years we have drawn analogies between software engineering and traditional engineering - making cars, buildings etc. Traditional engineering makes a clear distinction between designing and building. The designer(s) figure out exactly what should be done, then the builders create the physical objects according to the specified design.

Software engineering is different because no physical objects are constructed. It's all "brain work", so our analogies with traditional engineering are flawed. We should consider all software development, including coding, to be "design". In other words, making software is not like making a set of blueprints and building a house, it's like making only the blueprints.

Martin Fowler and Jack Reeves explain this very well in their respective articles.

If we accept that making software is like making blueprints, not houses, we can draw some important conclusions:

Making software is a creative process

Creative processes cannot be planned and scheduled in the same way as other processes

It is very important to have regular face-to-face collaboration between all interested parties

Imagine asking an architect to create a set of blueprints for a house. Would you provide a written specification and then review the design in the form of UML diagrams? Or would you prefer to communicate your needs verbally and review the design by discussing the actual blueprints, as they evolve according to your feedback? That's agile development :-)

Experience

My own "favourite" agile methodology is Alistair Cockburn's Crystal Clear. Crystal Clear is based on extensive research. Alistair spent 10 years visiting software projects all over the world, then he wrote Crystal Clear to summarise what he saw in successful teams. It's not a theory about what might work, it's a description of what actually does work.

Finally, my own experience corresponds with the claims made by agile methodologists. Simplicity, customer collaboration, and flexibility tend to create successful projects.

Perhaps the biggest criticism of agile methodologies is their apparent focus on "today's" needs, without exhaustive planning for future changes. And yet, I have seen several monumental failures due to excessive up-front design -- too much planning for imagined future changes, too many attempts to cater for things that "someone" might need "later", resulting in complex architectures too unwieldy for everyday use.

I want to create working, valuable software. For most projects, agile development is the best way to achieve that goal.

Page History: Created Feb 04. Split into two parts (this is the first) on 22 Oct 04

Minor edits: 11 Mar 06