Here are some brief notes on the presentation.

Introduction

General background on Crystal Clear

Big Lessons that I’ve Learned from Crystal Clear

• Methodology-per-process.  How you can (and should) tune your process to the project at hand.
• Defusing debates about documentation, by recognising the two purposes of documents: supporting the current project; and supporting future maintenance programmers.  In my projects, I have found it best not to kill two birds with one stone.
• Shu-Ha-Ri, the stages of learning skills and how that applies to agile.  People at different stages have different needs for prescriptive process versus flexibility; and different comfort levels with finalizing process elements “later”.
• Simple = good.  The simplicity of Crystal Clear is backed by both theory and practice.  Its simplicity supports the design goals of Crystal, which are safety (delivering a successful project), efficiency (cost-effectiveness) and habitability (people are comfortable using Crystal).

Links to other resources

I mentioned various other resources during the presentation.  They included:

• Material on iteration lengths: why there is no real value in saying “my iteration is shorter than yours”.
• Collaboration: the Dance of Contribution.  A lengthy piece, with a lot to take in.  I skim read it the first time I saw it, but I got more out of it when I re-read it some months later.  My favourite bit: “No idea is rejected out of hand and no idea gets a free pass.”  I’m still learning how to apply it ;-)
• Are requirements a document or a database?  I found Eric Sink’s article explains this dilemma very well.  (As for his comments about agile in the same article, I think he’s touching on an issue which James Bach described well here. Although they may not phrase it this way themselves, I’d suggest that Eric and James are both “Ri-style” practitioners, objecting to the over-emphasis of “Shu-style” agile.  As we discussed today, both styles have their place.)
• I also mentioned Eric’s “Career Calculus” page, and suggested that the same approach applies to process.  The most important thing is not where you start; but how fast you improve.
• Ron Jeffries on documentation.  His recommendations are broadly similar to the Crystal ones except (a) he has a firm view that the tests are the requirements documentation (Crystal is more flexible about how requirements are noted down) (b) he implies that you might not need future facing documentation (I would assume that if your software was worth writing, then it’s probably worth maintaining, which means it usually is worthwhile to write a future-facing document)

That all looks very brief, but on the day it took 50 minutes – honest.

By the way, I did it as an “incremental presentation”.  After each of the four main points, I paused for questions.  The idea was that each main point was like an “iteration”, and pausing for questions was like “releasing” that iteration.  It seemed to work.  We had good discussion after most points, and interleaving questions with presentation made it easier to fit the timeslot. (If all questions are left to the end, how much time should you leave?  Will people ask lots of questions?  Will they ask none?)

If you were there (or you weren’t) please let me know if you have any questions that are not covered in these notes.  Either leave a comment below, or email me.

Update 6 March 08:

I gave the presentation again today.  Here are some of the BA-related resources that I mentioned:

Scott Ambler on:

• Agile analysis
• Thoughts on the role of BA’s in agile projects
• Artifacts on agile projects

A couple of interesting points came out in discussion afterwards.  They included the idea of “emphasising facillitation over artifact creation”, the concept of ubiquitous language, and the notion that in some cases asking the customer to describe what they need is like a bunch of blind men describing an elephant.  Everyone you ask says something different, and what the business really needs might be different again.  In cases like these (where the customer organisation is, for whatever reason, unable to speak with a single authoritative voice) my thoughts are: don’t just say “that’s the customer’s problem”, or “we can’t do agile in these circumstances”, but instead apply BA skills to help the customer to articulate their needs.

>Finally, here is the page on negotiation, which I mentioned at the end.

I’ve just read a fascinating article on where talent comes from, over on Freakonomics.com.  It outlines research into the key factors that develop talent.  The key factors are:

"setting specific goals, obtaining immediate feedback and concentrating as much on technique as on outcome". [emphasis added]

For instance, life-long learning comes naturally to surgeons because they get instant feedback on the effectiveness of their decisions.  But similar learning doesn’t come naturally to doctors in fields where there is a  long delay between decision and outcome.

The research finds that development of talent is heavily dependent on the practitioner receiving prompt feedback on their decisions.  The timeliness of the feedback matters. 

Waterfall processes lack timeliness of feedback.  A classic example would be an analyst or architect making decisions at the start of a waterfall project.  Many months may pass before they learn the result of their decisions.  (Sometimes they never learn the result because they move to another project before the first one is finished.  Such so-called "experience" is of little value.)

In agile development, decision makers get feedback quickly.  Timeboxed iterations give feedback in a few weeks; unit tests give feedback in minutes; and paired programmers give feedback in seconds.  Research suggests this feedback doesn’t just improve the program; it also improves the programmers.

"You can’t have the same team do the same project twice; a bunch of stuff changes the second time around. You can’t have 2 teams do the same project; it’s too hard to control all the variables, and it’s prohibitively expensive to try it in any case. The same team doing 2 different projects in a row isn’t an experiment either.

About the best you can do is gather statistical data across a lot of teams doing a lot of projects, and try to identify similarities, and perform some regressions, and hope you find some meaningful correlations."

That’s exactly how Crystal Clear came about.  Alistair Cockburn spent a lot of time studying successful teams, looking for the similarities, then he documented them.  That’s not the kind of half-baked fad which Steve criticises, it’s about as scientific as it gets (in our field).  In fact, it was scientific enough to earn Alistair his Ph.D.  His doctoral dissertation is on-line and makes suprisingly easy reading.  I encourage you to read it, to see how Alistair grappled with the very problems Steve describes. (Eventually, he solved them much more successfully than Steve would expect. He documented his results as Crystal Clear).

Steve makes some very valid criticisms of the Agile movement.   However, his assertion of a lack of scientific rigour is incorrect (in Alistair’s case, at least).  Alistair conducted exactly the kind of research which Steve calls for, without any pre-conceived notions of what the outcome should be (he started years before the agile movement was founded).  He came up with a simple, effective process – which happens to be agile.

Interestingly, Alistair’s research-based process is somewhat different from XP.  As I’ve said before, XP is a subset of agile, not a synonym.

For instance: if the customer says they want a detailed Gantt chart, that’s a position. (It’s a position which is not very compatible with most agile processes).  But their interest is in understanding the status of the project. A chart like this might meet their interest equally well, while simultaneously meeting your interest in agility.

Customers also have a legitimate interest in protecting themselves from large cost increases and from large scope cuts.  Many agile teams unnecessarily adopt an incompatible position.

This is why its so important to understand that Agile is not just XP.  You have a much wider toolbox to draw on than just the practices in Extreme Programming.  Be prepared to draw on the whole toolbox, and your own creativity, to find the process that best suits your customer.  And be prepared to not use certain tools on certain projects.  For instance, this XP team (pdf) did not use pair programming; they replaced it with an alternative.

As agile developers, we know that good software is about customer-valued features. So is good process.

By the way, if XP is the only form of agile that you know, this page probably won’t make any sense. Am I really saying that TDD is not part of the core definition of agile? Yes I am. Look for TDD in the Agile Manifesto. It’s not there. While TDD is a good thing, and I like it, it’s not the core essence of agility. You’re welcome to define XP in prescriptive terms, mandating pairing and TDD, but those same prescriptive terms do not describe the agile movement as a whole. It is, after all, about “people and their interactions; over processes and tools“.

Defining agility by describing XP is like defining democracy by describing America. Such a “definition” obscures the underlying concept with details of the chosen example.

Agility is not defined by TDD, just as democracy is not defined by having a President. Democracy is defined as “choosing leaders in competitive elections”. It makes no difference whether you have a federal system or not; whether the executive is or isn’t separate from the legislature; or which method you use to count the votes – it’s still democracy.

Do we have a similar definition of agile development? One which is uncorrupted by specific examples and which defines the core underlying concept? In many respects, we don’t. The agile manifesto is the founding document of the agile movement (and the source of the term “agile development”). It’s a great definition of what agile developers think; but it doesn’t concisely define what they do. In other words, it talks about values and principles but not about practices.

Here’s my soundbite-size definition of agile:

The agile movement asserts that software development works best under:
Simple, iterative processes, which empahsise creativity and collaboration.

Other definitions include this one from Scott Ambler and two really nice ones from Alistair Cockburn, here.

Notes:

I use the word “creativity” to mean “inventiveness” – i.e. “producing solutions to difficult problems” (not just artistic ones). The most difficult problems are those involving (i) lack of clarity of the situation [e.g. requirements], (ii) multiple goals, (iii) complexity and (iv) time considerations – sounds like software!

The Crystal Clear book, by Alistair Cockburn, is now available.  The preface gives an excellent overview and this sample chapter is also available on line.  (You’ll have to buy the book to read the rest :-)

Briefly

Crystal Clear is an agile methodology for projects with small teams, less than about 10 people in size.

Team size is important, because if you have a small team it’s easiest to adopt a methodology that already fits.  It is more difficult to start with a larger one and prune it back.

Why Crystal Clear?>

Several aspects of Crystal Clear impress me:

• It’s based on observations of many successful teams.  Those observations give Crystal a sounder base than most competing methodologies, which typically trace their roots back to a smaller number of projects. For example, XP was “born” on the C3 project.
• It has a different slant on things.  We’ve come to expect a certain “mechanical” feel to processes, “Do A, B and C, and the result will be D, where D is great software!”.  We’ve come to expect methodologies to contain details such as diagramming notations and workflows.  And yet, somehow, that’s not what matters most.  Crystal Clear describes the things that really do matter, the things that make the most difference to the success or failure of a project.
• It supports fixed price contracts.  Whether you love them or hate them, you probably use them.  Crystal Clear is flexible enough to suit.
• It’s not “all or nothing”.  Crystal Clear will be useful to the people who adopt it, but it will also be useful to people who don’t adopt it.  At least, not all of it. Teams using RUP, XP or other methodologies can “borrow” useful techniques from Crystal Clear.  Likewise, if you are doing Crystal Clear, you can borrow good bits from other methodologies.
• It gives you useful guidance on how to adopt the methodology, and how to tailor it to your organisation.  I find this a stark contrast with RUP, which says “tailor me”, but doesn’t seem to give enough guidance on exactly how to do so.
• It “feels right”, because it has a lot in common with the successful projects I’ve worked on in the past.  This is not surprising, considering it’s based on observations of successful projects!
• Crystal Clear is just a book (and a growing community of users).  If you want to adopt Crystal Clear, it will only cost you the price of a book, plus a little of your time (days, rather than months).

More Information

While the book is the best resource, you may also like to see:

• The author’s site (See this page in particular.  It describes how Crystal Clear relates to other “Crystal” methodologies, which are for larger teams)
• Crystal Clear discussion on Wiki Wiki web
• My “Methodology Map“
• Martin Fowler’s comments about Crystal

Summary

While no methodology is a magic formula for success, if you’re involved in small-team software development, you should read this book.  Crystal Clear is not mutually exclusive with other methodologies, so you’ll find its insights valuable no matter how your team is organised.

Amazon’s page for the book is here.

Iterative vs Waterfall

T: Serial (“waterfall”) development is best (or at least normal/acceptable)
A: Always use iterative development

Workflow

T: Design, Build and Test follow each other, in that order
A: Design, Build and Test influence each other, so they should be concurrent and iterative

Requirements Quality

To ensure requirements are well-defined and well-understood:

T: document them thoroughly; users see the system when finished
A: document them concisely; users see the system early and often

Knowledge Transfer

To ensure everyone understands the system:

T: Place importance on written documentation
A: Place importance on frequent verbal communication; regular working releases; and clear, readable source code

Team Composition

T: Team members should be specialists (BA, Designer, Developer, Tester etc)
A: Team members should be generalists (each with their own particular strengths)

Team Size

A successful process is one that:

T: Organises a large team
A: Gets the same output from a small team

Planning

T: Make a Gantt Chart at the start; maintain it during the project
A: Make a Task/Feature List at the start; maintain it during the project

Monitoring

Measure progress:

T: against the Plan
A: against the Task/Feature List (tracking which ones are completed). But don’t just monitor your progress; also monitor the quality of that progress by testing and releasing regularly.

Managing Change

It’s hard to change software that’s already written, so:

T: Control or minimise change
A: Make change easier

Predicting Completion

Predict final completion date and cost using:

T: the updated Gantt Chart
A: the team’s progress to date

Process Definition

To help people successfully follow it, the development process should be:

T: thoroughly documented
A: concise and memorable

Process Improvement

Process improvement is best driven by:

T: metrics
A: the people involved

Philosophy

T: Software development is a construction process, so making detailed plans is advisable.
A: Software development is a creative process, so making detailed plans is unrealistic.

Instead, agile processes make less-detailed plans and manage risk through iteration, continuous quality control, and feedback.

Success is

T: Meeting the initial predictions of cost and schedule
A: Delivering value for money

1. The “toolbox” of iterative development techniques
2. The belief that people are more significant than process
3. Trust

David points out that trust underlies the greater efficiency of agile approaches. Lean Software Development also emphasises trust.

It is easy to fall into the trap of seeing trust as a prerequisite of agility.  (I used to think that way myself.)  Seeing trust as a prerequisite is both inaccurate and counter-productive. At the very start of a business relationship, trust is naturally absent.

If trust is absent, what do you do?  You can’t build trust with PowerPoint; you have to build it with collaboration.  So we need contracts that let us start collaborating before trust is fully established.  Then, as we begin collaborating, we’ll use agile practices to build trust.

We can build trust with:

• Visibility -  making visible progress early and often
• Honesty.
• Competence.

Visibility, honesty and competence reinforce each other. Competence won’t build trust unless it’s made visible.  Honesty is your only choice when progress is visible (and in itself, honesty builds trust).  Honesty is easy when it’s backed by competence (and rather awkward when it’s not!)

Finally, from day one, we need to harness the power of principled negotiation. Whether we’re trying to win the contract or simplify a complex requirement, we need the skills and mindset to seek fair and beneficial solutions for everyone.

I’m writing this guide to explain the foundations of agility, outline its benefits, and show why it’s not so crazy after all.

A Brief Definition

Agile development = “pragmatic iterative development”. If you’re agile, you produce software in small stages (i.e. you break each project into a series of mini-projects) and you use practical techniques pioneered by other successful teams. That’s a very simple definition of agile development. [For more thoughts on the surprisingly difficult task of defining agile development, see this post]

Many processes fit that definition. Extreme Programming (XP) is one of them; the others include Feature Driven Development, Crystal Clear and Scrum. Each process is different. For instance, up-front design is encouraged by Feature Driven Development but it’s discouraged by XP; pair programming is required by XP but other processes say it’s optional or don’t mention it.

The most important common thread, uniting all agile process, is their commitment to pragmatic iterative development. Other common elements include flexibility, concurrent workflows and rapid feedback.

History of Agility

In the mid 90’s, researchers documented a major company’s development process. The process bore the hallmarks of what we now call “agility”: it was iterative, encouraged individual creativity, allowed lightweight to non-existent design documents, and included frequent automated tests of the evolving software. Can you guess the company?

It was Microsoft. [1]

The Microsoft document, which you’ll find amongst the references listed below, illustrates two important points:

• Agile techniques are older than the “agile” buzzword. For instance, iterative development dates back to the 1950s, was used on the space shuttle, and is now endorsed by the US Department of Defense. [2]
• Agile techniques have proven their worth in successful teams. Microsoft is only one example. When Dr Alistair Cockburn set out to study successful teams in the mid 1990’s, he discovered similar techniques all around the world. Ironically, many teams apologized for not following traditional processes. Only when team after team made the same apology – all of them working on successful projects – did he realise that no apology was necessary. [3,7]

The techniques we now call “agile” evolved over several decades. They are popular because they work.

Benefits of Agility

When Dr Cockburn studied successful teams, he found more than just the widespread use of agile techniques. He also discovered that agility saves time. Dr Cockburn estimates that agile techniques reduce costs by at least 30%. [4]

When considering such big savings, it’s only natural to expect a corresponding drop in quality. Interestingly, that’s not what happens; agile processes enhance speed and quality.

Exactly the same thing happened when Japanese car makers adopted “lean manufacturing” in the 1980s – their cars became cheaper and better. Authors Tom and Mary Poppendieck report many parallels between the Japanese auto industry and agile development. For instance, both rely on rapid feedback, continuous quality control, and concurrent workflows (instead of the serial workflow we call “waterfall). [5]

I’d like to draw your attention to two points in particular:

1. Agile processes “debug” requirements. There’s more to quality than merely making the software conform to the requirements; you must also make the requirements conform to the users. Agile processes create software in small steps, getting rapid feedback from users at every step. Seeking regular feedback amounts to “debugging” the requirements – testing to make sure they really meet the users’ needs and correcting any deficiencies found. Traditional “waterfall” processes gather requirements carefully, but offer few tools to verify their correctness.
2. Agile processes encourage continuous quality control. They test throughout the project and fix bugs promptly. (They never defer testing and fixing to the end of the project.) Imagine taking the agile approach to its logical conclusion: we’d test so regularly that our software would always be fully tested; we’d fix all bugs immediately. With such a regime, the bug count would hover near zero for the entire project. This is a rather extreme approach – in fact, it’s the exact approach used in Extreme Programming.

Other agile processes are similar but not so strict. Typically you’re allowed a few unfixed bugs but you still have to produce a high-quality release at the end of each month.

By debugging requirements and continuously monitoring quality, agile processes detect problems early and fix them before they grow. The net effect is lower risk and higher certainty of project outcome.

Visible Progress

The agile approach to quality control has an important side effect: it demonstrates the project’s status. Because tested software is released at regular intervals, the team’s progress (or lack thereof!) is clearly displayed. You never have that awkward situation we see on traditional projects: the developers claim to be “70% complete” but no-one’s sure how many bugs there are or how long it will take to fix them. In fact, no-one’s sure whether the developers really are 70% complete; perhaps there’s another 70% which they haven’t even started! :-)

On an agile project, “70% complete” is a much more precise statement. It means 70% of the planned features are debugged, tested and could go live tomorrow (if we wanted). We can even fire up the latest release to see the features with our own eyes. We can’t kid ourselves (or our customers) with overly-optimistic progress reports. [6]

Fixed Price Contracts

A fixed price project is a great time for agile development, since under a fixed budget you need to work quickly without sacrificing quality, and that’s exactly what agile processes let you do.

Unfortunately, there’s been some confusion about agility under fixed price contracts. (I’m talking about contracts with fixed price and scope). Advocates of Extreme Programming (XP) say you can’t use agile processes on fixed price projects. While that may be true for XP, it’s not true for agile processes in general. Take Crystal Clear for example, it’s an agile process that was developed on fixed price projects. Crystal Clear supports changing requirements, just like XP, but it also supports the other alternative: fix the requirements at the start and don’t change them later.[7]

Conclusion

Agile development is good software engineering – different, but good.

Agile techniques have proven themselves in successful teams around the world. They save money, enhance quality, and demonstrate progress with the ultimate status report – working software.

References[ 1 ] How Microsoft Builds Software, M.A. Cusumano and R.W. Selby. (Costs US$10 unless you’re a member of the ACM) Cusumano and Selby studied Microsoft from 1993 to mid-1995, 6 years before the official “birth” of the agile movement.[ 2 ] Iterative and Incremental Development: A Brief History, Craig Larman and Victor R. Basili. (Costs US$19 unless you’re a member of the IEEE)

In particular, it is assumed that people will change their habits to follow detailed new instructions.  Unfortunately, humans aren’t very good at that!  Old habits die hard and, unlike the computers we program, we don’t reliably follow written instructions.

Perhaps this is why so many process improvement initiatives fizzle out (especially big ones, like adopting RUP).  The new process may be diligently used when first adopted, but then it falls into disuse as people revert to their old habits.

Agile processes are different.  Instead of basing themselves on something humans do badly, agile processes leverage human strengths.  These strengths include the ability to communicate, collaborate, and use initiative.

Agile processes add structure to these natural strengths.  The structure includes tools, techniques and disciples that have proven themselves in real-world software development.