Read on, and judge for yourself whether I’ve succeeded…

Example

Consider this example adapted from Wikipedia: An entomologist finds a beetle with spots on its back. She thinks it might be a rare subspecies, because 98% of the rare subspecies have such spots, but only 5% of the common variety have them.  The subspecies makes up only 0.1% of the population. Given the fact that the beetle in her hand is spotty, what is the probability that it is also rare?

The entomologist wants to know P(rare|spotty), which reads from left to right as “probability of being rare, given that it’s spotty”.

This is the time to use Bayes’ Theorem.  She wants P(rare|spotty) but only has P(spotty|rare).  The latter is 98%, but she suspects that the former may be much lower.  She’s right, the spotted beetle in her hand is probably not rare… and Bayes’ can tell us why.

Memorable

I want to explain Bayes’ Theorem in a way that is easy to remember.  To begin, let’s start from scratch and think about the probability that one randomly-selected beetle is both rare and spotty.  There are two mathematically-correct ways to express that probability.  One is to write:

P(rare and spotty) = P(spotty|rare) P(rare)       
= “(the probability of it being spotty, given that it is rare) times (the probability of it being rare in the first place)

The other is:

P(rare and spotty) = P(rare|spotty) P(spotty)  
=  “(the probability of it being rare, given that it is spotty) times (the probability of it being spotty in the first place)”

We’ve just written down two different ways to compute the same thing.  So they must be equal to each other. I.e.:

P(spotty|rare) P(rare)  = P(rare|spotty) P(spotty)

If we re-express this in a general-purpose notation, we get

P(A|B) P(B) = P(B|A) P(A)

That is the formula you need to remember.  To recall its left-hand side, remember this intuitive fact: the probability of two events both happening is the (conditional) probability of one happening given that the other already has happened, times the (un-conditional) probability of the other.     The right-hand side is just the same thing round the other way.

That’s all you need to remember.  Start with the above formula, then use basic algebra to re-arrange it into the form you want.  For instance, if you want to compute P(A|B), just divide both sides by P(B).  (You’ll end up with the usual form of the Bayes’ equation, as seen in textbooks.)

Intuitive

Where Bayes gets interesting is when its used to assess the probability of some hypothesis, given the available evidence.    Dividing both sides of our formula by P(B), and renaming  A as the “hypothesis” and B as the “evidence”, we get this:

P(hypothesis|evidence) = P(evidence|hypothesis)   P(hypothesis)
                                          —————————
                                          P(evidence)

Let’s consider a simple example involving a blond-haired young boy called Johnny, and an apple that’ is missing from the neighbour’s tree.  The hypothesis in question is “Little Johnny stole the apple” and the evidence is “blond hair left snagged in apple tree”.   The angry neighbour wants to know the probability that Johnny did indeed steal the apple, given the evidence of the hair – i.e.  P(Johnny-stole-apple|blond-hair-left-at-scene).  Our formula becomes

P(Johnny-stole-apple|blond-hair-left-at-scene) =

     P(blond-hair-left-at-scene|Johnny-stole-apple)  P(Johnny-stole-apple)
     ———————————————————– 
      P(blond-hair-left-at-scene)

When used is this way, the formula includes three things:

• the left-hand-side, P(Johnny-stole-apple|blond-hair-left-at-scene), represents our degree of belief that Johnny stole the apple, after taking into account the evidence of the blond hair.  This is called the posterior probability.
• the term on the far right, P(Johnny-stole-apple), represents our degree of belief that Johnny stole the apple, before taking into account the evidence of the blond hair.  This is called the prior probability. 
• The rest (i.e. middle) of the equation represents how the evidence of the blond hair influences the prior belief, by making it stronger or weaker.

Let’s consider each of the RHS terms, and see how they are relevant to the outcome.  Firstly, consider P(blond-hair-left-at-scene|Johnny-stole-apple).  For instance, if the neighbour knew that Johnny suffered from excessive hair loss, then this probability would go up.  I.e. if he was particularly likely to lose hair, then the presence of hair would be stronger evidence of his guilt (as opposed to someone else’s guilt).  Conversely, if Johnny always wore hats, then the presence of a hair would be weaker evidence of his guilt.

Secondly, let’s consider P(blond-hair-left-at-scene).  This is the overall probability of a blond hair left at the scene, regardless of who committed the crime.  If this number goes up, then the hair is considered weaker evidence of guilt. For instance, if the crime occurred in a Swedish town where most boys have blond hair, then a blond hair would be relatively weak evidence against Johnny.  It could have come from almost anyone.  But if the crime occurred in China, the probability of a blond hair being left at the scene would be much lower, so the fact that one was found is stronger evidence against little blond Johnny.

Finally, let’s consider P(Johnny-stole-apple).  This is the neighbour’s prior belief that Johnny stole the apple – i.e. his belief before considering this particular evidence.  If the neighbour knows that Johnny is a caring, honest, Boy Scout with an allergy to apples, then his prior belief in Johnny’s guilt will be relatively low.  To reverse that prior belief, and produce a high posterior probability of guilt, the evidence of the hair would have to be particularly compelling.  (E.g. Johnny is the only blond-haired boy in a Chinese village, and he suffers from persistent hair loss).   If the neighbour doesn’t know Johnny at all, then he has no option but to assign prior probability based on the population in general – e.g. if there are 10 kids in the neighbourhood, the prior probability of Johnny being the thief could be assumed to be 1/10th.

So we can see that all terms of the right hand side contribute to the final result in ways that are intuitively sensible.  Bayes’ is not so hard.  It actually makes intuitive sense.

What’s so difficult then?

In practice, there can be some obstacles to applying Bayes’ successfully. Obstacles such as:

1. Remembering that P(hypothesis) is an input to computing P(hypothesis|evidence).  If you use the formula, you won’t forget. But when we rely on our intuitions instead of the formula, we tend to neglect P(hypothesis).  This mistake is known as base rate neglect.  For instance, in the spotted beetle example, P(rare) is the base rate – and it’s only only 0.1%.  If you neglect the base rate, you’ll intuitively over estimate the chances of a spotted beetle being rare.
2. Being willing to use P(hypothesis) as an input.  Having to rely on P(hypothesis) may seem inconvenient because its often hard to obtain an objectively-correct value for
   P(hypothesis).  So, out of necessity, we may end up using subjective guesses for P(hypothesis).  This subjectivity is the cause of much of the controversy that dogged Bayes’ Theorem in the 20th century.  The solution is that, for many kinds of problems, “probability” actually means “measure of belief” – rather than some number which is computed from repeatable trials.  [Subjective evaluation of P(hypothesis) is also the cause of much of the power of Bayes, since the formula doesn’t mind if P(hypothesis) is subjective.  Therefore, it’s a great way to take an initial viewpoint, which may be subjective, and adjust it as we receive new objective evidence.]
3. Computational difficulties when generalizing beyond simple yes/no hypotheses and one piece of evidence.  For instance, you may want to consider several events (blond hair left at scene and passer-by reports glimpsing possible thief in pink tracksuit), or you may have random variables instead of boolean events.  These issues are entirely solvable, but beyond the scope of this blog post.

Bayes’ Theorem gives us a powerful tool for dealing with uncertainty.  I hope this page will help you (and me) recall the basics ;-)

Finally, for an enjoyable read on the history and uses of Bayes’ Theorem, check out this new book from Sharon Bertsch McGrayne.

Postscript: feedback about this post included the following comment, which is worth quoting here in full: “fundamentally the idea actually is very simple and the interpretation and application where all the difficulty lies.”   That seems like point worth noting: that the basics of Bayes are very simple, but successfully applying it to real world problems is much more difficult.  I hope that this blog post has helped to convey the simplicity of the underlying idea.  I wrote this post while reading the above book, since I couldn’t enjoy the book without understanding what Bayes’ Theorem really is :-)   As for the difficult problems of devising real-world models and hypotheses, I’ll have to plead (relative) ignorance I’m afraid – since I’m a software designer not a statistician.

So here are some of the key points that now I keep in mind when considering estimation, pricing and contracts for agile projects.  These points are particularly relevant when customers have asked for fixed price and fixed scope.

1. It’s a valid thing to ask for…

It’s perfectly natural for customers to want some up-front insight into likely cost and scope.  In the spirit of good negotiation, we should take their concerns seriously.

2. … but many agile teams successfully dodge the issue.

I asked someone recently what they were doing about this, and their reply was that they don’t really find themselves doing many contracts with fixed price and fixed scope.  That certainly solves the problem ;-)  I suspect it depends on the kind of company you work for.  If you’re an in-house IT shop (i.e. customer and supplier are the same organisation) then I would imagine that there is little call to fix both scope and budget.  However if you work as I do, in a company that undertakes development projects for other organisations, then you’re more likely to be asked to fix both price and scope.  It also depends on the “purchasing culture” in the city where you work.  For instance I work in Wellington, where government procurement processes heavily influence the way people expect contracts to be framed.  But in Auckland, which is only an 8 hour drive away, there’s a very different procurement culture that’s dominated by private companies.

3. There are lots of different kinds of contracts which you might consider

Alistair Cockburn has a big list of suitable contract types here.

4. Flexible pricing is probably good

Rather than absolutely fixing both price and scope, it makes the project run better if you can allow at least one of them to “flex” in a controlled way.  This still gives the customer early insight into commitments and benefits,  but avoids the harsher downsides of absolutely fixing both.  Here’s an approach that I’ve tried and liked.  It’s not for everyone, but in cases where it’s understood and embraced by both parties, I do believe its better than fixing everything.

5. Here are my favourite “ingredients for success”:

• Use a “discovery phase”.  This is a timeboxed phase at the start of the project where you learn more about what the customer needs, and prepare an itemized breakdown of functionality.  That itemized breakdown is the scope that you then  commit to.  A discovery phase is good for both parties, because it results in much better-informed scoping than projects which attempt to fix the scope without such a phase.  The discovery phase should be paid, rather than delivered for free – both because the customer gets value out of it, and because adequate discovery is too big to do for free.  In terms of sizing the discovery phase, the Feature Driven Development community uses a rule of thumb that this phase should last for 2 weeks for every 6 months of development (IIRC).  I think that’s about right.  For example, I used 3 weeks for a 6 month project once, but during those 3 weeks we also built a prototype.
• The bigger the project, the more coarse-grained your discovery phase must be. (Within a realistic timeframe, you simply can’t drill down into as much detail as you might in the discovery phase for a smaller project.   And if you do, it turns into BDUF, which is counter-productive for all the well-known reasons.  So I’d suggest making do with a coarse-grained breakdown on large projects.)
• The itemized breakdown of scope should be tabular rather than prose (i.e. a table rather than a document). This is a lesson I learned the hard way – the document was too ambiguous.  One good way to organise the tabular breakdown is with a story map.
• Ideally, that itemized breakdown of scope will actually serve three purposes: it will define the scope, it will form the basis of your estimate, and it will be what you track your progress against as you work through the project.  IMHO, if you’re not using the same list for all three purposes, you’re risking trouble.
• Typically, your cost estimate will be based on summing up points for every feature that’s in scope.  Make sure you add a “buffer” of additional budget over and above that.  There are two reasons why you might need such a buffer: (a) the inherent uncertainty in estimation and (ii) the need to allow time for for Validation (i.e. ensuring fitness for purpose, rather than merely compliance to spec). 
• Adding this buffer will typically produce an estimate which feels too high. So my rule of thumb is as follows, “if your estimate feels about right, its probably not.” ;-)  Add the buffer!
• A buffer that adequately accommodates both estimation uncertainty and validation effort tends to unfeasibly large. (Unfeasible in the sense that some manager somewhere won’t accept it ;-)  So I prefer to use a target cost contract.  Some of the necessary buffering is built into the target in the target cost contract, but the rest is provided by the (controlled) flexibility.
• Know how you’ll draw the line between “this is an addition to scope, and therefore requires a budget increase”, versus “this is something we should just do, and therefore requires no change in budget”.   You always need at least some of the latter, certainly for Validation and probably also for convenience (its too much paperwork for formally increase the budget for every tiny change).  I’ve outlined one suggestion for drawing this line, part way down the target cost page.
• It’s not enough for you  to know how you’ll draw that line.  Your customer must understand too.  And agree.  This applies to both management and subject matter experts, to both ‘gold owners’ and ‘goal donors’.  All must understand what kinds of changes will affect the budget and what will not.
• Use a project tracking and management tool that gives you a “whole of scope” view.  If your tool only shows the current iteration or release, but your commitments are for the whole-of-scope, then you’re flying blind.

6. “If I do all that, will it work?”

Maybe. Maybe not ;-)   Fixing price and scope of agile projects is fiendishly difficult.   But there is a genuine business need for it so we, as agile practitioners, owe it to our customers to continue trying (and learning).

7. Understand Estimation

When I finally started studying estimation, I was shocked to discover how little I, and my colleagues, really knew about it.  We’d been getting by for years with little more than a grab-bag of misconceptions.  And we didn’t even realise.  So read Steve McConnell’s Software Estimation: Demystifying the Black Art.  I haven’t found any other book that combines all the necessary material into one readable volume. 

8. Understand the bit that Steve omitted from his book

If you’re bidding for work, in a competitive situation, then you’ve got to read this post from Steve McConnell (including the comments) and my follow up here.  It’s not entirely comforting, in that it describes a problem which software vendors can’t easily solve on their own. (The obvious solutions require engagement from purchasers.)  On a more positive note, the US engineering profession did successfully solve this problem. 

I hope to eventually blog more about this in years to come, particularly some thoughts on what we can do in the absence of a purchaser-side solution.

9. See what others are doing.

A great example is Atomic Object in Michigan.  Here’s a post on their corporate blog to get you started.  (You’ll notice they also like to introduce controlled flexibility. In Atomic’s case, they flex the scope rather than the price.)  You should also check out their  whole series on estimation.

Good luck.

PS. Is there a tool that can help? 

Mostly its not about tools, but since you asked ;-) ….  I think several things help: taking a whole-of-scope view; using story mapping; using the same artefacts as the basis for scoping, estimating and tracking; and keeping the user story/feature descriptions concise in order to prevent BDUF.  I don’t know of any one single tool that meets all those criteria – so I started writing one.  It’s not finished yet, but with a little creativity there are many other ways to get by without an integrated tool. 

All in all, this is not really in tools problem.  Its really about understanding, attitude, negotiation and communication.

Here’s a wee table I put together, to help with visualising large numbers.  The table is based on the thickness of a New Zealand $1 coin.

• We start with just one, which is 2.74mm thick (about twice as thick as a US quarter).
• Then, imagine we make a stack of 1000 coins.  It would be about the height of a person (a tall person, with their arms up-stretched).
• For a million coins, we would have to lie the stack on its side.  It would form a “sausage” of coins about the length of an airport runway.
• For a billion coins, the sausage would be as long as a country (specifically, India).
• For a trillion coins, we’d end up way out in space (almost 8 times as far away as the moon).  That’s hard to visualise, because there’s nothing there.  So how about we try $14 trillion instead (the size of the US public debt)?  If we had the US public debt in $1 coins, the stack would reach all the way to Venus.

So to visualise the difference between a thousand, a million and a billion, you can say: “person, runaway, India”.

Footnotes Nov 2011:

1. Here’s an amazing visualisation of dollar values at various scales

2. Further to the billion-coins-in-India example: what if we lined up a billion people, instead of a billion coins?  That’s approximately the actual population of India.   To get a line the same length as the  billion coins, we’d have to ask the people to stand about 500-abreast.  I.e. the population of India could stretch from the north to the south of the country, if they formed a long, skinny loosely-packed crowd approximately 500m wide.

3. A similar crowd in New Zealand, with the entire population of 4 million stretching the full length of the country, would be only 2 or 3 people wide – in part because the country is sparsely populated, and in part because it’s a long skinny country.

Jan 2012:

4. The number of stars in the observable universe is estimated at 7 x 10²².   Given the world population, that’s 10 trillion stars per person – which equates to one stack of coins, reaching most of the way to Venus, for every person alive today.

It turns out the concept has a name, “The Winner’s Curse”, and it’s been known to economists for over 200 years!  

I’ve updated my original post with details and a link to an excellent article from the Simula Research Lab in Norway.

The goals of this approach are to:

• Share risk fairly between Customer and Supplier
• Give the Supplier the peace of mind of being protected from significant cost overruns
• Offer enough flexibility to get the best out of an agile development process
• Align goals by giving both parties an incentive to minimise scope

The ideas here are still under development, having only been used on a small number of projects so far.

Overview

The contract is based on two figures: the “target price” and the “cap”.  The cap is the maximum that you (the Customer) will pay. The target is lower than the cap and the contract gives both parties a financial incentive to meet the target.

If we (the Supplier) come in under the target, the savings are shared equally between both parties.  Likewise, if we come in over the target, the extra cost is shared evenly – but only up to the cap.  If we reach the cap, it acts like a fixed price.

Benefits

This approach gives you certainty about the maximum that you will pay (the cap), while still sharing risks and benefits fairly.

As explained below, both parties have a financial incentive to minimize the scope and cost.  Minimizing scope and cost reduces overall project risk.  Contrast this with standard fixed-price or time and materials contracts, where one party has an incentive to increase scope and the other has an incentive to decrease it.  Only a target cost contract incentivizes both parties in the same direction.

Details

There are two ways do formulate a target cost contract.  Both are mathematically identical, they are just different ways to describe the same thing.

Formulation 1:

1. Up to the target cost, you pay our standard rate of $x per hour.
2. If the project comes in under the target, we split the savings evenly.  For instance, if we come in $10k under budget, you pay us half of that (5k) and keep the other half yourself.
3. If the project goes over the target, we split the extra cost evenly. In effect, that means you pay half our standard rate ($x / 2) for any hours between the target and the cap.
4. Your total payment (for full-rate work up to the target plus any half-rate work beyond) will not exceed the cap.

Formulation 2:

1. Payment is composed of two elements.  You pay half the target as a fixed fee.  You also pay us on an hourly basis, as we work, at half our standard hourly rate.  For instance, if the target is $100k, you’ll pay a fixed fee of $50k plus half our standard rate ($x / 2) for every hour that we work.
2. Again, your total payment (fixed portion plus hourly portion) will not exceed the cap.
3. The fixed portion can be paid in several instalments, with timing negotiated up front .  The last instalment will typically be on completion of User Acceptance Testing.  For small to medium projects, two instalments is the easiest approach – one at the start and one at the end of UAT.  So for a project with a target cost of 100k, the two fixed payments would be 25k at the start of the project and 25k at the end of UAT (giving a total fixed payment of half the target, as always).
4. The per-hour portion will typically be invoiced on a month-by-month basis.

Graph

This graph shows how the contract works.  (Both formulations result in the same graph, because they are just different ways to describe the same thing).

Note that if the project completes exactly on target, working exactly the target number of hours, then total billing is exactly equal to what it would have been under a Time and Materials (T&M) contract.

Also note that there is no hard and fast rule for the gap between target and cap – it depends on the degree of risk in the project, with the gap being bigger if there is more risk.  One guideline is that the gap should be big enough that neither the supplier nor the customer expects to hit the cap.  (Since, if the cap is expected to be hit, the situation degenerates into a fixed-price contract.  When that happens, the beneficial effect of aligned incentives is lost).

Incentives

Under a target cost contract the customer has an incentive to minimize scope, because that minimizes the total cost.  The supplier also has an incentive to minimize scope, because under this model lower scope means higher profit.  (The supplier maximizes profit by minimizing the number of half-price hours they must work.)

Sample Wording

Here is sample wording for a project with a target of 100k and a cap of 115k. The wording is based on formulation #2.  Bold sections may vary for projects of other sizes or at different charge-out rates.

Target Cost: $ 100,000

Payments will be as follows:

$25,000 at the start of the development and $25,000 on completion UAT

Monthly payment of actual hours worked at $xx/hr until the development is complete or yyy hours of effort has been expended.  [yyy * xx = 65k.  That 65k plus the 50k in fixed payments = the cap of 115k]

Total payment of ($25,000 x 2 + $xx per hour) will not exceed the cap of $115,000.

In the event that the capped price is reached <supplier> will complete the development without billing <customer> for any further hours (other than agreed change controls).

Change Controls

The contract is based on an initial agreed scope.  If you want to add to that scope during the project, you can do so using an agreed change control process.  (Just like you can under a fixed-price contract.)  In such cases we will negotiate an appropriate change to the target and the cap.

We recommend that the following approach should be taken to change control:  if it is a brand new requirement, a change control must be raised.  However, if it is simply a question interpretation of an existing requirement, no change control is necessary.  The cap and target are not changed by issues of interpretation; only by brand new requirements.  We make this recommendation to preserve the flexibility and collaboration that is offered by agile processes.

Conditions Under Which this Approach Can be Used

Note: this approach can only be used where all of the following are true:

• The up-front scope can be identified and written down well enough to serve as the basis for any future discussions of what is, and isn’t, a change.  (The scope definition doesn’t have to be perfect, just good enough to be used as a baseline for telling the difference between what does, and doesn’t, require a change control)
• All parties agree that if changes are identified then either the cap will increase or something else must be removed from scope.
• The key stakeholders in both organizations truly understand the model.  (Don’t want nasty surprises where, for instance, customer is surprised to receive final fixed payment because they thought they were just paying by the hour!)
• Key stakeholders in both organizations know how they will process the payments through their respective billing/financial systems (since such systems tend to assume either fixed price OR hourly rate, but not both).

Ideally, the upfront scope will be accurate enough to serve as basis for future discussions, but loose enough leave room for refining the details, agile-style, during the project.  The level of detail depends on the organizations concerned, and the risk level of the project.  In some cases the level of detail required for this approach may be excessive, leading to too much Big Design Up Front, in which case this contract structure is probably inappropriate. In such cases, other contract structures should be considered instead.

We do not mandate any particular way to document the initial scope, as long as it meets the needs above.  E.g. index cards or other simple formats may be appropriate, depending on the situation.

Payment for Change Controls

The following approach is suggested:

Firstly, the size of the change is estimated by the development team.  Then, based on that estimate, two things happen:

1.     The capped number of hours is increased.  This is the figure yyy in the example above. The increase is equal to the estimated hours to implement the change, plus an appropriate contingency buffer.  AND

2.     An additional fixed payment is made.  This fixed payment corresponds to half the target cost of the change. For a small change this can be a single payment at a time agreeable to both parties.  For larger changes it can be broken into instalments, with the final instalment being due at the end of UAT.

The net effect is as shown in this graph:

[Update June 09: a similar approach is described here]

I have a theory that, when multiple suppliers are competing for the same contracts, market forces encourage selection of those suppliers who have under estimated (either knowingly or unknowingly).  It’s some kind of “reverse natural selection”, where the market tends to select the least fit proposals (those that are most underestimated).

“…market forces systematically bias the winning bid [towards under-estimation], even when the set of all bids is not biased at all…”

[Update March 2009:  I have found the name for this phenomenon.  It is an effect well known to economics, and first mentioned over 200 years ago!  It's called the Winner's Curse.  Very little has been written about it in software, although I did find this very good paper from Magne Jørgensen and Stein Grimstad from the Simula Research Laboratory in Norway.  The linked page has both text and PDF versions of the paper.  It's well worth reading.  As Jørgensen and Grimstad write, the winner's curse can easily become the customer's curse - so it is the interests of both parties to minimise it's effects!]

Steve McConnell and I discuss this at some length on his blog. He suggests that, while my logic is interesting, it cannot possibly be true — because if it was true all such suppliers would go out of business.  My reply is that my logic is still correct; they do struggle on projects with early fixed prices, but they stay afloat on revenue from other sources:

• One source is change requests and follow-on projects, which I believe can (and should) be done more ethically than this all-too-common-common model.
• Another source is competitive projects won by more realistic means.  Steve mentions things like two-phase contracts and delaying final fixing of price until later in the cone of uncertainly.   Ayende advocates competing on quality rather than price. I completely agree, and advocate some of the same ideas here and here.
• The final source is projects which simply never went through a competitive bid process.  There are many reasons why some projects do not go through competitive bid processes.  Such projects are a perfect opportunity for the customer to choose a supplier who they already know and like.

I believe that these three sources help to sustain many outsourcing companies.  If not for these three sources, things might get ugly.  I wrote:

“…selecting the lowest bid has been blamed for creating unsustainable markets in traditional (construction) engineering…

To me, the experience of the construction industry offers some indication that price-based selection can indeed skew the winning bids downwards, to the detriment of an entire industry.  If it is documented in traditional construction, perhaps it possible in IT too.  Perhaps it hurts all outsourcers…”

What do you think?  When prices must be fixed early, does a price-sensitive market make it “impossible” to win profitable work?

This approach is a natural fit for most agile processes.  However, other options should also be considered, because:

• Firstly, Alistair Cockburn points out that methodology and contract type are not tightly linked.  Like many people, I initially made the mistake of bundling “agile” together with “flexible contracts” in my mind.   He points out that the two issues are virtually independent.  His comments prompted me reconsider my own views, and to write about fixed price contracts here on this site.
• Secondly, I doubt that Cancel-After-Any-Phase, on its own, can satisfy customers’ concerns about cost overruns…

A Weakness in Cancel-After-Any-Phase?

Consider this scenario:

The project is about 25% complete.  It has become clear that the amount of work required will significantly exceed the initial estimates.  The customer can cancel after any phase, but will they really cancel at this stage just to get a “cheaper” supplier?  Will they carry on, and cut scope to fit the budget?  Can they really cut the scope that far, or do they have to find additional funding?  Is any additional funding possible?

Customers have a legitimate interest in protecting themselves from problems like this.  A good process will seek to meet that interest.

Conclusion

There’s a limit to scope cuts.  Cut too much and customers are bound to be disappointed.  Many customers can’t take that risk. So, it makes sense to look at other contractual options, such as Target Cost or Two-Phase Contracts.  These can be combined with Cancel-After-Any-Phase to offer better risk protection.

Finally, even the best contract in the world is no substitute for trust and collaboration.  That’s what we really need, and our contracts mustn’t stand in the way.

Background Reading

Martin Fowler, on fixing price but not scope.  This is Cancel-After-Any-Phase, but with a pre-arranged maximum expenditure.

By the way, he says you can’t fix both price and scope.  So why do I say you can?  Because Martin’s describing reality, and I’m describing contracts ;-)  Traditional contracts are not about reality, they’re about a plan, and about making the supplier responsible for the inevitable difference between the two. That’s OK, as long as you understand the downsides.  See this page for more discussion of Martin Fowler’s comments.

(Update March 2011:

1. Glen Alleman has posted a perspective here, on the inadequacy of cancel-after-any-phase.  While his kitchen remodelling example might be a bit waterfall-ish, I agree with his general premise that,  “How much will this cost” is a legitimate business question.  Personally, I think it is best if both parties can agree that any answer to that question should be just an approximation, but the question itself is a valid one.  It should be asked, and it should be answered.

2. Ron Jeffries is quoted here as saying “The value provided should be so visibly more than the cost as to make detailed cost accounting clearly a waste of time.”  I’d like to point out the difference between detail and accuracy. Yes, we don’t need task-by-task actual costs, but we do need an accurate grand total.  The latter can be obtained easily, from total hours worked by each team member per week (as Ron himself goes on to say).
As for the total value being so clearly more than the cost, I’m not sure that’s necessarily an adequate solution to the cost management problem.  Sure, if the value is approx $50m and the cost is approx $1m, then don’t worry too much about the cost.  But what if the value is approx $3m and the projected cost is $2m?  The gap between those two figures is large enough to make the project worth doing, from a business perspective, but not large enough to let anyone relax about cost tracking.  If you don’t manage cost and scope carefully, overrun could eat the projected benefits.)

Finally

No contract can fully protect you from a supplier who makes promises they can’t keep.  Therefore selecting a supplier must be about finding a competent partner not just a low bidder.

Like scope creep, effort creep is inevitable and manageable.  To manage effort creep we need to understand what causes it.  Three major causes are underestimation, over engineering, and discovering implicit requirements during system design.

Underestimation

I once helped convert a complex Windows application into a giant ActiveX control.  (Kids, don’t try this at home.) Our beta release had a huge memory leak.  Our memory management architecture worked well in stand-alone apps but broke down in ActiveX (it couldn’t cope with page re-loads).  As I recall, we leaked over 10MB per minute.

We never saw that coming.  It was what Robert Read calls an unk-unk, a problem so unforeseen we never even imagined it might exist.

I know what you’re thinking – we made a stupid mistake and we should have seen the problem coming.   Yes, perhaps we should have.  But we didn’t.  And that’s the way it is.  There’s always something unforeseen.

How do you plan for unk-unks, when you don’t know what they are? You simply have to include a bucket of time in the schedule to cover the things you haven’t thought of yet.  How big should the bucket be?  I know of no better answer than to look at past projects of similar size and complexity.  How much time did you spend on their unk-unks?

In the early phases of a project, there’s no such thing as perfect estimation.  You need to allow time in the schedule for unk-unks, and also for overrun on known tasks.

Over Engineering

Some developers tend to over-engineer solutions, building in extra complexity because “someone” might need it “later”.  I think we all have that tendency, to some degree.

XP addresses this issue with the slogan You Ain’t Gonna Need It.  Personally, I’d phrase it differently, although I do agree with the idea. I’m all for creating solid implementations of features that we know are needed; but I’m against speculative implementations of things that “might” be needed “later”.

It’s important to realise that this form of effort creep is completely different from underestimation.  If the customer asks for a Toyota Corolla, then the estimate should be for a Toyota Corolla.  It shouldn’t be for a giant SUV just in case we accidentally build one of those instead.

I don’t have a silver bullet to combat over engineering.  Two things seem to help: discussing ideas with other developers before you build them, and making the true project status visible to everyone.  Often developers don’t really know where the project stands in terms of progress and budget.  When you’ve got nothing but a few dates in a Gantt chart, it seems easy to justify extras that “might” be needed. It’s harder to justify them when everyone sees the full project status, including “budget burn”.   (You can also chart effort creep itself.)

Explosion of Implicit Requirements

When moving from requirements to design, there is an explosion of ‘derived requirements’ … caused by the complexity of the solution process.  The list of these [derived] requirements is often 50 times longer than the list of original requirements…. What may have seemed straightforward from a problem point of view explodes into something quite complex from a solution point of view. – Robert L. Glass, in Facts and Fallacies of Software Engineering

We unearth numerous details and special cases when we take a human-readable problem description, and turn it into a solution so precise that even a machine can execute it.  Were all those special cases (implicitly) included in the original scope?  Customers tend to say “Yes” while suppliers tend to say “No”.   In reality, some are effort creep and some are scope creep, while many others lie somewhere in between.

Fuzzy Grey Boundaries

There’s no clear line between scope creep and effort creep.  Implicit requirements are hard to classify; there’s no precise definition of over engineering; and even the best estimates are never perfect.  If we find ourselves doing something we hadn’t planned, is it:

Here’s what I think: In the very best projects there’s something special about the relationship between customer and supplier.  It’s a working relationship that fairly shares the costs of scope creep and effort creep, and avoids squabbling over which is which.

• Firstly, you have to give up certain agile techniques when working on fixed price projects.  For instance, you can’t use “just-in-time” requirements definition [1].  However, even though you have to give up some agile techniques, you don’t have to give up all of them.  You can still use iterative delivery, automated testing, close communication and most other agile techniques.
• Secondly, we must consider what a fixed price contract really is.  A fixed price contract is based on an estimate of the work required.  Customers don’t want fixed contracts because estimates are good; they want fixed contracts because estimates are bad.  The actual effort never matches the estimate, and fixed price contracts are simply a way of making the supplier responsible for the difference.

So, as long as the requirements are stable – stable enough [and sufficiently well-understood] for the supplier to accept responsibility for the cost difference — a fixed price contract is OK.   (Do suppliers accept such responsibility too willingly?  Do price-based procurement processes encourage them to do so?  How accurate can estimates actually be, when requirements are stable?  These are all excellent questions, which I hope to write about some day…)

As Alistair Cockburn suggests, the supplier can follow the fixed quote with an agile process.

Everything changes if the requirements are not stable.  As Martin Fowler suggests, changing requirements need agility and a flexible contract.

I’ve written more on this subject here.

There are many reasons.  To pick just one: we believe it works in “real” engineering – making bridges, roads and buildings.

But it doesn’t.  Although it is widely used in engineering; it’s not widely successful!

The Truth about Engineering

The latest issue of e.nz, published by New Zealand’s Institute of Professional Engineers, reports that “the indiscriminate urge for lowest price” leads to “late and unsatisfactory completion, disputes and litigation”.  (Almost sounds like software!)

The New Zealand Construction Industry Council says that “the lowest-bid approach is compromising design quality and integrity, health and safety, training, the environment… [Furthermore,] the lowest bid approach encourages unsustainable markets” [1].

Discussing solutions to these problems, e.nz explains that: “A substantial culture change is involved in progressing towards best practice in procurement.  All stakeholders must accept that they have a duty to collaborate towards a common objective: the full satisfaction of the project’s objectives, in exchange for fair rewards for all who contributed.” [2]

Why is the Lowest Price Dangerous?

The lowest price is dangerous because you don’t know why it’s low.  In the software industry, a company may offer a low quote for any of the following reasons:

1. They have misunderstood the difficulty of the task [3]. (See these posts too)
2. They understand the task; the estimate is low simply because software estimation is inherently difficult.
3. They understand the task, and are deliberately bidding low because they’re eager (or desperate) for your business.
4. They understand the task, and have outstanding skills and technology which will allow them to complete it quickly.

Only the last reason is a good one.  The others, to greater or lesser degrees, may all threaten your project.  Yet in terms of price, they look the same.

So How Do You Tell the Difference?

You have to base your decision on assessments of the supplier’s capability: the quality of their staff, the quality of their technology, and their track record.

Price tells you nothing about capability. A low price may signal superior capability (reason 4), inferior capability (reason 1), desperation (reason 3), or nothing (reason 2).

A Better “Real” Engineering Practice to Emulate

If we want to emulate “real” engineering, we would be better to take our lead from the US federal government. Since 1972, price-based selection for engineering services has been prohibited for federal government agencies, with selection being based on supplier quality instead [4].  The approach has proved highly successful and has been widely emulated by state governments.

I believe the US approach reflects an understanding of the risks noted above – that price tells you nothing about capability.

The US approach may also reflect an awareness of just how embarrassing price-based selection would be after a major engineering disaster.  Imagine a TV interview shortly after the collapse of a bridge:

Reporter: So how did you award the contract?

Official: We… errrr…chose the cheapest.

Indeed.

Why don’t we ask the same question after a software project collapses?

[1] The Construction Industry Council (CIC) document is available here (700k PDF file):  http://www.nzcic.co.nz/CIC_Procument_document_internet.pdf Although it is entirely about construction, it makes interesting (and surprisingly relevant) reading for IT professionals. The CIC points out that government, as a major purchaser in the sector, must lead the way. The same applies in IT.

For another perspective, see this Canadian document: http://www.peo.on.ca/publications/DIMENSIONS/marapr2001/ma01price_consult.pdf Like New Zealand, Canada has experienced a “leaky building” scandal in which price-based procurement was a contributing factor.

[2] Ernesto E. Henroid (FIPENZ, FICE), writing in the March/April 2005 issue of e.nz.  Magazine web site: http://e.nz-magazine.co.nz/main.htm.

[3] Well-known author Steve McConnell documents a case study in which the highest bid was actually the best, because the other bidders misunderstood the task. Needless to say, the high-yet-accurate quote wasn’t chosen and the project went on to overrun its (unrealistically low) budget by 1400%. His description covers many relevant issues and can be found here: http://www.stevemcconnell.com/sg-complit.htm

[4] Price-based selection for engineering services was banned by the 1972 Brooks Act (the second Brooks Act). I was going to ask my American readers why the same rule was not applied to IT but a little research unearthed a surprising fact:  for IT, price based selection was required by the federal government.  The legislation was the 1965 Brooks Act. Yes, that’s the same congressman Brooks!   It seems Brooks had a very active interest in procurement, an interest which he expressed in two quite different laws.  The earlier (IT) act was designed to promote competition to break IBM’s stranglehold on the industry.  It achieved that goal, became widely disliked, and was repealed in 1996.  The second (engineering) act still has widespread popularity and success – and that’s an important lesson for today’s IT industry.

More on the Brooks Act(s):

http://www.asce.org/pressroom/news/grwk/event_release.cfm?uid=1777

http://www.gcn.com/21_33/community/20522-1.html

http://www.fcw.com/fcw/articles/2004/0315/pol-reform-03-15-04.asp If you’re interested in the fact that the (engineering) Brooks Act covers design, but not (as far as I know) construction, read these articles by Jack Reeves: http://www.developerdotstar.com/mag/articles/reeves_design_main.html See this page on my site too.