There’s little-known answer to these questions, backed by several decades of research.  It’s not pretty, since it shows the true reasons why these problems are so incredibly hard to solve.  Fortunately, it also offers a solution.  The solution is difficult, requiring much commitment, patience and humility.  …But then, if you think about it, you already knew there wouldn’t be an easy solution, right?  :-)

In a nutshell

The heart of the problem is that we don’t learn because we don’t communicate effectively.  When humans communicate, we tend to value these things:

• achieve my goals (as defined by me)
• win, don’t lose
• avoid triggering negative emotions in myself or others

Prof. Chris Argyris of Harvard University has found that these core values, or “governing variables” as  he calls them, underlie most human communication.  People espouse all sorts of ideas about we ought to communicate, but when Argyis and his colleagues observed how people actually do communicate, they found these same values occurred over and over again – across cultures, across genders and across many thousands of people.  I don’t know whether these values are encoded in our DNA, installed in us by our culture(s), or just a pattern that our minds subconsciously settle upon.  In any case, they form a consistent pattern of human communication, which I will refer to here as the “Default Communication Style”.

One consequence of the Default Communication Style is that, in trying to prevent triggering negative emotions, people unilaterally protect themselves and others.  Rather than seeking the other party’s true thoughts, via genuine dialog, a person will guess or presume the other person’s thoughts and never put the guesses to the test.  So we end up walking around our offices assuming that certain people have “bad” goals or intentions, without ever testing to see if our assumptions are actually true. (Often they are not).  Wikipedia describes a few familiar symptoms: withholding information, creating rules to censor information and behaviour, and holding private meetings (aka talking about people behind their backs).

The values of “win; don’t lose” and “achieve my goals (as I define them)”  also hinder the free and full exchange of information.  All in all, the Default Communication Style causes people to withhold relevant thoughts and information.  So, deprived of full information, our organisations fail to learn.

What’s the alternative?

Argyris suggests that we should adopt the following value set instead.

• give and receive valid information
• favour informed choice for all concerned (as opposed to unilateral control)
• mutual responsibility for “looking out” for each other

So we become more concerned with sharing what is on our minds, and equally we become more concerned with helping others to share what is on their minds. (Even when its different from our own views.).  The best terms I’ve heard to describe this are “Advocacy and Inquiry” (*).  Advocating our own thoughts with skill and openness; and inquiring into the thoughts of others with skill (again) and curiosity.  We want to get the full picture flowing both ways: out of our own heads and into the group’s consciousness, and equally out of each other member of the group and into our own awareness.

When engaging in Advocacy and Inquiry, we should not be hamstrung by the desire to avoid causing offence.  That’s not an open licence to be deliberately offensive, but rather a recognition that our normal cop-outs are self-defeating.  When we avoid raising awkward issues, we ultimately let down ourselves and also the person who our silence is designed to “protect”.

More evidence

You might not have heard of Argyris’s work before.  But you probably will have heard of other people who have independently discovered the same ideas.  For instance, the authors of the book Crucial Conversations  followed around individuals who were good at “getting things done” in the workplace. The authors analysed the behaviour of these communication stars, to find out how they did it.  The answer centres around adding one’s own contributions to the group’s “pool of meaning”, and helping others to do likewise. I see Crucial Conversations as a excellent “how to” manual, for Advocacy and Inquiry.

The classic guidebook to negotiation, Getting to Yes, emphasises sharing of valid information instead of “winning” one’s opening position.

Bob Sutton of Stanford writes a great blog, and summarizes the same ideas with the phrase

Argue like you’re right; listen like you’re wrong

Why is it so difficult?

There’s a danger that all of the above will sound like common sense.  And to some degree it is.  Its easy to look at, and say, “Yes, I think that’s a good idea”.  It’s much harder is to actually put it into practice.  When it comes to the Default Communication Style, we’ve each had a whole lifetime of practice.  So, it is deeply ingrained in us, and we use it instinctively.  And that is the real challenge: not simply to believe that Advocacy and Inquiry is the right way to go, but to actually act consistently with that belief in our moment-by-moment communication.

It is easy to fail.  Argyris found that people can agree with his ideas, but still have trouble putting them in to practice.  He calls this the difference between espoused theory (the theory expressed by what we claim to believe) and theory-in-use (the theory implied by what we actually do).  I recently experienced the difference first hand.  While having a conversation with a manager, I felt that the manager was becoming upset with what I was saying, so I backed away from fully expressing myself.  And what was the topic?  I was trying to say that we need more Advocacy and Inquiry!!!  In a conversation about these exact issues, I still slipped back into Default behaviour when the going got tough!

Why do we fall back into our old habits? Because we are used to them — very skilled in them, in fact.  We are much less skilled in Advocacy and Inquiry.  So, when the going gets tough, we unconsciously fall back to our old ways.

What do we need to do?

Practice.  Effective communication requires practice, just like playing tennis, reading music or programming a computer.  You won’t be perfect when you first start practicing, but everyone has to start somewhere.

Actually, we don’t need only practice.  It also helps to obtain some up-front knowledge of the topic.  Something to “seed” our practice, so to speak. Two great starting points are the books Crucial Conversations, which I see as a manual for Advocacy and Inquiry, and Discussing the Undiscussable, which is a beginner’s guide to Argyris’s work.

You can find additional resources at Benjamin Mitchell’s blog, and at Action Design.

Finally, I hope to post more on this topic myself over the coming months.  So stay tuned…  In the meantime, please leave questions, comments, and links to additional resources below.

—-

(*) I’m calling the two sets of values the “Default Communication Style”, for the first, and “Advocacy and Inquiry” for the second.  If you’re familiar withe Argyris’s work you’ll know that these are not the standard terms.  Argyris himself uses the terms “Model I” and “Model II” respectively – with a self-deprecating grin at the blandness of the names.  Other authors use the terms “Unilateral Control Model” and “Mutual Learning Model”.   Those are good descriptive terms, but I just can’t imagine someone standing up at the front of a team meeting and saying, “Hey guys, we need to promote the Mutual Learning Model!”.   It seems more plausible to say, “Hey guys, we need more Advocacy and Inquiry!”.

/p

How does expertise develop?

Kahneman’s book points out that expertise develops, virtually inevitably, when a practitioner has long experience in a field where both of the following apply:

1. There is indeed some correlation between decisions and outcomes. (Conversely, no amount of practice will ever give you expertise in predicting the next outcome on a roulette wheel!) In the field of software development, our design decisions do indeed influence the outcomes of correctness, performance and maintainability.
2. Practitioners receive feedback on their decisions (so that their inner neural network can learn). In other words, we make decisions, and then actually see how those decisions affect the correctness, performance and maintainability of our software. As long as we maintain an open-minded willingness to learn, and we don’t change jobs too often, we will receive this feedback and therefore expertise will develop.

A case in favour of pairing

While I’ve questioned the efficiency of expert-expert pairings, the science of expertise suggests a possible benefit of expert-novice pairings – a benefit which I have not seen described elsewhere. I suspect that such pairing may help the novice to develop expertise more quickly. Not because of what they expert says (as implied by some other descriptions of expert-novice parings), but because of what the expert does. Remember that much of what we call “expertise” takes place unconsciously. So the expert couldn’t put it into words even if they tried. But, perhaps, there is some value in less experienced programmers seeing how the expert works. Maybe that will give the less-experienced member of the pair a chance to pick up on things which the expert could never teach verbally.

A related point is that much of programming is about how the software is produced. Yes, you can learn from reading other people’s code after it’s finished (and we should do this more often) but I think we might learn more from observing how that code is produced – What kind of things did the author try as they worked? Where did they backtrack? How did unit tests guide their work? What are their special Google search tricks for finding relevant information?

(Note: don’t forget the conventional advice that “pair programming is best done by peers”. An expert-novice pairing is arguably not at all the same thing as regular pair programming.  My point here is that it might be a valuable tool for transferring the expert’s “automatic” (“non-concsious”) skills.  I would see it being used in moderation; rather than  for 6 or 8 hours a day like regular pairing.)

The mathematics of paired experts

I discussed my previous post with a friend before writing it. We agreed that experts can’t discuss the first part of their thinking (the automatic bit), and that discussing the second part (the effortful bit) takes longer because the experts have to slow down to the speed of human speech.  However my friend suggested that the collaboration will still pay off, due to improved quality of the final decision.

That could be a very strong argument if feedback from a peer was the only kind of feedback available.  But, it’s not.  Experts also get feedback from the situation, via unit tests etc.

Example: An expert is looking for the cause of a defect.  The bug is reproducible, but the expert doesn’t know what causes it.  Hunting for the bug is like a search.  At each step in the process, the expert chooses a “move” that will narrow the range of possible causes.  Moves may include setting particular breakpoints, looking for patterns in the inputs that cause the bug, stepping through the code, adding additional unit tests, and so on.  Each move narrows down the search space, until eventually the defect is found.  The better the move, the more it narrows the search space.  A really good first move might eliminate 90% of possible causes, leaving only 10% remaining.  A less efficient move may eliminate only 50% of the possible causes, leaving the other 50% still to be searched.

Question: if the expert is paired with another, will the pair choose substantially better moves?  Yes, they probably will.  After some discussion, the pair may choose a “90% move” (eliminating 90% of possible causes) while an expert working alone might only come up with a “70% move”.

But what about the time cost of the pair’s discussion?  In the time that the pair spend discussing one move, a solo expert might try several. Imagine that a solo expert can make two moves in the time it takes a pair to discuss one.  If the solo expert tries one 70% move, followed by another, they’ll cut the search space down to 9% of the original space. [ (100% – 70%)^2 =  30%^2 = 9%].   That’s about the same amount of progress as the pair’s “90% move”.   So the solo expert has produced the same result as the pair, in the same amount of time, but for only half the personnel cost.  If the solo expert’s cycle time is even shorter still, allowing them to try 3 or 4 things in the time that the pair tries one, the solo expert will outperform the pair in both cost and elapsed time.

In summary I suggest that: by using a shorter cycle time a solo expert might outperform a pair of experts.  This a special case of the general principle that “starting to iterate” often beats “improving the plan”.

So how should experts interact?

If experts avoid pair programming, will they end up working completely alone?  No.  I think there are at least two options worth considering:

Option 1: Expert “pair preview”

I’m suggesting this based on a real life example.  I’m calling it “preview” rather than “review” because it comes before the code is written.  I’ll anonymise it by changing the names to “Alice” and “Bob”.

Alice was (and still is) an awesome programmer, whose expertise I greatly respect.  She was working alone on a difficult part of the system.  Once Alice had an initial design in mind, she ran it by Bob for review.  So far, so good.  But the review is where things came unstuck.  Alice and Bob didn’t know Kahneman’s concept of effortful versus automatic thought.  Alice described her proposed design to Bob.  Bob understood it, but had a hunch that an alternative design would be simpler.  Bob didn’t know it, but his half-formed alternative design was generated by the automatic “part” of his mind.  Because it was automatically-generated, it seemed like nothing but a vague hunch.  Consequently neither Bob nor Alice gave it the attention it deserved.

In hindsight, they should have recognised Bob’s hunch for what it was: an instinctive product of Bob’s long expertise, and therefore something worth looking into.  Perhaps Alice should have spent a couple of days spiking Bob’s idea.  That would have tested the hunch, and by having Alice do the spiking (rather than Bob, who had suggested it), we would have leveraged Alice’s existing detailed knowledge of the problem space.  (Having Alice do the spike would have also made it psychologically easier for her to embrace the new idea, if it did indeed prove to be the best.)

Option 2: “Expert Escalates”

Even experts get stuck.  Part of being a responsible expert is to realise when you’re not making productive progress, and seek out someone to bounce ideas off.  Perhaps they will see something that you haven’t, or spark some new ideas.

What about the evidence?

Two commenters on my previous post suggested that my concerns were all very well, but couldn’t be valid because the research shows that pair programming works.  I promised to respond with some comments on “the evidence”.

It turns out that the evidence is rather more mixed that you might suppose. Here’s a quote from the abstract of a meta analysis done by the excellent Simula Research Laboratory.

…between-study variance is significant, and there are signs of publication bias among published studies on pair programming.  A more detailed examination of the evidence suggests that pair programming is faster than solo programming when programming task complexity is low and yields code solutions of higher quality when task complexity is high. The higher quality for complex tasks comes at a price of considerably greater effort
– from here, or alternate link [emphasis added]

The finding, that paired effort for complex tasks is much greater than solo effort, appears consistent with my above concerns and reasoning about paired experts.

Imagine two novice chess players, working as a team. (We’ll assume their opponent is a computer, so it can’t overhear them talk.)  Our two novices will benefit greatly from their collaboration.  They’ll discuss all their thinking – everything from possible moves, to correcting each others mistakes, to “can you remind me how a knight moves?”.  Working together like this they will make fewer mistakes, and generate better moves than each would have done alone.

But what if we paired two chess masters?  Scientific research proves that the thinking of experts is different. Much of it happens as automatic pattern recognition rather than conscious reasoning.  When a chess master looks at a chess position, (good) possible moves spring to mind immediately.  As described in my previous post, these “candidate moves” are generated directly by the brain’s underlying neural network.  Neural networks can’t explain their own reasoning, so the master doesn’t consciously know why those moves came to mind.  Of course, the moves are indeed based on the master’s long experience, but the mapping from prior experience to current ideas is not open to inspection by the conscious mind.

So how will our two chess masters collaborate?  Neither can explain to the other why they thought of particular moves – because the conscious, effortful part of the mind doesn’t actually know. So what will their conversation consist of?

• They can’t discuss the justification for their proposed moves, because as we just saw, they don’t know the justification.  (Several writers, including Nobel-winner Daniel Kahneman, point out that if you ask an expert, “Why did you suggest that?”, their effortful mind will indeed come up with a justification of the idea, but the justification is produced after the fact. It doesn’t necessarily match the invisible reasoning that originally created the idea.  In fact, I’d suggest that since the invisible reasoning consisted of pattern matching in a neural network, any attempt at a logical sequential description would be incomplete at best.)
• Instead, imagine that they pool their sets of candidate moves, and then discuss which of those moves to actually play.  This approach makes sense because, as we saw in the previous post, selecting one of the candidate moves requires effortful thought.  And effortful thoughts can be perceived and described by the thinker.  So let’s assume our two chess masters try to share these thoughts.  If they do, I believe they’ll run into another problem: verbalizing the thought process slows it down – a lot.  A master thinks through many possibilities: What would happen if I play this move?  How would my opponent probably respond?  How would I respond to that?  A master can process these “what ifs” relatively quickly.  Not nearly as quickly as the automatic thinking that produced the initial set of candidate moves, but much, much faster than the rate of human speech.   To share thoughts verbally, the master must slow down to speaking speed.  In summary, although it’s possible to verbalize these thoughts, there’s a high (performance) cost in doing so.

Pair Programming

Now let’s consider the analogy to pair programming. Novice programmers don’t yet make significant used of automatic pattern recognition, instead they rely more heavily on effortful thought.   Therefore their thoughts are open to introspection and can be verbally shared.  Experts, on the other hand, make significant use of the “automatic” part of their mind.  So most of their thought processes are not open to conscious introspection, and the remainder are so fast that verbalization carries a huge performance cost.

I think this may explain the confusion and frustration that some experienced programmers, including myself, feel when we’re asked to pair.  We can’t see how to map our automatic thought processes into the driver/navigator model of pair programming.  Furthermore we probably can’t explain what the problem is – at least, I know I couldn’t, until I read Kahneman’s effortful/automatic description of the brain.  His book points out that because automatic thought is, well, automatic, we lack awareness of its role in our thinking.  Since we lack awareness, we struggle to explain the difficulty we have with pairing. I hope that greater awareness of Kahneman’s work will give us a suitable vocabulary to describe the problem.

I also hope to stimulate discussion.  The thought processes of an expert, which elegantly combine the automatic with the logical, are extremely efficient. I believe pairing undermines this efficiency – by leading coders to create after-the-fact justifications of their automatic intuitions, and forcing them to unpack their conscious thoughts into spoken language.  I suspect paired experts may even find themselves forced back into novice-like patterns of thought.

What do you think, does pairing prevent experts from performing at their best?

Update, 31 Jan: I should clarify what I mean by “expert”.  I use the term in the sense used by Daniel Kanheman, Anders Ericsson and other researches into expertise.  But you might have seem some pair programming studies use the term “expert” in a different way, particularly those studies where all test subjects were students.  In those studies the word “expert” simply means “good performer” – a student who gets unusually good grades.  These cannot be experts in the Kahneman/Ericsson sense of term.  Developing that kind of expertise takes many years, far longer than any university degree.  Furthermore, Kahneman/Ericsson expertise is not simply about possessing knowledge, in the manner of an A student; instead it is about a mode of thought in which much of the knowledge is possessed and processed unconsciously, through automatic pattern-recognition honed by long experience.

Actually, no.

1. I’m interested in people who are not managers improving their people skills.
2. I’m interested in busting the myth that “people skills” is a topic exclusively for managers.
3. I’m interested in busting the myth that improvements in people skills must be initiated by managers or trainers.  I want them to be initiated by   people who actually want to improve their own skills.

If you’re interested in the same things, please comment below or email me (address on About page).

Is it really incredible?  Yes.  It just seems easy because you’ve been doing it effortlessly since about age three.  To remind yourself how difficult it actually is, imagine designing an algorithm to recognise dogs.  Exactly what would be the rules for distinguishing a small small dog from a large cat?  How would you define the category “dog” such that it included Chihuahuas and Great Danes,  but not foxes and wolves?

Nobel prize winner Daniel Kahneman uses the dog example to explain the two ways humans think.  One way is effortful thought – what we do when consciously thinking about something.  The other doesn’t even feel like thinking it all.  It’s just effortless automatic perception – like seeing a dog.  Much of our brain’s activity is of the automatic kind.  Kahneman gives several examples to show the difference:

Expertise

What does this have to do with expertise?  The answer is simply this: expertise involves significant “automatic” thought.  For example, you happen to be an expert in recognising dogs.  You accomplish that task with no conscious effort whatsoever.

The same applies to activities that we normally associate with “expertise”.  Chess masters are a compelling example.  When a chess master looks at a board, several strong moves immediately spring to mind.  Just like you effortlessly “see” that an animal is a dog, a chess master effortlessly “sees” which moves make sense.  This set of “candidate moves” is generated automatically, without conscious effort.  Only after the moves spring to mind does the master actually start consciously thinking about them – to decide which of the candidate moves is best.

As Kahneman writes:

[Experts’] intuitive judgements come to mind with the same immediacy as [a child’s] “doggie!"

An Analogy

It may seem hard to believe, that many of our most difficult mental tasks are performed instantly and unconsciously.  As Kahneman points out, their very automatic-ness leads us to overlook their importance. 

When I was reading his book, I found it helpful to recall the university paper I took in artificial intelligence.  There we learned about artificial neural networks. Neural networks implement an approach to computation which is inspired by the structure of the human brain. They are very fast and effective “pattern recognisers” but, having recognised an input as matching a particular pattern, they are completely unable to tell you why it matched the pattern. I.e. they can’t explain their “reasoning”. This exactly matches Kahneman’s description of our automatic thought – it gives you an impression/hunch very quickly, but is unable to tell you why. So:

• I imagined the brain’s underlying hardware as a neural network – quick to recognise patterns, but unable to explain its logic.  It is here that our automatic thinking takes place.  So it’s easy to see why our automatic thinking is so fast – neural networks are naturally fast pattern recognisers, and ours is implemented directly in “hardware”. 
• I imagined effortful thought as a simulation running on top of the underlying hardware.  Rather like a virtual machine.  Inside the virtual machine, thinking is sequential and logical.  But, because the virtual machine is just an emulation, it runs slowly and has limited working memory.  (Question: does this imply our effortful thoughts aren’t “real”?  No. Just that they have a complicated origin, but our automatic thoughts are not subject to the same performance limitations.)

I wouldn’t dare to suggest that this model is accurate in terms of the underlying biology, but as a software engineer I found it helpful in understanding Kahneman’s work.

Developing Expertise

Consider two chess players, a novice and a master.  The novice relies almost entirely on effortful thought.  “How does a knight move?”, “Let me think what would happen if I moved my rook to here…”.  But the master makes heavy use of automatic thought.  The master’s automatic mind takes care of most of the details that the novice struggles with, leaving the master’s effortful mind free to add value where it is most needed.  Consequently the master can produce better decisions in less time.

But how does the novice become a master?  How does someone using effortful thought slowly transition to creating (good-quality) automatic thoughts?  Through practice.  With enough time, and enough examples, your underlying neural network trains itself to recognise the patterns.

Three sources have influenced my thinking about this kind of practice:

• James Bach’s Myths of Rigour presentation (which to me, could equally well have been entitled “What is learning?”).  [Video here].
• Anders Ericsson’s work on deliberate practice
• Alistair Cockburn’s concept of Shu-Ha-Ri.  To join the dot’s between Shu-Ha-Ri and Kahneman’s work, I’d suggest that the Shu-Ha-Ri progression equates to the same progression described above: from a novice who uses only effortful thought, to an expert who uses a highly efficient blend of automatic and effortful thought.  

A Software Example

I recently noticed how various people debug the large software solution that I’ve been working on for some years.  When I need to debug it myself, I don’t get “stuck”. I might not know the cause of the problem, but I always have an idea on what to do next – something that will take me one step closer to finding the problem. Just like experienced players instantly “see” good moves in a chess game; I tend to “see” good moves in the debugging process. A series of such moves, possibly with some backtracking, eventually leads to a solution.  But I’ve noticed junior developers are much more likely to get stuck – when faced with a problem they sometimes have trouble generating the “next move”.   I’ve also noticed that although I generate moves more successfully than juniors, I seem to expend less conscious effort in doing so.

Of course, this is no excuse for me to take an ego trip – after about 7000 hours with this code base, and a decade’s experience on other systems before it, I darn well should have automatic expertise by now!  Junior programmers, or senior ones with less experience of the system at hand, have no choice but to generate fewer moves automatically and fill the gaps with effortful thought.  Over time, their balance will gradually shift from effortful to automatic.

Implications

The science of expertise has many implications for how we recruit, train and deploy software engineers.  In a future post, I intend to explore the implications for pair programming.

Read on, and judge for yourself whether I’ve succeeded…

Example

Consider this example adapted from Wikipedia: An entomologist finds a beetle with spots on its back. She thinks it might be a rare subspecies, because 98% of the rare subspecies have such spots, but only 5% of the common variety have them.  The subspecies makes up only 0.1% of the population. Given the fact that the beetle in her hand is spotty, what is the probability that it is also rare?

The entomologist wants to know P(rare|spotty), which reads from left to right as “probability of being rare, given that it’s spotty”.

This is the time to use Bayes’ Theorem.  She wants P(rare|spotty) but only has P(spotty|rare).  The latter is 98%, but she suspects that the former may be much lower.  She’s right, the spotted beetle in her hand is probably not rare… and Bayes’ can tell us why.

Memorable

I want to explain Bayes’ Theorem in a way that is easy to remember.  To begin, let’s start from scratch and think about the probability that one randomly-selected beetle is both rare and spotty.  There are two mathematically-correct ways to express that probability.  One is to write:

P(rare and spotty) = P(spotty|rare) P(rare)       
= “(the probability of it being spotty, given that it is rare) times (the probability of it being rare in the first place)

The other is:

P(rare and spotty) = P(rare|spotty) P(spotty)  
=  “(the probability of it being rare, given that it is spotty) times (the probability of it being spotty in the first place)”

We’ve just written down two different ways to compute the same thing.  So they must be equal to each other. I.e.:

P(spotty|rare) P(rare)  = P(rare|spotty) P(spotty)

If we re-express this in a general-purpose notation, we get

P(A|B) P(B) = P(B|A) P(A)

That is the formula you need to remember.  To recall its left-hand side, remember this intuitive fact: the probability of two events both happening is the (conditional) probability of one happening given that the other already has happened, times the (un-conditional) probability of the other.     The right-hand side is just the same thing round the other way.

That’s all you need to remember.  Start with the above formula, then use basic algebra to re-arrange it into the form you want.  For instance, if you want to compute P(A|B), just divide both sides by P(B).  (You’ll end up with the usual form of the Bayes’ equation, as seen in textbooks.)

Intuitive

Where Bayes gets interesting is when its used to assess the probability of some hypothesis, given the available evidence.    Dividing both sides of our formula by P(B), and renaming  A as the “hypothesis” and B as the “evidence”, we get this:

P(hypothesis|evidence) = P(evidence|hypothesis)   P(hypothesis)
                                          —————————
                                          P(evidence)

Let’s consider a simple example involving a blond-haired young boy called Johnny, and an apple that’ is missing from the neighbour’s tree.  The hypothesis in question is “Little Johnny stole the apple” and the evidence is “blond hair left snagged in apple tree”.   The angry neighbour wants to know the probability that Johnny did indeed steal the apple, given the evidence of the hair – i.e.  P(Johnny-stole-apple|blond-hair-left-at-scene).  Our formula becomes

P(Johnny-stole-apple|blond-hair-left-at-scene) =

     P(blond-hair-left-at-scene|Johnny-stole-apple)  P(Johnny-stole-apple)
     ———————————————————– 
      P(blond-hair-left-at-scene)

When used is this way, the formula includes three things:

• the left-hand-side, P(Johnny-stole-apple|blond-hair-left-at-scene), represents our degree of belief that Johnny stole the apple, after taking into account the evidence of the blond hair.  This is called the posterior probability.
• the term on the far right, P(Johnny-stole-apple), represents our degree of belief that Johnny stole the apple, before taking into account the evidence of the blond hair.  This is called the prior probability. 
• The rest (i.e. middle) of the equation represents how the evidence of the blond hair influences the prior belief, by making it stronger or weaker.

Let’s consider each of the RHS terms, and see how they are relevant to the outcome.  Firstly, consider P(blond-hair-left-at-scene|Johnny-stole-apple).  For instance, if the neighbour knew that Johnny suffered from excessive hair loss, then this probability would go up.  I.e. if he was particularly likely to lose hair, then the presence of hair would be stronger evidence of his guilt (as opposed to someone else’s guilt).  Conversely, if Johnny always wore hats, then the presence of a hair would be weaker evidence of his guilt.

Secondly, let’s consider P(blond-hair-left-at-scene).  This is the overall probability of a blond hair left at the scene, regardless of who committed the crime.  If this number goes up, then the hair is considered weaker evidence of guilt. For instance, if the crime occurred in a Swedish town where most boys have blond hair, then a blond hair would be relatively weak evidence against Johnny.  It could have come from almost anyone.  But if the crime occurred in China, the probability of a blond hair being left at the scene would be much lower, so the fact that one was found is stronger evidence against little blond Johnny.

Finally, let’s consider P(Johnny-stole-apple).  This is the neighbour’s prior belief that Johnny stole the apple – i.e. his belief before considering this particular evidence.  If the neighbour knows that Johnny is a caring, honest, Boy Scout with an allergy to apples, then his prior belief in Johnny’s guilt will be relatively low.  To reverse that prior belief, and produce a high posterior probability of guilt, the evidence of the hair would have to be particularly compelling.  (E.g. Johnny is the only blond-haired boy in a Chinese village, and he suffers from persistent hair loss).   If the neighbour doesn’t know Johnny at all, then he has no option but to assign prior probability based on the population in general – e.g. if there are 10 kids in the neighbourhood, the prior probability of Johnny being the thief could be assumed to be 1/10th.

So we can see that all terms of the right hand side contribute to the final result in ways that are intuitively sensible.  Bayes’ is not so hard.  It actually makes intuitive sense.

What’s so difficult then?

In practice, there can be some obstacles to applying Bayes’ successfully. Obstacles such as:

1. Remembering that P(hypothesis) is an input to computing P(hypothesis|evidence).  If you use the formula, you won’t forget. But when we rely on our intuitions instead of the formula, we tend to neglect P(hypothesis).  This mistake is known as base rate neglect.  For instance, in the spotted beetle example, P(rare) is the base rate – and it’s only only 0.1%.  If you neglect the base rate, you’ll intuitively over estimate the chances of a spotted beetle being rare.
2. Being willing to use P(hypothesis) as an input.  Having to rely on P(hypothesis) may seem inconvenient because its often hard to obtain an objectively-correct value for
   P(hypothesis).  So, out of necessity, we may end up using subjective guesses for P(hypothesis).  This subjectivity is the cause of much of the controversy that dogged Bayes’ Theorem in the 20th century.  The solution is that, for many kinds of problems, “probability” actually means “measure of belief” – rather than some number which is computed from repeatable trials.  [Subjective evaluation of P(hypothesis) is also the cause of much of the power of Bayes, since the formula doesn’t mind if P(hypothesis) is subjective.  Therefore, it’s a great way to take an initial viewpoint, which may be subjective, and adjust it as we receive new objective evidence.]
3. Computational difficulties when generalizing beyond simple yes/no hypotheses and one piece of evidence.  For instance, you may want to consider several events (blond hair left at scene and passer-by reports glimpsing possible thief in pink tracksuit), or you may have random variables instead of boolean events.  These issues are entirely solvable, but beyond the scope of this blog post.

Bayes’ Theorem gives us a powerful tool for dealing with uncertainty.  I hope this page will help you (and me) recall the basics ;-)

Finally, for an enjoyable read on the history and uses of Bayes’ Theorem, check out this new book from Sharon Bertsch McGrayne.

Postscript: feedback about this post included the following comment, which is worth quoting here in full: “fundamentally the idea actually is very simple and the interpretation and application where all the difficulty lies.”   That seems like point worth noting: that the basics of Bayes are very simple, but successfully applying it to real world problems is much more difficult.  I hope that this blog post has helped to convey the simplicity of the underlying idea.  I wrote this post while reading the above book, since I couldn’t enjoy the book without understanding what Bayes’ Theorem really is :-)   As for the difficult problems of devising real-world models and hypotheses, I’ll have to plead (relative) ignorance I’m afraid – since I’m a software designer not a statistician.

So here are some of the key points that now I keep in mind when considering estimation, pricing and contracts for agile projects.  These points are particularly relevant when customers have asked for fixed price and fixed scope.

1. It’s a valid thing to ask for…

It’s perfectly natural for customers to want some up-front insight into likely cost and scope.  In the spirit of good negotiation, we should take their concerns seriously.

2. … but many agile teams successfully dodge the issue.

I asked someone recently what they were doing about this, and their reply was that they don’t really find themselves doing many contracts with fixed price and fixed scope.  That certainly solves the problem ;-)  I suspect it depends on the kind of company you work for.  If you’re an in-house IT shop (i.e. customer and supplier are the same organisation) then I would imagine that there is little call to fix both scope and budget.  However if you work as I do, in a company that undertakes development projects for other organisations, then you’re more likely to be asked to fix both price and scope.  It also depends on the “purchasing culture” in the city where you work.  For instance I work in Wellington, where government procurement processes heavily influence the way people expect contracts to be framed.  But in Auckland, which is only an 8 hour drive away, there’s a very different procurement culture that’s dominated by private companies.

3. There are lots of different kinds of contracts which you might consider

Alistair Cockburn has a big list of suitable contract types here.

4. Flexible pricing is probably good

Rather than absolutely fixing both price and scope, it makes the project run better if you can allow at least one of them to “flex” in a controlled way.  This still gives the customer early insight into commitments and benefits,  but avoids the harsher downsides of absolutely fixing both.  Here’s an approach that I’ve tried and liked.  It’s not for everyone, but in cases where it’s understood and embraced by both parties, I do believe its better than fixing everything.

5. Here are my favourite “ingredients for success”:

• Use a “discovery phase”.  This is a timeboxed phase at the start of the project where you learn more about what the customer needs, and prepare an itemized breakdown of functionality.  That itemized breakdown is the scope that you then  commit to.  A discovery phase is good for both parties, because it results in much better-informed scoping than projects which attempt to fix the scope without such a phase.  The discovery phase should be paid, rather than delivered for free – both because the customer gets value out of it, and because adequate discovery is too big to do for free.  In terms of sizing the discovery phase, the Feature Driven Development community uses a rule of thumb that this phase should last for 2 weeks for every 6 months of development (IIRC).  I think that’s about right.  For example, I used 3 weeks for a 6 month project once, but during those 3 weeks we also built a prototype.
• The bigger the project, the more coarse-grained your discovery phase must be. (Within a realistic timeframe, you simply can’t drill down into as much detail as you might in the discovery phase for a smaller project.   And if you do, it turns into BDUF, which is counter-productive for all the well-known reasons.  So I’d suggest making do with a coarse-grained breakdown on large projects.)
• The itemized breakdown of scope should be tabular rather than prose (i.e. a table rather than a document). This is a lesson I learned the hard way – the document was too ambiguous.  One good way to organise the tabular breakdown is with a story map.
• Ideally, that itemized breakdown of scope will actually serve three purposes: it will define the scope, it will form the basis of your estimate, and it will be what you track your progress against as you work through the project.  IMHO, if you’re not using the same list for all three purposes, you’re risking trouble.
• Typically, your cost estimate will be based on summing up points for every feature that’s in scope.  Make sure you add a “buffer” of additional budget over and above that.  There are two reasons why you might need such a buffer: (a) the inherent uncertainty in estimation and (ii) the need to allow time for for Validation (i.e. ensuring fitness for purpose, rather than merely compliance to spec). 
• Adding this buffer will typically produce an estimate which feels too high. So my rule of thumb is as follows, “if your estimate feels about right, its probably not.” ;-)  Add the buffer!
• A buffer that adequately accommodates both estimation uncertainty and validation effort tends to unfeasibly large. (Unfeasible in the sense that some manager somewhere won’t accept it ;-)  So I prefer to use a target cost contract.  Some of the necessary buffering is built into the target in the target cost contract, but the rest is provided by the (controlled) flexibility.
• Know how you’ll draw the line between “this is an addition to scope, and therefore requires a budget increase”, versus “this is something we should just do, and therefore requires no change in budget”.   You always need at least some of the latter, certainly for Validation and probably also for convenience (its too much paperwork for formally increase the budget for every tiny change).  I’ve outlined one suggestion for drawing this line, part way down the target cost page.
• It’s not enough for you  to know how you’ll draw that line.  Your customer must understand too.  And agree.  This applies to both management and subject matter experts, to both ‘gold owners’ and ‘goal donors’.  All must understand what kinds of changes will affect the budget and what will not.
• Use a project tracking and management tool that gives you a “whole of scope” view.  If your tool only shows the current iteration or release, but your commitments are for the whole-of-scope, then you’re flying blind.

6. “If I do all that, will it work?”

Maybe. Maybe not ;-)   Fixing price and scope of agile projects is fiendishly difficult.   But there is a genuine business need for it so we, as agile practitioners, owe it to our customers to continue trying (and learning).

7. Understand Estimation

When I finally started studying estimation, I was shocked to discover how little I, and my colleagues, really knew about it.  We’d been getting by for years with little more than a grab-bag of misconceptions.  And we didn’t even realise.  So read Steve McConnell’s Software Estimation: Demystifying the Black Art.  I haven’t found any other book that combines all the necessary material into one readable volume. 

8. Understand the bit that Steve omitted from his book

If you’re bidding for work, in a competitive situation, then you’ve got to read this post from Steve McConnell (including the comments) and my follow up here.  It’s not entirely comforting, in that it describes a problem which software vendors can’t easily solve on their own. (The obvious solutions require engagement from purchasers.)  On a more positive note, the US engineering profession did successfully solve this problem. 

I hope to eventually blog more about this in years to come, particularly some thoughts on what we can do in the absence of a purchaser-side solution.

9. See what others are doing.

A great example is Atomic Object in Michigan.  Here’s a post on their corporate blog to get you started.  (You’ll notice they also like to introduce controlled flexibility. In Atomic’s case, they flex the scope rather than the price.)  You should also check out their  whole series on estimation.

Good luck.

PS. Is there a tool that can help? 

Mostly its not about tools, but since you asked ;-) ….  I think several things help: taking a whole-of-scope view; using story mapping; using the same artefacts as the basis for scoping, estimating and tracking; and keeping the user story/feature descriptions concise in order to prevent BDUF.  I don’t know of any one single tool that meets all those criteria – so I started writing one.  It’s not finished yet, but with a little creativity there are many other ways to get by without an integrated tool. 

All in all, this is not really in tools problem.  Its really about understanding, attitude, negotiation and communication.

Agile development calls for a certain amount of ambiguity and flux in the project. Not everyone enjoys ambiguity and flux. I would suggest that most people don’t.

A very good point.  I think this affects some agile implementations – causing them to back away from being really agile, into a no-man’s-land between agile and waterfall. 

Personally, I prefer the honest ambiguity of agile to the Clayton’s certainty of waterfall.  Software development is risky.  Customers and suppliers do learn during the project.  Users do give better feedback from trying software than reading documents. To pretend otherwise may feel safer in the short term, but it disappoints in the long run.

The key principle of the book is that conversations work best when everyone is able to “add their meaning to the pool”.  I.e. to share what’s on their mind.  Conversely, conversations work poorly when people withhold information and viewpoints:

When people purposefully withhold meaning from one another, individually smart people can do collectively stupid things.

Therefore, successful dialog depends on two learnable skills: (a) helping other people to share what’s on their mind, and (b) being able to share what’s on your mind even in difficult situations.  I’d like to share a recent example of the latter…

Our (excellent) manager was describing a change to our processes.  I hated the idea. It seemed bureaucratic, time-consuming and just plain wrong.  As he said how much “we need to do it”, I found myself giving up.  My instinct was to withdraw from the conversation.  If he was so sure, what point was there in discussing it?  He wasn’t actually asking for our opinions.  And besides, everyone else seemed to agree.  At least, they were sitting there nodding. 

But just in that moment of giving up, I caught myself. Sharing meaning is good, right?  Yes!  And even fun?  Yes, as I have discovered. So in that moment I decided to take the enjoyable, productive route, instead of the grumpy silent one. I said something like this:

I feel a bit concerned about the cost of doing this.  On <project X> we deliberately avoided gathering this data, and it was fine.  We used high-level figures, instead of the detailed breakdown we’re talking about here, and …”  I went on to briefly expand on our experience on <project X>.

Instead of giving up, I had managed to share “my meaning” with the group.  It wasn’t so hard.  It didn’t seem like I’d offended anybody, and I felt much better within myself for having shared. 

(Incidentally, talking along the lines of “I feel concerned about…” is a useful collaboration technique that I’ve picked up along the way.  It works much better, and is more honest and more informative, than bluntly saying “I don’t like that”.    Another handy technique is to talk in terms of concrete stories, as I did about <project X>.  Both these techniques deserve blog posts of their own one day…)

So did it work?  Did adding my meaning to the pool actually advance the discussion?  Yes it did.  The boss replied with a clarification of his underlying goals.  Which was great, because then we were able to think about all possible ways to meet that goal – instead of just debating between his original plan and my suggested alternative.  We haven’t finalised the decision yet, but it looks like we’re going to end up with something better than either of us had imagined.