Imagine two novice chess players, working as a team. (We’ll assume their opponent is a computer, so it can’t overhear them talk.)  Our two novices will benefit greatly from their collaboration.  They’ll discuss all their thinking – everything from possible moves, to correcting each others mistakes, to “can you remind me how a knight moves?”.  Working together like this they will make fewer mistakes, and generate better moves than each would have done alone.

But what if we paired two chess masters?  Scientific research proves that the thinking of experts is different. Much of it happens as automatic pattern recognition rather than conscious reasoning.  When a chess master looks at a chess position, (good) possible moves spring to mind immediately.  As described in my previous post, these “candidate moves” are generated directly by the brain’s underlying neural network.  Neural networks can’t explain their own reasoning, so the master doesn’t consciously know why those moves came to mind.  Of course, the moves are indeed based on the master’s long experience, but the mapping from prior experience to current ideas is not open to inspection by the conscious mind.

So how will our two chess masters collaborate?  Neither can explain to the other why they thought of particular moves – because the conscious, effortful part of the mind doesn’t actually know. So what will their conversation consist of?

• They can’t discuss the justification for their proposed moves, because as we just saw, they don’t know the justification.  (Several writers, including Nobel-winner Daniel Kahneman, point out that if you ask an expert, “Why did you suggest that?”, their effortful mind will indeed come up with a justification of the idea, but the justification is produced after the fact. It doesn’t necessarily match the invisible reasoning that originally created the idea.  In fact, I’d suggest that since the invisible reasoning consisted of pattern matching in a neural network, any attempt at a logical sequential description would be incomplete at best.)
• Instead, imagine that they pool their sets of candidate moves, and then discuss which of those moves to actually play.  This approach makes sense because, as we saw in the previous post, selecting one of the candidate moves requires effortful thought.  And effortful thoughts can be perceived and described by the thinker.  So let’s assume our two chess masters try to share these thoughts.  If they do, I believe they’ll run into another problem: verbalizing the thought process slows it down – a lot.  A master thinks through many possibilities: What would happen if I play this move?  How would my opponent probably respond?  How would I respond to that?  A master can process these “what ifs” relatively quickly.  Not nearly as quickly as the automatic thinking that produced the initial set of candidate moves, but much, much faster than the rate of human speech.   To share thoughts verbally, the master must slow down to speaking speed.  In summary, although it’s possible to verbalize these thoughts, there’s a high (performance) cost in doing so.

Pair Programming

Now let’s consider the analogy to pair programming. Novice programmers don’t yet make significant used of automatic pattern recognition, instead they rely more heavily on effortful thought.   Therefore their thoughts are open to introspection and can be verbally shared.  Experts, on the other hand, make significant use of the “automatic” part of their mind.  So most of their thought processes are not open to conscious introspection, and the remainder are so fast that verbalization carries a huge performance cost.

I think this may explain the confusion and frustration that some experienced programmers, including myself, feel when we’re asked to pair.  We can’t see how to map our automatic thought processes into the driver/navigator model of pair programming.  Furthermore we probably can’t explain what the problem is – at least, I know I couldn’t, until I read Kahneman’s effortful/automatic description of the brain.  His book points out that because automatic thought is, well, automatic, we lack awareness of its role in our thinking.  Since we lack awareness, we struggle to explain the difficulty we have with pairing. I hope that greater awareness of Kahneman’s work will give us a suitable vocabulary to describe the problem.

I also hope to stimulate discussion.  The thought processes of an expert, which elegantly combine the automatic with the logical, are extremely efficient. I believe pairing undermines this efficiency – by leading coders to create after-the-fact justifications of their automatic intuitions, and forcing them to unpack their conscious thoughts into spoken language.  I suspect paired experts may even find themselves forced back into novice-like patterns of thought.

What do you think, does pairing prevent experts from performing at their best?

Update, 31 Jan: I should clarify what I mean by “expert”.  I use the term in the sense used by Daniel Kanheman, Anders Ericsson and other researches into expertise.  But you might have seem some pair programming studies use the term “expert” in a different way, particularly those studies where all test subjects were students.  In those studies the word “expert” simply means “good performer” – a student who gets unusually good grades.  These cannot be experts in the Kahneman/Ericsson sense of term.  Developing that kind of expertise takes many years, far longer than any university degree.  Furthermore, Kahneman/Ericsson expertise is not simply about possessing knowledge, in the manner of an A student; instead it is about a mode of thought in which much of the knowledge is possessed and processed unconsciously, through automatic pattern-recognition honed by long experience.

Agile development calls for a certain amount of ambiguity and flux in the project. Not everyone enjoys ambiguity and flux. I would suggest that most people don’t.

A very good point.  I think this affects some agile implementations – causing them to back away from being really agile, into a no-man’s-land between agile and waterfall. 

Personally, I prefer the honest ambiguity of agile to the Clayton’s certainty of waterfall.  Software development is risky.  Customers and suppliers do learn during the project.  Users do give better feedback from trying software than reading documents. To pretend otherwise may feel safer in the short term, but it disappoints in the long run.

PS If you spot any problems with the email subscription, please let me know.  I’ve never used this particular email plug-in before, so I don’t know how it will work out.

It’s called Tactyle, in reference to its emphasis on touching and interacting with the content of your project.  Other key design principles include:

• simple, effective, and fully-automated Earned Value – in keeping with my blog posts in recent years
• suitable for fixed price and/or fixed scope – again, in keeping with this blog
• based in the Story Mapping paradigm (it’s not a database, it’s not a kanban board; it’s just totally a Story Map)
• designed to have the direct “tangible-ness” of index cards (more so than any other tool I’ve seen – IMHO) while also being authentically digital
• low-friction (i.e. quick and easy to use)
• deliberate simplicity (to the point that there are only two screens in the whole product)

You can find videos, documentation, and the product itself at www.tactyle.net.

I’ve been using it myself for about 4 months (to manage it’s own development).  I hope you find it as useful, and addictive, as I have.

(And yes, development of the tool does explain the relative silence on this blog in recent months! ;-)

An accessible and comprehensive introduction to Earned Value (EV). [It] makes extensive use of graphical charts, instead of mathematical formulae, and is suitable for readers with no prior knowledge of the subject. Key topics are the fundamental principles of EVM, its history and current usage, and special considerations for applying it to software development projects.

Enjoy!

The design problem at hand is actually very small, just two user interface controls – a slider and a button – but this post is long.  That’s kind of the point, to illustrate that even simple things involve a non-trivial amount of “discussion” with the situation at hand.

Background

I’m building a scope management tool for agile teams.  This post is about one very small part of it: the portion of the user interface that controls zooming in and out.  Like Word, Excel and virtually every “document” centric application, mine will have a slider near the bottom right corner to let the user zoom in and out.  But unlike Word and Excel, my app suffers from two unique design constraints:

• The space available for the zoom controls is a square.  It is much taller, but also much narrower, than the zoom controls which most other apps slot into their status bars.  (Mine doesn’t have a status bar.)
• My app needs a really easy way for users to “toggle” between being zoomed in to see detail, and zoomed out to see the big picture.  If I can’t make this easy… well I wouldn’t even want to use the app myself!

My initial design sketch looked  like this:

My Conversation with the Situation

Here’s how it unfolded as I did the coding.  In the left hand column I’ve listed the actions I took.  In the right hand column, I have represented the situation’s “back talk” – the unexpected things that I learned as I went along.

Here’s the end result:

When you click the button, the app jumps to zoomed out, and the icon changes to this:

Click it again, and it toggles back to the first state (with your original zoom level, whatever it was)

[Update 5 months after initial posting: I have made more changes since writing this blog post.  The conversation with the situation continued, even after I thought it was finished ;-) ]

Done

That’s it.  My conversation with the situation (or my internal design monologue, if you prefer) in designing one small part of the UI.  I hope it helps to demonstrate several points:

1. We do a lot of this, when we build software.  My simple example here has 14 cycles of action and reflection, and that’s just for two UI controls – and those 14 cycles don’t include the smaller act/reflect cycles involved in actually coding the functionally.  In this case the coding consisted of hooking up these two controls to the rest of the UI, so they did actually make zooming happen.  I did that coding as I worked, but have glossed over it in this post.
2. Notice that, even before I started these 14 cycles, I had already done a so-called “design”.  I had done gone though about 3 iterations of hand drawn sketches, finishing with the one at the top of this post.  I had also designed the algorithm that would compute the right “auto zoom” level for large documents.  Everything on this page happened after finishing “design”.  That is normal. When a good software engineer does a professional job, he or she has a detailed conversation with the situation while they are coding.
3. A good conversation with the situation includes asking yourself questions like, “How will the user feel about using this?”, “Is it still working out OK?”.  It also includes being observant and noticing variants of the problem which were not explicitly thought about before – e.g. “What does auto-zoom mean in a small document?”.  Finally, it includes responding to pleasant surprises, such as my realisation that I could solve the space problem by making the slider partially transparent.
4. Imagine if I had not had this conversation with the situation, and had simply implemented the original design, exactly as it was.  I would have ended up with something that was correct in theory, but annoying and confusing in practice.  Does that remind you of anything?  Most bespoke enterprise software, perhaps? ;-)

FAQ

Was I being too picky, doing all this for just two UI controls? Well, this example is from a personal project.  In my day job, I don’t recall putting so much attention into UI design – because that is not the priority for the business customers I deal with.  Instead, most of our “conversations with the situation” relate to business logic. Those conversations are both more important, and more intricate, than the example on this page.  Unfortunately they’re much harder to write about because they’re so abstract.

…conversation with the situation.

What a lovely phrase.  It comes from the classic book The Reflective Practitioner: How Professionals Think In Action by Donald Schön. Schön was an influential professor at MIT.  His book was published in 1983, so none of his case studies come from IT, and yet he finds something that agile software developers will recognise.  In all the professions he studied, professionals relied heavily on feedback and reflection (i.e. being observant and thoughtful, and changing your mind as necessary).  He found that this approach is not just the normal way for professionals to work – it’s also the best way.

Here are some favourite quotes from the book:

At the same time that the enquirer [i.e. professional] tries to shape the situation…, he must hold himself open to the situation’s back-talk. He must be willing to enter into new confusions and uncertainties.  Hence, he must adopt a kind of double vision.  He must act in accordance with the view he has adopted, but he must realize that he an always break it open [change it] later, indeed must break it open later in order to make new sense of his transaction with the situation.

… he recognizes that the situation, having a life of its own distinct from his intentions, may foil his [plans] and reveal new meanings.

Implications

Schön’s work sheds light on some key issues in software development:

Issue 1 – BDUF

Firstly, consider the agile aversion to Big Design Up Front (BDUF).  BDUF is not an on-going conversation with the situation.  It’s just a period of thinking, followed by an attempt to give the situation a lecture!  That doesn’t work in any other professions.

Issue 2 – Handing over “designs”

Secondly, there are implications for passing work from one person to another.  Some years ago, as a software architect, I had difficulty working with a more junior developer.  Our interactions just didn’t seem to be successful.  I now realise what was going on.  He thought I was handing over a description of what he should build; but I thought I was handing over a conversation with the situation.

It’s difficult to hand over an in-flight conversation, for someone else to continue in your absence.  And yet, when a senior hands over a “design” to a junior, that is exactly what’s happening.  The senior is stepping back from the conversation with the situation, and expecting the junior to continue it.

Here’s this conversation I’ve been having with the problem. Here’s a rough outline of what I said and how the situation replied.  So I think we might proceed as follows…<outline of suggested solution goes here>.  But, don’t just take my word for it.  Continue the conversation, thinking and altering course when necessary, as I would have done if I had continued the conversation myself. Finally, if you need to change course, but don’t know how, ask me and we’ll figure it out together.”

That’s the verbose way to say it.  Unfortunately, we often cut corners and say, “Here, build this”.  Junior programmers don’t know that, “Here, build this”, really means , “Please continue my conversation.”  In fact lots of senior programmers don’t know that either – I know I didn’t.  It was only when I read Shön’s book that I finally had the mental tools to really understand the situation.

Issue 3 – “BA” or “Designer”

Finally, the conversation metaphor tells us about the role of business analysts, on both traditional and agile projects.  At the recent  #10yrsagile event in Utah, Jeff Patton and others commented on the difference between “BAs” and “designers”. The “designer” model directly supports “conversations with the situation”, since the person in the designer role clearly undertakes such dialog with the situation.  But in the BA model, if interaction between the BA and developer is either (i) infrequent, (ii) asynchronous or (iii) one-way, then the conversation with the situation breaks down.

Example

Here is a detailed real example, of a conversation with the situation while coding.

Support from Other Authors

Several other authors have shared similar findings.  For instance, in his book “The Design of Design” Fred Brooks quotes the following diagram (originally from Maher, Poon and Boulanger):

In this diagram, the problem and solution both shape each other, as we move left-to-right through time.  (I remember it as the “crocodile diagram”, since the zig-zags look like the teeth of a crudely-drawn crocodile.)

And, of course, no discussion of these topics would be complete without referring to the classic paper from Jack Reeves. Schön’s work adds weight to his arguments.

Conclusion

When we put all this together, we are led to a surprising conclusion regarding specifications (and their agile equivalents, such as acceptance tests or criteria that are prepared in advance of coding).

Regrettably, we are all encouraged to make this mistake.  On traditional projects, there may be implicit or explicit pressure to minimise the number of change requests.  On agile projects, there may be pressure to keep the velocity up, and some developers may respond by doing just enough to meet the pre-written acceptance tests, and quite deliberately “not asking too many questions”.  In so doing, they rob the project of their natural talent for “looking around and taking initiative”.  Finally, on projects of all kinds, there is often insufficient time budgeted for the Validation (fitness to purpose) part of “Verification and Validation”.

I hope that, with increasing recognition of Schön’s work, will come the realisation that true professionals do not comply with a spec, they surpass it.

The event was both a celebration of the first 10 years of agile, and an attempt to answer three questions which Alistair circulated before the meeting:

1. What problems in software or product development have we solved (and therefore should not simply keep re-solving)?
2. What problems are fundamentally unsolvable (so therefore we should not keep trying to “solve” them)?
3. What problems can we sensibly address – problems that we can mitigate either with money, effort or innovation? (and therefore, these are the problems we should set our attention to, next.)

There was clearly much to celebrate from the past 10 years of agile, but when it came to the future, there seemed to be concerns in several areas: the slowness with which good practices are adopted in some quarters; the risk of losing momentum now that there is no longer a common “enemy” (as heavy process was, when agile began); and a perceived lack of strength, clarity and consensus in the group’s answer to Alistair’s third question – “what next?”

Reflecting on those concerns, here are my “ah-has” from my virtual participation:

Gradual Change is Normal (Part 1)

Mike Cohn wrote a great post likening agile to Object Orientation.  Eventually, OO was taken for granted, and people stopped talking about it.  He suggests that agile will go the same way, and I agree.

However, these things can take a really, really long time. Here’s an example, that extends Mike’s OO one: By the late 90’s, OO had blossomed so much that the premier OO conference, OOPSLA, moved away from its original goal of making OO practical and understandable.  OOPSLA’s work was done.  But, at the turn of the century, there were still a lot of people who weren’t using OO.  Legions of VB6 developers making business apps on the Microsoft platform were not using OO at all.  Slowly, over the decade that followed, OO seeped into the Microsoft development community.  Languages were replaced and improved, and starting in about 2007 we saw wide adoption of Object-Relational Mapping and a variant of Smalltalk’s decades-old Model-View-Presenter pattern.  (Yes, Microsoft played a role in the timing, but so did changing sentiment within the developer community.)

There are two key things to note from this example:

• We are currently seeing (real) OO go mainstream in the Microsoft community –  about a decade after it went mainstream in most other parts of the development community.
• This happened without any industry association driving it.   There was no equivalent to the Agile Alliance, cajoling Microsoft developers into using Objects.  There was no need for such a body, because OO had become adequately embedded in the industry’s  consciousness back in the 90s.  It was like a virus that had infected enough  of the population that it could no longer be eliminated.  Slowly, over the decade that followed, it reached the rest.

So don’t sweat it. Agile isn’t going away.  It will continue to influence people, and to be embraced by new users, long after the early adopters have taken it for granted and stopped talking about it.

Gradual Change is Normal (Part 2)

In a post on his delightfully-named blog, Jonathan House wrote the following after the Snowbird event:

Does Agile really make a difference? – Not so much a question that showed up at the conference, but one that kept running around in my head, kicking over garbage cans and spray-painting the cat. … It’s clear we’ve made excellent progress over the past 10 years, but it’s still not so clear to me why businesses aren’t beating down the door to really adopt Agile throughout the enterprise …

Jonathan, I’m fairly sure this is not a reflection on agile.  It is reflection on business.  Business leaders often fail to follow advice, even when they actually agree that the advice is good.  I don’t say this out of personal dissatisfaction; it is a well-researched fact.  There’s even a book on the subject, called The Knowing-Doing Gap by Jeff Pfeffer and Bob Sutton.  (I mentioned Pfeffer and Sutton in my Agile Roots talk.  As an agilist and geek, I find their work extremely credible and relevant).

I am convinced that you are observing not a failure in agile, but a significant problem in business in general.

(Having said that, it wouldn’t hurt if we in the agile community could familiarize ourselves with how the more self-aware sections of business community do already understand agile, just with different words and under different names.  It will be much more polite, and therefore more likely to succeed, if we listen before we lecture).

Diversity is Good

I think some people who were following the Snowbird event may have been disappointed at an apparent lack of consensus.  I know I was, at first.  But then I remembered that diversity is good.  Think of the importance of genetic diversity in populations of animals; remember how the messy diversity of capitalism out-performed Soviet central planning; and think back to the diversity of the original 17 signatories to the Agile Manifesto – a diversity which was alive and well both before and after they signed the Manifesto.

By having a range of people, with differing interests and priorities, the agile community is much more likely to successfully address the many and varied challenges of the next decade.

I think the point with diversity is not just to accept it, but to actually encourage it.  Good agile, particularly by experienced practitioners, will be situationally specific.  I suspect this point has not fully seeped into the overall consciousness of the wider agile community.  Perhaps one of the tasks for the next 10 years is to promote the acceptance of “situationally specific agile”, and to help people learn (i.e. practice) how to do it.

There is Plenty to Do

Several people pointed out that agile lacks a clear “enemy” now.  When agile started, heavyweight process was the enemy. But today there is no clearly-identifiable external enemy.  So what is the point of the agile movement now?

David Anderson seemed to reflect the mood of the group when he wrote:

The mission now is incremental improvement. It’s evolution, education and improving levels of maturity, rather than a revolution. The enemy is now within. The enemy is as Joshua Kerievsky put it “all the crap I see out there” despite 10 years of Agile methods.

David Anderson went on to write

I don’t believe the Agile movement knows how to operate without something to revolt against. Agile came, it served its purpose, it had a positive effect. What next? Perhaps it is time to move on?

Is the really nothing more to do?  Nothing more but combating the enemies within, of complacency and poor execution?  I can see the logic of what Joshua and David (and others) are saying, but please, please don’t let it be true!  There is so much more to do.  Here are just a few topics that come to mind:

• The spread of agile beyond software (as per Jonathan House’s comment above, and others on the day)
• Jeff Patton’s story mapping and feature thinning – two wonderful techniques which deserve to become much more prominent in agile’s second decade
• Advancement of knowledge/skill in Ri-level/situationally-specific agile
• Philippe Kruchten’s herd of elephants (elaborated/commented on here by Scott Ambler)
• Really paying attention to “Individuals and Interactions”.  For 10 years we’ve been getting this first point of the Agile Manifesto round the wrong way!!  So you can’t tell me there’s nothing left to do ;-)  James Coplien phrased it well in his outstanding reflection on the first 10 years: “We have test scripts and jUnit trumping individuals and interactions”.
• Sharing our failures as well as our successes, as several people mentioned on the day
• Contracts for agile projects – currently, only a partially-solved problem at best.

We’ve only been doing knowledge work in teams for a short part of human history.  The previous millennia did not prepare us well for the last few decades of complex, abstract, co-operation in non-trivial teams.  So of course this is still more to learn.

Summary

We need to keep thinking, keep talking and keep learning.

We need to keep agile’s sense of humanity – of valuing, respecting and caring for people in the workplace.

We need to retain and treasure the global and local communities of practitioners, which the agile movement has created.

We need to keep all these things, even though there is no longer a common enemy to unite us.  We need to learn to operate in an agile community that doesn’t just accept, but actively promotes, its own diversity.

I recently stumbled across a quote from the French poet Stéphane Mallarmé:

to define is to kill, to suggest is to create

Let us not define our future.  Let’s suggest it.