Imagine two novice chess players, working as a team. (We’ll assume their opponent is a computer, so it can’t overhear them talk.)  Our two novices will benefit greatly from their collaboration.  They’ll discuss all their thinking – everything from possible moves, to correcting each others mistakes, to “can you remind me how a knight moves?”.  Working together like this they will make fewer mistakes, and generate better moves than each would have done alone.

But what if we paired two chess masters?  Scientific research proves that the thinking of experts is different. Much of it happens as automatic pattern recognition rather than conscious reasoning.  When a chess master looks at a chess position, (good) possible moves spring to mind immediately.  As described in my previous post, these “candidate moves” are generated directly by the brain’s underlying neural network.  Neural networks can’t explain their own reasoning, so the master doesn’t consciously know why those moves came to mind.  Of course, the moves are indeed based on the master’s long experience, but the mapping from prior experience to current ideas is not open to inspection by the conscious mind.

So how will our two chess masters collaborate?  Neither can explain to the other why they thought of particular moves – because the conscious, effortful part of the mind doesn’t actually know. So what will their conversation consist of?

• They can’t discuss the justification for their proposed moves, because as we just saw, they don’t know the justification.  (Several writers, including Nobel-winner Daniel Kahneman, point out that if you ask an expert, “Why did you suggest that?”, their effortful mind will indeed come up with a justification of the idea, but the justification is produced after the fact. It doesn’t necessarily match the invisible reasoning that originally created the idea.  In fact, I’d suggest that since the invisible reasoning consisted of pattern matching in a neural network, any attempt at a logical sequential description would be incomplete at best.)
• Instead, imagine that they pool their sets of candidate moves, and then discuss which of those moves to actually play.  This approach makes sense because, as we saw in the previous post, selecting one of the candidate moves requires effortful thought.  And effortful thoughts can be perceived and described by the thinker.  So let’s assume our two chess masters try to share these thoughts.  If they do, I believe they’ll run into another problem: verbalizing the thought process slows it down – a lot.  A master thinks through many possibilities: What would happen if I play this move?  How would my opponent probably respond?  How would I respond to that?  A master can process these “what ifs” relatively quickly.  Not nearly as quickly as the automatic thinking that produced the initial set of candidate moves, but much, much faster than the rate of human speech.   To share thoughts verbally, the master must slow down to speaking speed.  In summary, although it’s possible to verbalize these thoughts, there’s a high (performance) cost in doing so.

Pair Programming

Now let’s consider the analogy to pair programming. Novice programmers don’t yet make significant used of automatic pattern recognition, instead they rely more heavily on effortful thought.   Therefore their thoughts are open to introspection and can be verbally shared.  Experts, on the other hand, make significant use of the “automatic” part of their mind.  So most of their thought processes are not open to conscious introspection, and the remainder are so fast that verbalization carries a huge performance cost.

I think this may explain the confusion and frustration that some experienced programmers, including myself, feel when we’re asked to pair.  We can’t see how to map our automatic thought processes into the driver/navigator model of pair programming.  Furthermore we probably can’t explain what the problem is – at least, I know I couldn’t, until I read Kahneman’s effortful/automatic description of the brain.  His book points out that because automatic thought is, well, automatic, we lack awareness of its role in our thinking.  Since we lack awareness, we struggle to explain the difficulty we have with pairing. I hope that greater awareness of Kahneman’s work will give us a suitable vocabulary to describe the problem.

I also hope to stimulate discussion.  The thought processes of an expert, which elegantly combine the automatic with the logical, are extremely efficient. I believe pairing undermines this efficiency – by leading coders to create after-the-fact justifications of their automatic intuitions, and forcing them to unpack their conscious thoughts into spoken language.  I suspect paired experts may even find themselves forced back into novice-like patterns of thought.

What do you think, does pairing prevent experts from performing at their best?

Update, 31 Jan: I should clarify what I mean by “expert”.  I use the term in the sense used by Daniel Kanheman, Anders Ericsson and other researches into expertise.  But you might have seem some pair programming studies use the term “expert” in a different way, particularly those studies where all test subjects were students.  In those studies the word “expert” simply means “good performer” – a student who gets unusually good grades.  These cannot be experts in the Kahneman/Ericsson sense of term.  Developing that kind of expertise takes many years, far longer than any university degree.  Furthermore, Kahneman/Ericsson expertise is not simply about possessing knowledge, in the manner of an A student; instead it is about a mode of thought in which much of the knowledge is possessed and processed unconsciously, through automatic pattern-recognition honed by long experience.

Actually, no.

1. I’m interested in people who are not managers improving their people skills.
2. I’m interested in busting the myth that “people skills” is a topic exclusively for managers.
3. I’m interested in busting the myth that improvements in people skills must be initiated by managers or trainers.  I want them to be initiated by   people who actually want to improve their own skills.

If you’re interested in the same things, please comment below or email me (address on About page).

Is it really incredible?  Yes.  It just seems easy because you’ve been doing it effortlessly since about age three.  To remind yourself how difficult it actually is, imagine designing an algorithm to recognise dogs.  Exactly what would be the rules for distinguishing a small small dog from a large cat?  How would you define the category “dog” such that it included Chihuahuas and Great Danes,  but not foxes and wolves?

Nobel prize winner Daniel Kahneman uses the dog example to explain the two ways humans think.  One way is effortful thought – what we do when consciously thinking about something.  The other doesn’t even feel like thinking it all.  It’s just effortless automatic perception – like seeing a dog.  Much of our brain’s activity is of the automatic kind.  Kahneman gives several examples to show the difference:

Expertise

What does this have to do with expertise?  The answer is simply this: expertise involves significant “automatic” thought.  For example, you happen to be an expert in recognising dogs.  You accomplish that task with no conscious effort whatsoever.

The same applies to activities that we normally associate with “expertise”.  Chess masters are a compelling example.  When a chess master looks at a board, several strong moves immediately spring to mind.  Just like you effortlessly “see” that an animal is a dog, a chess master effortlessly “sees” which moves make sense.  This set of “candidate moves” is generated automatically, without conscious effort.  Only after the moves spring to mind does the master actually start consciously thinking about them – to decide which of the candidate moves is best.

As Kahneman writes:

[Experts’] intuitive judgements come to mind with the same immediacy as [a child’s] “doggie!"

An Analogy

It may seem hard to believe, that many of our most difficult mental tasks are performed instantly and unconsciously.  As Kahneman points out, their very automatic-ness leads us to overlook their importance. 

When I was reading his book, I found it helpful to recall the university paper I took in artificial intelligence.  There we learned about artificial neural networks. Neural networks implement an approach to computation which is inspired by the structure of the human brain. They are very fast and effective “pattern recognisers” but, having recognised an input as matching a particular pattern, they are completely unable to tell you why it matched the pattern. I.e. they can’t explain their “reasoning”. This exactly matches Kahneman’s description of our automatic thought – it gives you an impression/hunch very quickly, but is unable to tell you why. So:

• I imagined the brain’s underlying hardware as a neural network – quick to recognise patterns, but unable to explain its logic.  It is here that our automatic thinking takes place.  So it’s easy to see why our automatic thinking is so fast – neural networks are naturally fast pattern recognisers, and ours is implemented directly in “hardware”. 
• I imagined effortful thought as a simulation running on top of the underlying hardware.  Rather like a virtual machine.  Inside the virtual machine, thinking is sequential and logical.  But, because the virtual machine is just an emulation, it runs slowly and has limited working memory.  (Question: does this imply our effortful thoughts aren’t “real”?  No. Just that they have a complicated origin, but our automatic thoughts are not subject to the same performance limitations.)

I wouldn’t dare to suggest that this model is accurate in terms of the underlying biology, but as a software engineer I found it helpful in understanding Kahneman’s work.

Developing Expertise

Consider two chess players, a novice and a master.  The novice relies almost entirely on effortful thought.  “How does a knight move?”, “Let me think what would happen if I moved my rook to here…”.  But the master makes heavy use of automatic thought.  The master’s automatic mind takes care of most of the details that the novice struggles with, leaving the master’s effortful mind free to add value where it is most needed.  Consequently the master can produce better decisions in less time.

But how does the novice become a master?  How does someone using effortful thought slowly transition to creating (good-quality) automatic thoughts?  Through practice.  With enough time, and enough examples, your underlying neural network trains itself to recognise the patterns.

Three sources have influenced my thinking about this kind of practice:

• James Bach’s Myths of Rigour presentation (which to me, could equally well have been entitled “What is learning?”).  I can’t find the audio or video online, unfortunately.
• Anders Ericsson’s work on deliberate practice
• Alistair Cockburn’s concept of Shu-Ha-Ri.  To join the dot’s between Shu-Ha-Ri and Kahneman’s work, I’d suggest that the Shu-Ha-Ri progression equates to the same progression described above: from a novice who uses only effortful thought, to an expert who uses a highly efficient blend of automatic and effortful thought.  

A Software Example

I recently noticed how various people debug the large software solution that I’ve been working on for some years.  When I need to debug it myself, I don’t get “stuck”. I might not know the cause of the problem, but I always have an idea on what to do next – something that will take me one step closer to finding the problem. Just like experienced players instantly “see” good moves in a chess game; I tend to “see” good moves in the debugging process. A series of such moves, possibly with some backtracking, eventually leads to a solution.  But I’ve noticed junior developers are much more likely to get stuck – when faced with a problem they sometimes have trouble generating the “next move”.   I’ve also noticed that although I generate moves more successfully than juniors, I seem to expend less conscious effort in doing so.

Of course, this is no excuse for me to take an ego trip – after about 7000 hours with this code base, and a decade’s experience on other systems before it, I darn well should have automatic expertise by now!  Junior programmers, or senior ones with less experience of the system at hand, have no choice but to generate fewer moves automatically and fill the gaps with effortful thought.  Over time, their balance will gradually shift from effortful to automatic.

Implications

The science of expertise has many implications for how we recruit, train and deploy software engineers.  In a future post, I intend to explore the implications for pair programming.

The key principle of the book is that conversations work best when everyone is able to “add their meaning to the pool”.  I.e. to share what’s on their mind.  Conversely, conversations work poorly when people withhold information and viewpoints:

When people purposefully withhold meaning from one another, individually smart people can do collectively stupid things.

Therefore, successful dialog depends on two learnable skills: (a) helping other people to share what’s on their mind, and (b) being able to share what’s on your mind even in difficult situations.  I’d like to share a recent example of the latter…

Our (excellent) manager was describing a change to our processes.  I hated the idea. It seemed bureaucratic, time-consuming and just plain wrong.  As he said how much “we need to do it”, I found myself giving up.  My instinct was to withdraw from the conversation.  If he was so sure, what point was there in discussing it?  He wasn’t actually asking for our opinions.  And besides, everyone else seemed to agree.  At least, they were sitting there nodding. 

But just in that moment of giving up, I caught myself. Sharing meaning is good, right?  Yes!  And even fun?  Yes, as I have discovered. So in that moment I decided to take the enjoyable, productive route, instead of the grumpy silent one. I said something like this:

I feel a bit concerned about the cost of doing this.  On <project X> we deliberately avoided gathering this data, and it was fine.  We used high-level figures, instead of the detailed breakdown we’re talking about here, and …”  I went on to briefly expand on our experience on <project X>.

Instead of giving up, I had managed to share “my meaning” with the group.  It wasn’t so hard.  It didn’t seem like I’d offended anybody, and I felt much better within myself for having shared. 

(Incidentally, talking along the lines of “I feel concerned about…” is a useful collaboration technique that I’ve picked up along the way.  It works much better, and is more honest and more informative, than bluntly saying “I don’t like that”.    Another handy technique is to talk in terms of concrete stories, as I did about <project X>.  Both these techniques deserve blog posts of their own one day…)

So did it work?  Did adding my meaning to the pool actually advance the discussion?  Yes it did.  The boss replied with a clarification of his underlying goals.  Which was great, because then we were able to think about all possible ways to meet that goal – instead of just debating between his original plan and my suggested alternative.  We haven’t finalised the decision yet, but it looks like we’re going to end up with something better than either of us had imagined.

I’m also thrilled to see that one of these speakers, Barbara Fredrickson, comes from outside the agile community. I’m a firm believer that agile can learn a lot from outside the agile community, so it’s great to see that happening at our largest conference.

1. Important, and
2. Learnable (with time and practice)

The video is an edited extract of my talk at the AgileRoots conference, at Salt Lake City in 2009.  For the benefit of people who, like me, prefer skim reading to watching videos, this post also includes a written transcript (below).

(or click here to open at YouTube)

(Update 2011: I now have an expanded version of this talk, prepared for a recent APN meeting. The new talk goes into detail about some concrete actions that we can take to improve our people skills. Hopefully I’ll find time to write blog posts about some of the details… Here’s one.)

Transcript

(Transcript matches video, but has had headings added, and some minor grammatical tidy-ups)

My Own Story

I started off as the stereotypical shy nerd when I entered the industry [in 1995] and I had a realisation that, if I carried on like this, it wasn’t going to be very good for my career.  So I said to myself, “You’ve got to do something about this”.  So one of the things I remember doing was forcing myself, every morning, to say hello to the receptionist – and that felt like a challenge for me.  (I hope that most people in the room are starting from a less “challenged” starting point ;-)

Your People Skills are Changeable

The big thing that I’ve learnt from my personal journey, and from some of these fields that we’re going to talk about today, is the level of change that you can undergo personally, in your skills and enjoyment of this side of work.

…[As] Alistair [Cockburn] mentioned earlier, we should study psychology.  As best I can, as someone with a full-time job as a nerd, I have tried to study some of these things in my spare time.  And I’ve been really encouraged to find that this notion of, ah, these key skills and aspects of our personality,.. I’ve been very pleased to discover that they are far more changeable that I had ever thought they were.  And probably most people in IT don’t realise how changeable these things are.

Examples of “Personality Change”

One of the fields that touches on this area is this thing called Positive Psychology (which is this really seriously-cool thing, which I don’t have a lot of time to talk about today).  One of the foundation books in that field is a book called “Learned Optimism”.  It describes how you can permanently change your level of optimism.  Wow! Who thought you could do that?  Before the book was written, who thought that was a valid and useful thing for a person to do?  So the level of change in this area is potentially very large.

Now you can’t change everything.  The corresponding finding on happiness, from Positive Psychology, is that you can change about 40% of the happiness that you have in life.  The other 60%, you probably can’t do anything about.

The Agile Movement is Missing the Boat

In another aspect of my personal journey, not only have I had to learn the stuff I’ve just been talking about, but I’ve had some real ups and downs in the successes of my interactions with people at work.  Some have been very successful, and some have been just complete train wrecks.

So I thought OK, Agile says it’s about “Individuals and their Interactions” – surely the Agile movement can tell me how to do this stuff.   So I jumped on Google and I found Alistair’s wonderful collaboration page… and nothing else.  There are thousands of things on tools and processes, thousands of things that tell me how to do TDD – various factions of TDD – the mockists versus the classicists, and the factions of the mockists.  But when the agile manifesto was put together on the hills up there [above Salt Lake City], 8 years ago, it was saying “Individuals and Interactions” is more important than processes and tools.  I certainly agree with the person who said earlier in the panel discussion (Mike?), “Is Agile broken?”.  In this sense I think it is broken.  We’re not putting emphasis in the right place here.  We’re talking and talking and talking about processes and tools, when Individuals and Interactions is the thing that makes the difference, and we’ve always said it’s the thing that makes the difference [we just haven’t done very much about it!]

Who’s Not Missing the Boat?

So, I started to look elsewhere [outside Agile].  And I discovered this thing called Organizational Behavior, which defines itself like this: “the study and application of knowledge about how individuals and groups act in organizations”.   I read that definition and thought, “That’s individuals and their interactions.  That’s the thing I’m looking for.  That’s the thing that Agile’s not telling me about.”

There’s Positive Psychology [as I mentioned earlier] and there’s this very new thing where the two meet, called Positive Organizational Psychology [aka Positive Organizational Behaviour].  There’s not a lot of stuff out there yet, but it’s touching on some of what I’ve talked about today, these notions that you can learn all sorts of stuff that’s relevant to your ability to have a successful interpersonal relationship at work.

“Political Skill” at work is a fascinating thing.  Now, normally “political” has some kind of negative connotations but basically what these [researchers] are talking about is the “good side of the force”, so to speak, and the idea that you can learn how to influence people, for good, at work.  The interesting thing that they teach – and I was relieved to find it, because I wanted to tell you guys that it was true, so I had to find out whether it was ;-) – is that you can learn this stuff: being able to influence colleagues or managers more successfully than you do now, being able to sell an idea or interact is, by and large, a learnable skill.

A Wake-Up Call to the IT Industry

And yet, so much of the industry, including the part of the industry that we’re in, is not necessarily investing enough effort into it.  It’s perfectly normal for geeks to talk about doing their Microsoft certification, or their Java certification, but who ever talks about learning this stuff?  Who even knows that you can learn it?  If you don’t think you can learn it, of course you’re not going to talk about it, of course you’re not going to try.

And so one of the things that I hope comes out of this talk here today, is that when we go back to our various places of work, y’know, there’ll be people sitting on adjacent desks who are like I was 10 or 15 years ago – geeks who think they can’t do this, and don’t know its learnable.  If we can take some of this message back to them, I think that could be a really powerful thing.

What Does It Actually Mean to Learn This Stuff?

There’s this guy [Anders Ericson] who’s done a lot of research on expertise: what does it take to become an expert in tennis, in some other sport, in computer programming, or in interpersonal interaction?  And what he’s found is that there are several key notions.  The one I’m going to single out today is a thing called “Deliberate Practice”, which basically means not just accumulating a lot of years of experience, but learning. So, for example, if I go down to the tennis court, and I do the same bad serve 20 times in a row, I haven’t learnt.  If I do my bad serve the first time, and I reflect on how I could do it better, and I do it slightly better, and I keep repeating that cycle, that’s deliberate practice (at least, in my naive understanding of it).  The thing he finds, is that it does take time.  The same research said it takes 10 years to become an expert.  I’ve talked a bit about my journey in this area, and I’m not an expert yet, but certainly it’s taken me a number of years.

So this is not a thing you take away and “sha-bang” it’s wonderful!  This is a process you can take away, and start to put into place, and it will bear fruit over the course of months and years.

On Being Yourself

When I first started trying to learn this some time ago, I read a book on how to be a leader in a technical field, and I felt like it was saying, “John, you’re like this, and good technical leaders, well, they’re like that”. It was kind of offensive, to suggest I should have some kind of personality transplant, and it seemed unlikely to succeed.  So it really put me off, and I actually said, “Well, I don’t want to be a technical leader then”.

The interesting thing that comes out of the management research, completely independent from Agile, is that the best managers are not trying to conform to some stereotype of who a manager is. They are being themselves, more skilfully.  This is a learnable skill, known as Authentic Leadership.  It takes the pressure off.  If you say to yourself, and it’s true, that, “The goal here is to simply be a more skilful version of myself”, the pressure’s off, because you’re not trying to give yourself a personality transplant.  You’re not trying to conform to something that’s completely different.  You’re actually trying to do the thing that will work best for you, and happens to be the best way to lead and motivate anyway.

Who is this Message For?

This is not about the people who are on your org chart.  This is about the people who are not even on your org chart: people who sit cutting code, people who sit at their desks and don’t talk to their fellow developers, and people who sit at their desks and talk too much and push their opinions around.  (I’ve been in both camps, and I’m trying to learn to step out of them into more positive territory.)

This is about empowering the team.

Learning by Reflection

This is not necessarily about training courses.  The great thing about this deliberate practice stuff, is that you can learn it by yourself.  One of the interesting stories I came across was a CEO, 35 years old, in charge of this big company, and he got there on the back of his interpersonal skills.  They asked him, “How did you build those skills?”  And he said, “After every meeting, when I came out of the meeting, I’d come back to my desk, I’d get a notebook, and I’d write down lessons learned from that meeting (in terms of interpersonal interaction)”.  He basically never read this note book, but it was a mini personal retrospective thing (like we do in Agile at the end of each iteration). This was a little thing after each meeting and it took him to a point where he was in this very important position, on the back of his interpersonal skills.  For me, I haven’t gone down the notebook route, but I’ve certainly found that deliberate practice, even without a mentor or a buddy [to discuss things with] is a powerful, powerful thing.

So you can learn this.  Teams can learn this, individuals can learn this.  You don’t need high-flying consultants.  You don’t need team building courses with ropes and swings and bridges and stuff.

Conclusion

It does take time.  There’s an element of time and patience involved, but there’s a great, great payoff at the end.

Evidence-Based Management

All the key resources can be found at evidence-basedmanagement.com. The site gives an overview, links to books by Jeff Pfeffer and Bob Sutton (the key authors in the field), and also contains some podcasts that make a great introduction to the ideas. Note that their focus on adapting to your particular situation is similar to “situationally specific processes” in agile’s Declaration of Interdependence.

Principled Negotiation

The classic book is Getting to Yes by Fisher and Ury (eBook). I’ve described here how it relates to negotiating scope, and defining a situationally-specific process.

Qualifications-Based Selection aka Competitive Tenders without Prices

I’ve written an overview in the second half of this page, and find this FAQ very relevant (even though it is written about architecture and engineering rather than software).

The Winner’s Curse

I’ve posted an overview here, which includes links to the (few) resources that I’ve been able to find on how the Winner’s Curse relates to Software

Individuals and Interactions

In the talk I tried to break this down into four key themes, as follows:

1. Important

My favourite book highlighting the importance of these issues is In Good Company by Cohen and Prusak, which focuses on the topic of Social Capital.

2. Learnable

The “learnable” theme is supported by a collection of different resources.

On the general notion that we can change more than we might realise, I’d cite the literature of Positive Psychology as an example – even though it focuses on topics like happiness and optimism rather than workplace interaction. (E.g. Learned Optimism by Seligman and The How of Happiness by Lyubomirsky)

On the particular topic of changes in personality, either “real” or “indistinguishable from real”, check out Personality Trait Change in Adulthood (pdf) by Roberts and Mroczek (particularly the section on individual changes), plus a couple of podcasts which I’ll post links to shortly. (The podcasts contain the examples I cited in my talk, but I’ve temorarily misplaced the links!)

And for learnabilty of sensitivity to interpersonal dynamics, see a brief section in the middle of this podcast from Rob Goffee.

Admittedly, of all the themes in my talk, learnability-through-practice is perhaps the least supported by research and literature. I suspect this reflects a relative lack of interest in learnability on the part of academics, particularly in those courses which emphasis the traditional HR skills of screening people at recruitment time and then training them with short, focussed training courses. The process that I talked about, which consists of gradual self-directed learning over months and years, seems to have received less attention.

3. Deliberate practice

This was the key learning technique that I focused on in the talk. My views are inspired by the work of Anders Ericson, who coined the term “Deliberate Practice”.

4. Authenticity

The concept of Authentic Leadership can be summarized as “Be yourself with more skill”. That description comes from the book Why Should Anyone Be Led By You? by Goffee and Jones (pdf eBook).

It’s also worthwhile to make particular mention of political skill, which is one aspect of workplace interpersonal skill. “The” book on the subject appears to be Political Skill at Work by Ferris, Davidson and Perrewe, although I also found some of the chapters in Positive Organizational Behavior by Nelson and Cooper (ebook) to be helpful and relevant.

Finally, Alistair’s post on collaboration is an excellent resource (and a topic worthy of more attention, both from academics and the agile community).

- via Bob Sutton

From the back of the book:

“In Good Company is the first book to examine the role that social capital — a company’s “stock” of human connections such as trust, personal networks, and a sense of community — plays in thriving organizations.”

As the authors readily admit, some of it seems like common sense — but it happens to be the kind of common sense that is too commonly ignored! And it’s not all common sense, I learnt some important things which I’d never noticed before.

For those who don’t know, James Bach specializes in testing, but most of what he says is applicable to agile in general. He covers the “people over tools” point very well. As I’ve written before, the importance of people over tools is incredibly important, but seems to have been overlooked by much of the agile community. (Perhaps that’s because it requires on-going effort and learning, in an area that doesn’t come naturally to many of us geeks, and so it doesn’t look like a silver bullet.)

James’ own blog post on the video is here.