Agile with Fixed Scope

It’s a common misconception that agile processes can’t be used with fixed scope.  But a misconception is exactly what it is.   A number of the founders of the agile movement developed their forms of agile on fixed-scope projects. As I write this, I’m working myself on an 18-month project with about 20 people and a fixed(ish) scope (see below).  So it can be done.  But how?

There are several different strategies you can use:

Strategy 1: Fix the scope and flex the price

This keeps scope management very simple, you just build all of it.  The catch is it may take longer than you expected, so you may need to flex the price through a time-and-materials contract or some kind of sharing of financial risk.  Understandably, this risk of cost overruns renders this simple approach unsuitable in many environments.

Strategy 2: Work in priority order and stop when the money runs out

(Admittedly, this is not exactly fixed scope.) This is very commonly recommended on agile projects, too commonly in my opinion.  But again, it has the virtue of being relatively simple.  Do the most beneficial stuff first, leaving the least beneficial until last.  When the money runs out, just stop and don’t do the rest.  Agile makes this approach possible – but not mandatory.

Strategy 3: Implement remaining features more simply when short of time

There are many factors that influence the effort required to develop a feature (or user story, depending on your terminology).  Some of those factors are probably under your control: e.g. How extensive is the validation? How much effort do we put into optimising the user experience (UX) and appearance?  Do we fully automate everything, or do we allow manual overrides so we don’t have to code every single  edge case?  Can we think of something that would save development time, and still meet the overall business goal (albeit in a different way from what was originally expected)?

If you are using good earned value tracking you should know, within the first quarter of the project, whether you are likely to run out of time at the end.  Once you find that out, immediately start seizing all these opportunities to simplify the remaining 75% of the project.  And, because you have good earned value tracking, you can justify the simplifications to your stakeholders.  The aim is to deliver all of the planned business benefits, just with simpler implementations than might have been originally expected.

We’re using a variant of this strategy on my current project.  We built the highly-used parts of the system first, taking a lot of care with their appearance and usability. The second half of the project consists of functionality that is much less commonly used, so here usability and appearance are much less important. (If it takes a user a few extra minutes to do something, it doesn’t really matter if they only do that thing a few times each year.) So for the second half of the project, we have consciously shifted our design approach away from ease of use, and towards simplicity of implementation.  Because we were using earned value-like tracking, we can justify this change of approach to users and management.

Strategy 4: Split each feature (or user story) into essential and nice-to-have parts

This a refinement of the previous strategy. Right from the start of the project, you split features/user stories into two pieces: an essential minimum piece, which you implement early, and nice-to-have embellishments (such as advanced data validation or visual styling) which you defer to the tail end of the project.  If you run out of time, you drop some of the embellishments from the tail, and still deliver a working system with the full scope of capability/functionality.

Strategy 5: Make multiple passes over each story, doing the basics first and then improving it later

Similar to strategy 4, but you may “visit” a given user story 3 or more times within the project, instead of just twice as in Strategy 4.  I like this in theory, but in practice I think it’s too hard to used earned value or burn-chart tracking in this strategy.  Whereas in strategy 4, I feel that earned value remains (just) feasible.

[Tim Wright's comment,  below, gives more details on how this strategy can be done]

Summary

The last three strategies are all variations on a theme. Within a single project, you may use several of them, and maybe also resort to strategy 2 for a few user stories.

I recently heard the phrase “value management” to describe the work of deciding not only what to build, but also how simply or thoroughly to build it  The aim is to meet the business goals with the optimal expenditure of effort – i.e. do what needs to be done, without overspending on superfluous details.

Further Reading

All of the following are excellent.

Alistair Cockburn’s Trim the Tail.  A rich explanation of the theory and practice of strategies 4 and 5, with significant additional benefits in risk management.

Alistair’s list of related strategies.

Jeff Patton’s concept of Feature Thinning: Jeff’s a leading practitioner of Strategy 5. See: Finish on time by managing scale, Difference between incremental and iterative, Accurate estimation = red herring  Jeff has often used these techniques on fixed-scope, fixed-price projects.

Description from an agile company called Atomic Object, of how they operate with fixed budget and controlled (rather than fixed): here and here.

Martin Fowler’s Scope Limbering

The opening section my my own agile earned value (pdf) has more info on why fixed scope is a valid option in agile.

I’ve also posted a summary of estimation tips for agile projects.

QBS and Novopay

Here in NZ, we’ve had a lot of press recently a Novopay, which is a system to pay school teachers that has suffered from significant, embarrassing (and very widely-publicised) problems.

I was recently asked to write an article on the subject for the Institute of IT Professionals (IITP), explaining how I believe the emphasis on price, in tender evaluation, is probably a significant root cause of such failures – and how there is a much better way to structure tenders.  The article seems to have been very well received – with one (male) reader even going so far as to tweet, “Who is John Rusk and can I have his babies?”  (Which is both the best and the weirdest feedback I’ve ever received!) Continue reading “QBS and Novopay”

Alistair Cockburn in Wellington

If you work in Wellington, NZ, you have an opportunity this month which may not come your way again for a long time. Alistair Cockburn, one of my all-time agile heroes, is visiting in Wellington.  Not only is he speaking at the Agile NZ conference, and running courses before and after the conference, but he’s available for half-day, full-day or multi-day consultation at your workplace.

Alistair has a great approach to  agile, extremely well-informed without becoming dogmatic.  I’ve described more about his approach to agile, and how important it is, here.  When you talk with Alistair, he makes you think – and he also makes himself think (so you get advice tailored to your particular situation).  At the risk of sounding like some book’s dust jacket: Alistair is co-author of the Agile Manifesto, co-author of the Declaration of Interdependence (great idea), co-founder of the International Consortium for Agile (I really like their approach to agile training), author of 5 books from use cases to agile development, and a specialist in agile requirements gathering, development, methods, and organizational processes.

I asked Alistair to describe what he can offer to Wellington teams, and he replied:

    • align different parties’ views on what agile is and how it fits into the company (e.g. 1/2 day)
    • organizational "blood-pressure" check on adoption of agile and unaddressed ailments in general (2-3 days)
    • coaching on use cases, user stories, project management, agile adoption (1/2 – 2 days)

He’s done this kind of work for many other teams around the world.  If you’re interested in tapping into his expertise while he’s in town, email Alistair directly. To prevent it being harvested by spam bots, I’ll describe his email address to you as follows: the part before the ‘@’ is ‘TOtherAlistair’ (short for The Other Alistair) and the part after the ‘@’  is ‘aol.com’.

By the way, also in Wellington this month, and speaking at the conference and at AgileWelly, is Jeff Patton.  Jeff’s an excellent speaker on defining and managing the scope of agile projects.  Both Jeff and Alistair have been heavily involved in the agile community in Salt Lake City, Utah, which seems to be a breeding ground for great agile ideas.  I don’t know whether Jeff is available for consulting work while he’s here, but if you’re struggling with defining or scoping agile projects, get in touch with him and ask!

Finally, Alistair’s surname is pronounced “Co-burn” the Scottish way.  I don’t know of any other Scottish words with a silent “ck” :-)

Faith, doubt and evidence – agile as religion versus agile as social science

My first exposure to Agile was in the early 2000’s, when a friend leant me the Schwaber and Beedle Scrum book.  It kept talking about mysterious  “emergent properties”, which were apparently the good things that would inevitably happen when you followed the process.  But it never really explained why those properties would emerge and, to my mind, it never even gave satisfactory evidence that they really would emerge.

That was my first taste of agile as a belief-based movement.  It’s dangerous ground, a philosophical quicksand which has grown over the following years – to enthral some and to alienate others.  I vividly remember a presentation, again on Scrum, which I paid to attend some years later. It reminded me of something which no professional event should ever resemble: a recruiting session for a strange religious cult.  The kind where the charismatic speaker sways all listeners with his eloquence and fervour… then gets arrested three years later for embezzlement and child abuse.  (I once stayed in the commune of just such a cult.  After one uneventful yet unsettling night, I was glad to see the last of it.  Years later, the leader was indeed arrested!)

Agile as a belief system

At its worst and most cult-like extremes, belief-based agile can be dismissed out-of-hand, and rightly so.  But at the milder end of the scale, such as the Schwaber-Beedle book, it serves a valid purpose.  It presents a clear and compelling introduction to agile, which is great for people who have never heard about it before.  It worked for me.  My team tried what we read in the book, and we loved it. It was great to have such a simple introduction to agile.  Without that simplicity, we might never have got started.

But then we started to struggle.  Our projects didn’t seem to fit the book.  We had constraints which the book didn’t even mention.  I began to suspect Scrum was good for other people but not quite right for us.

But what bothered me more was that I still didn’t understand it.  What the heck are these so-called emergent properties anyway? And how do you force them to happen? ;-)

Agile as (social) science

I wasn’t until I read Alistair Cockburn’s book Crystal Clear that things started to make sense.  The book is not about Scrum, it’s about Alistair’s rather different formulation of agile (more on that in a moment).  And yet, it was only by reading Crystal Clear that I actually understood Scrum.  Although it uses different terminology from Scrum, Crystal Clear filled in the missing pieces and answered my questions about Scrum’s “emergent properties”.

But Crystal Clear did three things which even more important:

  1. It showed that agile can be based on evidence and science.  (Although, when you base it on science, agile turns out slightly, yet significantly, different from the “agile” you may be accustomed to.)  I’m not talking here about the popular misconception of science-as-blind-faith-in-experts, but about the true nature of science where even the experts question their own judgement and are determined to follow the evidence where ever it may lead.  Crystal Clear grew out of Alistair’s PhD research (which in turn grew from his process work at IBM and elsewhere). Based on observations of real-world projects, Alistair formed hypotheses about the factors that led to success in software development.  Then he painfully discarded one hypothesis after another, as the evidence demanded.  He eventually arrived at the family of methodologies known as Crystal, of which Crystal Clear is the version applicable to small teams.  (Alistair’s development of Crystal preceded the founding of the agile movement.  He was later one of the 17 co-creators of the Agile Manifesto.)
  2. It showed that agile should be flexible, to suit the nature of humans.  It is impossible to read Crystal and not be struck by how human-friendly it is;  how it allows, and in fact requires, variation to meet your circumstances; and how one-size-fits-all rules are the very antithesis of agile.
  3. It explained that simple belief-based formulations of agile are the starting point, not the destination.  The common rigid, belief-based formulation of agile is an ideal learning tool, but it’s only the starting point on the learning journey, not the end.

Conclusion

Today, I happily use Scrum – informed by and infused with a healthy dose of Crystal Clear.  If you are not familiar with Crystal Clear and Alistair’s work in general, you owe it to yourself, and your project, to learn.

Seek a broader perspective and, at the risk of being facetious, don’t stay in the commune until the leader gets arrested!

Challenges for 2013

Reflecting on the end of another year, I feel that the interesting challenges in Agile are not about process, but instead about people.

Promoting and Enhancing People Skills

This has been a theme of this blog for sometime. The challenge for 2013, to myself and to anyone else who’s interested in the Agile community, is, “How can we encourage the development of better skills in this area?”  It’s one thing to highlight the fact that the Agile community seems to have forgotten its original value of “People over process”, it’s another to actually put the people-first approach into practice.  I see a lot of potential in the ideas of Chris Argyris, and in the techniques described in Crucial Conversations.  But how do we translate those ideas into our day-to-day work?  How do we encourage and help other people to use those ideas?  And, what other ideas should be be also taking on board? 

Knowledge vs Skill

We tend to think that, if we can just know the “right” thing to do – the right way to unit test, the right way to plan an iteration etc – then the problem will  be solved.  But much of human performance is not just about knowledge, it’s also about skill.  On an intellectual level, I “know” how to play tennis – but I seldom translate that knowledge into a winning performance on the court. 

When we look at the way we, and our team, are performing on a software project, it’s tempting to treat any deficiency as a deficiency of knowledge – something that could be addressed by training.  But really there are at least 5 different aspects to performance:

  1. Knowledge.  You can get this through training, or you can follow a defined process.  I think we put too much of “agile” into this bucket.
  2. Short-term skill: skill that can be developed, through practice and experience, during the course of the project.  Investing in such skills early in the project will pay off during the course of the project.    But it’s hard to draw the line between this and the next….
  3. Long-term skill development.  This is deep expertise, which takes about 10,000 hours to develop.  You need at least some of your team members to have such expertise,  since without it your project will be much more difficult.  But since this kind of expertise takes years to develop, they can’t get it during the project.  They must have it before you start.  (However, just because a person is experienced, that’s no guarantee they have this kind of expertise.  Some experienced people do, some don’t.  Simplistically, its the difference between learning from one’s mistakes, versus merely repeating them.)
  4. Innate general mental ability: this is the “RAM capacity” and “CPU speed” of the person’s brain.   To suggest that these qualities differ between people is both politically incorrect and self-evidently true.
  5. Conscientiousness – the personal, internal motivation to work carefully and produce a successful outcome.  Research suggest conscientiousness plays a significant role in job performance.

For agile coaches, I hope this list poses a challenge to you, to extend your practice into the longer-term aspects of skill (#2 and #3).

For team managers, I hope this list underscores the importance (and difficulty!) of recruiting well.

For all of us, I hope this list reminds us that #3 on the list, expertise, is something that we must take individual responsibility for developing, within ourselves. It’s important because it has more impact on our success than short-term knowledge (#1) and skill (#2).  It is worth investing in because it is probably easier to to change your level of expertise than your level of intelligence (#4) or your personality (#5).  But, we must take responsibility for it ourselves, because the 10,000 hours it takes to develop will probably span many different jobs, and so can’t be left to any one employer.

Expecting and Accepting Individual Differences

When a doctor gives medication, she expects that different patients will respond to the same drug in different ways.  Some may require higher doses than others.  Same may have an adverse reaction to the exact same drug.

So why then, do we fall into the trap of one-size-fits-all agile?  Individuals differ greatly in their mix of the above 5 factors that contribute to performance.  Furthermore, the projects we work on, and the organisations we work in, also differ greatly.  So a given “prescription” for how to build software may be appropriate for some but not others.

The easy part here is to recognise the project context.  It’s easy to look around and see that projects are different.  The hard part is to recognise the differences within us.  When I read code, what goes on in my head may be very different from what goes on in my colleagues head – but of the two brains working on the problem, I only get to look inside one, my own.  It is naive (but common!) to assume that my colleague’s brain experiences the problem in a similar way to my own.  (By the way, I’ve found Advocacy and Inquiry helpful here.  It’s helped me to enquire not just into what other people are thinking, but into the data and evidence that underlie their thoughts – data and evidence that I might not have seen.  This can be as simple as asking questions like “Could you give me an example?” or “What happened when you tried this in the past?”)

Conclusion

So, these are the issues that I find most challenging and relevant as 2013 dawns.  Can you suggest any relevant links, resources or thoughts? 

Become practiced in making changes

Agile has always embraced change.  But businesses are still be tempted to minimise those changes, by trying to get things “right” first time. 

On my current project, we’re realised that minimising change is not necessarily a good thing. Why? After the system goes live, we must be able to change it.  (In our opinion, an unmaintainable system would be a failure.)  So… if we are going to successfully change the system after go-live, then we must practice making changes before go live.  Without such practice, how would we know whether the architecture is maintainable?  How would we know whether the team has the necessary skills and confidence to safely change the code?

So each mistake, dead-end, or change of direction has a silver lining: it’s a chance for us to practice the essential skill of changing our code.  We need these mistakes to happen, so we can practice correcting them.

Are you pair-programming in New Zealand?

I’ve just been engaged in very interesting discussion of pair programming with Arlo Belshee.  Unlike me, Arlo has lots of successful experience with pairing.  If you’ve been following my recent posts on pairing, I definitely recommend that you check out Arlo’s new comments on them, and also his own post and its comments here.

He shows that pairing is a form of expertise, that takes time and effort to develop, just like any other type of expertise.

Furthermore, I would suggest, it’s very hard to imagine successful pairing if you’ve never actually seen it.  Which brings me to the point of this post: is there anyone here in New Zealand who is doing pair programming and who might be interested in letting myself and a couple of colleagues see how you do it?  We work for a registered charity (so we’re definitely not one of your competitors).  I’m hoping that, like many agile teams, you’d like to share your success stories.  If so, I’d love to hear from you.  My email address is on the contact page.

Looking for a Senior Tester

At work, I need a senior tester for the TBfree New Zealand programme. We’re world leaders in our field, our work helps to safe-guard New Zealand’s biggest industries, and along the way we happen to protect lots of native wildlife from predators.

Do you have strong experience in agile testing? Or are you a good senior tester who wants to learn agile? Either way, we’re keen to hear from you.

Our project is just getting started. If you’d like to be part of a friendly, co-located agile team in central Wellington (NZ), let’s talk! The best way to contact me is via email.

Advocacy and Inquiry

Have you ever wondered why it is so hard to achieve real change in organisations?  Have you ever wondered why organisations don’t really seem to learn?  Have you ever feared that, in spite of all the talk about change, it will really amount to nothing in the long run?

There’s little-known answer to these questions, backed by several decades of research.  It’s not pretty, since it shows the true reasons why these problems are so incredibly hard to solve.  Fortunately, it also offers a solution.  The solution is difficult, requiring much commitment, patience and humility.  …But then, if you think about it, you already knew there wouldn’t be an easy solution, right?  :-)

In a nutshell

The heart of the problem is that we don’t learn because we don’t communicate effectively.  When humans communicate, we tend to value these things:

  • achieve my goals (as defined by me)
  • win, don’t lose
  • avoid triggering negative emotions in myself or others

Prof. Chris Argyris of Harvard University has found that these core values, or “governing variables” as  he calls them, underlie most human communication.  People espouse all sorts of ideas about we ought to communicate, but when Argyis and his colleagues observed how people actually do communicate, they found these same values occurred over and over again – across cultures, across genders and across many thousands of people.  I don’t know whether these values are encoded in our DNA, installed in us by our culture(s), or just a pattern that our minds subconsciously settle upon.  In any case, they form a consistent pattern of human communication, which I will refer to here as the “Default Communication Style”.

One consequence of the Default Communication Style is that, in trying to prevent triggering negative emotions, people unilaterally protect themselves and others.  Rather than seeking the other party’s true thoughts, via genuine dialog, a person will guess or presume the other person’s thoughts and never put the guesses to the test.  So we end up walking around our offices assuming that certain people have “bad” goals or intentions, without ever testing to see if our assumptions are actually true. (Often they are not).  Wikipedia describes a few familiar symptoms: withholding information, creating rules to censor information and behaviour, and holding private meetings (aka talking about people behind their backs).

The values of “win; don’t lose” and “achieve my goals (as I define them)”  also hinder the free and full exchange of information.  All in all, the Default Communication Style causes people to withhold relevant thoughts and information.  So, deprived of full information, our organisations fail to learn.

What’s the alternative?

Argyris suggests that we should adopt the following value set instead.

  • give and receive valid information
  • favour informed choice for all concerned (as opposed to unilateral control)
  • mutual responsibility for “looking out” for each other

So we become more concerned with sharing what is on our minds, and equally we become more concerned with helping others to share what is on their minds. (Even when its different from our own views.).  The best terms I’ve heard to describe this are “Advocacy and Inquiry” (*).  Advocating our own thoughts with skill and openness; and inquiring into the thoughts of others with skill (again) and curiosity.  We want to get the full picture flowing both ways: out of our own heads and into the group’s consciousness, and equally out of each other member of the group and into our own awareness.

When engaging in Advocacy and Inquiry, we should not be hamstrung by the desire to avoid causing offence.  That’s not an open licence to be deliberately offensive, but rather a recognition that our normal cop-outs are self-defeating.  When we avoid raising awkward issues, we ultimately let down ourselves and also the person who our silence is designed to “protect”.

More evidence

You might not have heard of Argyris’s work before.  But you probably will have heard of other people who have independently discovered the same ideas.  For instance, the authors of the book Crucial Conversations  followed around individuals who were good at “getting things done” in the workplace. The authors analysed the behaviour of these communication stars, to find out how they did it.  The answer centres around adding one’s own contributions to the group’s “pool of meaning”, and helping others to do likewise. I see Crucial Conversations as a excellent “how to” manual, for Advocacy and Inquiry.

The classic guidebook to negotiation, Getting to Yes, emphasises sharing of valid information instead of “winning” one’s opening position.

Bob Sutton of Stanford writes a great blog, and summarizes the same ideas with the phrase

Argue like you’re right; listen like you’re wrong

Why is it so difficult?

There’s a danger that all of the above will sound like common sense.  And to some degree it is.  Its easy to look at, and say, “Yes, I think that’s a good idea”.  It’s much harder is to actually put it into practice.  When it comes to the Default Communication Style, we’ve each had a whole lifetime of practice.  So, it is deeply ingrained in us, and we use it instinctively.  And that is the real challenge: not simply to believe that Advocacy and Inquiry is the right way to go, but to actually act consistently with that belief in our moment-by-moment communication.

It is easy to fail.  Argyris found that people can agree with his ideas, but still have trouble putting them in to practice.  He calls this the difference between espoused theory (the theory expressed by what we claim to believe) and theory-in-use (the theory implied by what we actually do).  I recently experienced the difference first hand.  While having a conversation with a manager, I felt that the manager was becoming upset with what I was saying, so I backed away from fully expressing myself.  And what was the topic?  I was trying to say that we need more Advocacy and Inquiry!!!  In a conversation about these exact issues, I still slipped back into Default behaviour when the going got tough!

Why do we fall back into our old habits? Because we are used to them — very skilled in them, in fact.  We are much less skilled in Advocacy and Inquiry.  So, when the going gets tough, we unconsciously fall back to our old ways.

What do we need to do?

Practice.  Effective communication requires practice, just like playing tennis, reading music or programming a computer.  You won’t be perfect when you first start practicing, but everyone has to start somewhere.

Actually, we don’t need only practice.  It also helps to obtain some up-front knowledge of the topic.  Something to “seed” our practice, so to speak. Two great starting points are the books Crucial Conversations, which I see as a manual for Advocacy and Inquiry, and Discussing the Undiscussable, which is a beginner’s guide to Argyris’s work.

You can find additional resources at Benjamin Mitchell’s blog, and at Action Design.

Finally, I hope to post more on this topic myself over the coming months.  So stay tuned…  In the meantime, please leave questions, comments, and links to additional resources below.

—-

(*) I’m calling the two sets of values the “Default Communication Style”, for the first, and “Advocacy and Inquiry” for the second.  If you’re familiar withe Argyris’s work you’ll know that these are not the standard terms.  Argyris himself uses the terms “Model I” and “Model II” respectively – with a self-deprecating grin at the blandness of the names.  Other authors use the terms “Unilateral Control Model” and “Mutual Learning Model”.   Those are good descriptive terms, but I just can’t imagine someone standing up at the front of a team meeting and saying, “Hey guys, we need to promote the Mutual Learning Model!”.   It seems more plausible to say, “Hey guys, we need more Advocacy and Inquiry!”.

/p