Is software development an art, a craft, science, engineering, or something else entirely? Does it even matter?
Yes, it does matter, and it matters to you. Your actions and their results will differ depending on which of those is more correct.
The main thing is this: You want your software out soon and defect free, but more than that, you need a way to examine how your team is doing along the way.
It is time to reexamine the notions underlying software development.
The trouble is that as we look at projects, what we notice is constrained by what we know to notice. We learn to distinguish distinct and separable things in the extremely rich stream of experience flowing over us, and we pull those things out of the stream for examination. To the extent that we lack various key distinctions, we overlook things that are right in front of us.
We anchor the distinctions in our memories with words and use those words to reflect on our experiences. To the extent that we lack words to anchor the distinctions, we lack the ability to pull our memories into our conversations and the ability to construct meaningful strategies for dealing with the future.
In other words, to reexamine the notions that underlie software development, we have to reconsider the distinctions that we use to slice up our experience and the words we use to anchor our memories.
This is, of course, a tall order for any book. It means that some of the earlier parts of this book will be rather abstract. I see no way around it, though.
The last time people constructed a vocabulary for software development was in the late 1960s, when they coined the phrase software engineering, as both a wish and a direction for the future.
It is significant that at the same time the programming-should-be-engineering pronouncement was made, Gerald Weinberg was writing The Psychology of Computer Programming. In that book, software development doesn't look very much like an engineering discipline at all. It appears to be something very human-centric and communication-centric. Of the two, Weinberg's observations match what people have reported in the succeeding 30 years, and software engineering remains a wishful term.
In this book, I will
- Build distinctions and vocabulary for talking about software development
- Use that vocabulary to examine and anchor critical aspects of software projects that have been pushed to the sidelines too often
- Work through the ideas and principles of methodologies as "rules of behavior"
- Merge our need for these rules of behavior with the idea that each project is unique, and derive effective and self-evolving rules
I hope that after reading this book, you will be able to use the new vocabulary to look around at your project, notice things you didn't notice before, and express those observations. As you gain facility, you should be able to
- Discuss Extreme Programming, the Capability Maturity Model, the Personal Software Process, or your favorite process
- Determine when each process is more or less applicable
- Understand people who have differing opinions, abilities, and experience
Each person coming to this book does so with a different experience level, reading style, and role. Here's how you might read the book to use it to your greatest advantage: by experience, by reading style, or by role.
This book is written for the more experienced audience. The book does not contain procedures to follow to develop software; in fact, core to the book is the concept that every technique has limitations. Therefore, it is impossible to name one best and correct way to develop software. Ideally, the book helps you reach that understanding and then leads you to constructive ideas about how to deal with this real-world situation.
If you are an intermediate practitioner who has experience with software-development projects, and if you are now looking for the boundaries for the rules you have learned, you will find the following topics most helpful:
- What sorts of methodologies fit what sorts of projects
- Indices for selecting the appropriate methodology category for a project
- The principles behind agile methodologies
Being an intermediate practitioner, you will recognize that you must add your own judgement when applying these ideas.
If you are an advanced practitioner, you already know that all recommendations vary in applicability. You may be looking for words to help you express that. You will find those words where the following topics are presented:
- Managing the incompleteness of communication
- Continuous methodology reinvention
- The manifesto for agile software development
A few topics should be new even to advanced software developers: the vocabulary for describing methodologies and the technique for just-in-time methodology tuning.
By Reading Style
The earlier chapters are more abstract than the later chapters.
If you enjoy abstract material, read the book from beginning to end, watching the play of abstract topics to see the resolution of the impossible questions through the course of the book.
If you want concrete materials in your hands as quickly as possible, you may want to skip over the early chapters on the first read and start with Chapter 4, "Methodologies." Return to the sections about "Cooperative Games" and "Convection Currents of Information" to get the key parts of the new vocabulary. Dip into the introduction and the chapters about individuals and teams to fill in the gaps.
People who sponsor software development can get from this book an understanding of how various organizational, behavioral, and funding structures affect the rate at which they receive value from their development teams. Project sponsors may pay less attention to the details of methodology construction than people who are directly involved in the projects. They should still understand the consequences of certain sorts of methodology decisions.
Team leads and project managers can see how seating, teaming, and individuality affect their project's outcome. They can also learn what sorts of interventions are more likely to have better or worse consequences. They will need to understand the construction and consequences of their methodology and how to evolve their methodologymaking it as light as possible, but still sufficient.
Process and methodology designers can examine and argue with my choice of terms and principles for methodology design. The ensuing discussions should prove useful for the field.
Software developers should come to know this material simply as part of being in the profession. In the normal progression from newcomers to leaders, they will have to notice what works and doesn't work on their projects. They will also have to learn how to adjust their environment to become more effective. "Our methodology" really means "the conventions we follow around here," and so it becomes every professional's responsibility to understand the basics of methodology construction.
Organization of the Book
The book is designed to set up two nearly impossible questions at the beginning and derive answers for those questions by the end of the book:
- If communication is fundamentally impossible, how can people on a project manage to do it?
- If all people and all projects are different, how can we create any rules for productive projects?
To achieve that design, I wrote the book a bit in the "whodunit" style of a mystery. I start with the broadest and most philosophical discussions: "What is communication?" and "What is software development?"
The discussion moves through still fairly abstract topics such as "What are the characteristics of a human?" and "What affects the movement of ideas within a team?"
Eventually, it gets into more concrete territory with "What are the elements and principles of methodologies?" This is a good place for you to start if you are after concrete material early on.
Finally, the discussion gets to the most concrete matter: "What does a light, sufficient, self-evolving methodology look like?" and "How does a group create a custom and agile methodology in time to do the project any good?"
The two appendixes contain supporting material. The first contains the "Agile Software Development Manifesto," signed by 17 very experienced software developers and methodologists.
The second appendix contains extracts from three pieces of writing that are not as widely read as they should be. I include them because they are core to the topics described in the book.
Heritage of the Ideas in This Book
The ideas in this book are based on 25 years of development experience and 10 years of investigating projects directly.
The IBM Consulting Group asked me to design its first object-oriented methodology in 1991. I looked rather helplessly at the conflicting "methodology" books at the time. My boss, Kathy Ulisse, and I decided that I should debrief project teams to better understand how they really worked. What an eye-opener! The words they used had almost no overlap with the words in the books.
The interviews keep being so valuable that I still visit projects with sufficiently interesting success stories to find out what they encountered, learned, and recommend. The crucial question I ask before the interview is, "And would you like to work the same way again?" When people describe their experiences in words that don't fit my vocabulary, it indicates new areas in which I lack distinctions and words.
The reason for writing this book now is that the words and distinctions finally are correlating with descriptions of project life and project results. They are proving more valuable for diagnosis and intervention than any of the tools that I used previously.
The ideas in this book have been through dozens of development teams, eight methodology designs, and a number of successful projects on which I participated.
I am not the only person who is using these ideas:
- Kent Beck and Ward Cunningham worked through the late 1980s on what became called Extreme Programming (XP) in the late 1990s.
- Jim Highsmith studied the language and business use of complex adaptive systems in the mid-1990s and wrote about the application of that language to software development in his Adaptive Software Development.
- Ken Schwaber and Jeff Sutherland were constructing the Scrum method of development at about the same time, and many project leaders made similar attempts to describe similar ideas through the same years.
When a group of us met in February 2001 to discuss our differences and similarities, we found we had a surprising number of things in common. We selected the word agile to describe our intent and wrote the Agile Software Development Manifesto (Appendix A).
We are still formulating the principles that we share and are finding many other people who could have been at that meeting if they had known about it or if their schedules had permitted their presence.
Core to agile software development is the use of light-but-sufficient rules of project behavior and the use of human- and communication-oriented rules.
Agility implies maneuverability, a characteristic that is more important now than ever. Deploying software to the Web has intensified software competition further than before. Staying in business involves not only getting software out and reducing defects but tracking continually moving user and marketplace demands. Winning in business increasingly involves winning at the software-development game. Winning at the game depends on understanding the game being played.
The best description I have found for agility in business comes from Goldman (1997):
"Agility is dynamic, context-specific, aggressively change-embracing, and growth-oriented. It is not about improving efficiency, cutting costs, or battening down the business hatches to ride out fearsome competitive 'storms.' It is about succeeding and about winning: about succeeding in emerging competitive arenas, and about winning profits, market share, and customers in the very center of the competitive storms many companies now fear."
The Agile Software Development Series
Among the people concerned with agility in software development over the last decade, Jim Highsmith and I found so much in common that we joined efforts to bring to press an Agile Software Development Series based around relatively light, effective, human-powered software-development techniques.
We base the series on these two core ideas:
- Different projects need different processes or methodologies.
- Focusing on skills, communication, and community allows the project to be more effective and more agile than focusing on processes.
The series has these three main tracks:
- Techniques to improve the effectiveness of a person who is doing a particular sort of job. This might be a person who is designing a user interface, gathering requirements, planning a project, designing, or testing. Whoever is performing such a job will want to know how the best people in the world do their jobs. Writing Effective Use Cases (Cockburn 2001c) and GUIs with Glue (Hohmann, forthcoming) are two individual technique books.
- Techniques to improve the effectiveness of a group of people. These might include techniques for team building, project retrospectives, decision making, and the like. Improving Software Organizations (Mathiassen 2002) and Surviving Object-Oriented Projects (Cockburn 1998) are two group technique books.
- Examples of particular, successful agile methodologies. Whoever is selecting a base methodology to tailor will want to find one that has already been used successfully in a similar situation. Modifying an existing methodology is easier than creating a new one and is more effective than using one that was designed for a different situation. Crystal Clear (Cockburn, forthcoming) is a sample methodology book. We look forward to identifying other examples to publish.
Two books anchor the Agile Software Development Series:
- This one expresses the thoughts about agile software development using my favorite vocabulary: that of software development as a cooperative game, methodology as conventions about coordination, and families of methodologies.
- The second book is Highsmith's forthcoming one, Agile Software Development Ecosystems. It extends the discussion about problems in software development, common principles in the diverse recommendations of the people who signed the Agile Software Development Manifesto, and common agile practices. Highsmith's previous book, Adaptive Software Development, expresses his thoughts about software development using his favorite vocabulary, that of complex adaptive systems.
You can find more about Crystal, Adaptive, and other agile methodologies on the Web. Specific sites and topics are included in the References at the back. A starter set includes these sites:
- My home site, members.aol.com/acockburn
Thanks to Specific People
Ralph Hodgson has this amazing library of obscure and interesting books. More astounding, though, is how he manages to have in his briefcase just that obscure book I happen to need to read next: Vinoed's Sketches of Thought and Wenger and Lave's Situated Learning, among others. The interesting and obscure books you find in the References chapter probably came from Ralph's library.
Luke Hohmann tutored me about Karl Weick and Elliot Soloway. Jim Highsmith taught me that "emergent behavior" is a characteristic of the rules and not just "lucky." Each spent a disproportionate amount of time influencing the sequencing of topics and accuracy of references, commenting on nearly every page.
Jason Yip beautifully skewered my first attempt to describe information dissemination as gas dispersion. He wrote, "Kim is passing information. Information is green gas. Kim is passing green gas . . ." Yikes! You can guess that those sentences changed!
Bo Leuf came up with the wonderful wordplay of argh-minutes (in lieu of erg-seconds) as the unit of measure for frustrating communications sessions. He also was kind enough to double-check some of my assertions. For example, he wrote to some Israelis to check my contention that in Israel, "politeness in conversation is considered more of an insult than a compliment." That produced an exciting e-mail exchange, which included (from Israelis): "Definitely wrong on this one, your author. . . . We always say hello and shake hands after not seeing for a few days. . . . I think your author is mistaking a very little tolerance for mistakes at work for a lack of politeness." Another wrote, "Regarding your being flamed. There is no way out of it, no matter what you say. According to me, Israelis would demand of you to have your own opinion and to stand behind it. And of course they have their own (at least one :-)." Benny Sadeh offered the word I finally used, "frankness."
Martin Fowler contributed the handy concept of "visibility" to the methodology discussion, in addition to helping with constructive comments and being very gentle where he thought something was terrible.
Other energetic reviewers I would like to recognize and thank (in first-name alphabetical order) are Alan Harriman, Allen Galleman, Andrea Branca, Andy Sen, Bill Caputo, Charles Herbaut, Charlie Toland, Chris Lopez, Debbie Utley, Glenn Vanderburg, James Hanrahan, Jeff Miller, Jeff Patton, Jesper Kornerup, Jim Sawyer, John Brewer, John Cook, Keith Damon, Laurence Archer, Michael Van Hilst, Nick Fortescue, Patrick Manion, Phil Goodwin, Richard Pfeiffer, Ron Holiday, Scott Jackson, Ted Young, Tom DeMarco, and Tracy Bialik.
The Silicon Valley Patterns Group took the trouble to dissect the draft as a group, for which I doubly thank them.
The Salt Lake production team of Elizabeth Wilcox, Cathy Gilmore, John Roberts, and Malia Howland did a fantastic job of turning the manuscript into a final book in an unreasonably short period of time.
All these people did their best to see that I fixed the weak parts and kept the good parts. If I had another few years to keep reworking the book, I might even have been able to get it to the point that they would have accepted it.
In the absence of those extra years, I thank them for their efforts and apologize for not being able to fix all the awkward spots.
Thank goodness the Beans & Brews coffee shop finally started playing jazz and rock again. I lost several months of writing to heavy metal and country music. Thanks to the Salt Lake Roasting Company for staying open until midnight.
To save us some future embarrassment, my name is pronounced "Co-burn," with a long o.
Additional Copyright Information
"Scandinavian Design" by Pelle Ehn, from Usability: Turning Technology into Tools, Paul S. Adler and Terry A. Winograd (Eds.), Copyright © 1992 by Oxford University Press, Inc. Used with permission, Oxford University Press, Inc.
"Programming as Theory Building" by Peter Naur, from Computing: A Human Activity, ACM Press, 1992. Used with permission, Peter Naur.
From The Book of Five Rings by Miyamoto Musashi, translated by Thomas Cleary, © 1993, 1994. Reprinted by arrangement with Shambhala Publications, Inc., 300 Massachusetts Avenue, Boston, www.shambhala.com.
"Shu Ha Ri" by Ron Fox, from The Iaido Newsletter, Volume 7 number 2 #54, February, 1995. Used with permission, Ron Fox, MSU Kendo Club for Shu Ha Ri.
References to eBucks.com and the description of Crystal Orange Web used with permission, Michael Jordaan, eBucks.com.
References to a discussion of communication patterns in open source projects used with permission, Mike Nygard.
Figure 3-10 used with permission, Joshua Kerievsky, Industrial Logic, Inc., www.industriallogic.com, Copyright © 2001.
Figure 3-11 used with permission, Ron Jeffries, Ann Anderson, & Chet Hendrickson, Extreme Programming Installed, Addison-Wesley, 2001.
Figures 3-1, 3-9, 3-15, and 3-16 used with permission, Evant Solutions Corporation, www.evantsolutions.com, Copyright © 2001.
Figures 3-2, 3-3, 3-6, 3-7, and 3-8 used with permission, ThoughtWorks, Inc., www.thoughtworks.com, Copyright © 2001.
Figures 3-12 and 3-13 used with permission Ken Auer, RoleModel Software, Inc., www.rolemodelsoft.com, Copyright © 2001.
Preface to the Second Edition
The agile model of software development took the world by storm in 2001. Within a year there were books and conferences on it around the world. Within five years, it had influenced everything from project management and how corporate executives write contracts with their clients, to military procurement procedures, and even to college curricula.
It is time to look at the changes and see what we can learn about the agile model, and more generally, the cooperative game.
At the time that the Manifesto for Agile Software Development was writtenFebruary 2001I was already deep into writing about software development as a cooperative game and the tailoring of methodologies to individual projects. The manifesto merely echoed what I and others were already doing.
The agile model took the world by storm. Hundreds of developers signed the online signing board (the original, informal Agile Alliance, not to be confused with the AgileAlliance corporation that was formed later and runs the Agile conferences in the U.S.) to show their alignment with its values and principles.
After the initial questions around "What does this mean?," people asked:
- "Where does it fit in the total set of development situations?"
- "How do we blend these ideas with others?"
- "How do we extend these ideas to other fields?"
These are the questions picked up by the additional text in this second edition.
The cooperative game model grew alongside the agile model. Originally constructed to explain software development, it struck a chord with business people, who rightly saw that business is also predominantly a cooperative (and competitive!) game of invention and communication.
You would imagine that during the past five years, something should have come as a surprise to me. I highlight four:
- Engineering is also a cooperative game of invention and communication. With a bit of reframing, we can now place software engineering as a proper member of the engineering fields.
I leave intact in the second edition the fairly strong words I wrote originally against the idea that software development should be treated as engineering. I do so because those words still apply to the way in which most people still incorrectly view engineering and draw from it correspondingly incorrect ways of how to manage software development. In this second edition text, I turn the investigation back to the question, "What is engineering?"
After the second world war, discipline envy of applied physics caused wishful thinking in engineering academia, and popular understanding of engineering got off track. Looking afresh at engineering practices, we notice that it is itself a cooperative game of invention and communication. From this, we can create an updated notion of software engineering that makes sense both practically and pedagogically.
- The specialty area of user experience design has made strong inroads in the last five years. It was ignored by the authors of the manifesto, and deserves serious attention.
All of the early agile methodologies skipped completely over this issue, probably because none of the manifesto authors had strong expertise in that area. That area has been and still is an area of hot debate, with few solid conclusions. I discuss the conclusions and points of contention in that section.
- I have learned how to separate project management strategies from methodologies.
Project management strategies too often get placedincorrectlyas part of a corporate process or methodology. Encoding what should be on-the-scene strategies as fixed policies often forces managers to make ineffective strategy decisions and costs the organization greatly. It is therefore important to learn to separate the two.
- In addition to the surprises, all of the development around "agile software development" has occurred since the writing of the first edition.
This means that there are a lot of topics to cover, with a lot of misconception to clear up around the meaning and practice of agile software development.
Thus, the chapter on the evolution of agile software development is considerably longer than the other evolution chapters. This is natural and appropriate. I hope that readers already knowledgeable about agile development will find new nuggets of information in that chapter, and that newcomers to agile development, confused by some of these issues, will gain some clarity and insight.
Reading This Book
The second edition is to be read in much the same way as the first edition.
Those interested in "the rules of the game," including but not restricted to software development, will find the first four chapters (including the Introduction) their main meal.
Those who want to get quickly to the agile model of software development and the construction of methodologies for their teams may want to leap to the second half of the book, or Appendix A on the Agile Manifesto, and from there, go to the methodology sections.
Some will come to this book to read about the foundations of the Crystal family of methodologies. They should start with Chapter 6, "The Crystal Methodologies," which I have updated with an improved description of the family. They can then go back to the beginning and read about the communication and cooperative game, as that is the anchor for the Crystal family.
Those who have already read the first edition may want to read each "Evolution" chapter. These new chapters are easily identified by the gray thumbtab along the edge of the page. I have limited the changes in the rest of the book to corrections of mistakes, so you would not have to reread every page to search for the newer ideas.
However you read this book, I urge you to dip into Appendix B, with reprints of key writings by Musashi (the 17th Century samurai), Peter Naur, and Pelle Ehn.
Over the past five years, I have come to appreciate their writings steadily more. I now teach from Musashi in the first hour of agile training, use Naur's theory as a core element in describing software development, and revisit Ehn's paper periodically to help mature my own views. I suspect you will find them similarly valuable.
Thanks to Specific People
The reviewers at the Silicon Valley Patterns Group were once again very considerate and helpful in helping me tune the text. They found errors and quarreled, and in the rare case I decided against their wishes, I take full blame on myself.
Russ Rufer and Tracy Bialik in particular held my feet to the fire on several occasions to make sure I meant what I wrote. Thanks to them, I removed a couple of assertions I couldn't back up. They also helped improve the explanations of several diagrams.
Other reviewers who kindly read the draft and sent in wishes, comments, and corrections include John Rusk, Paul Oldfield, Brandon Campbell, Romilly, Dave Churchville, Ilja Preuß, Géry Derbier, Mark Levinson, and Paul McMahon. John's agileKiwi.com site is an excellent resource for more information on burn-up and earned-value charts (and other topics agile); Dave's ExtremePlanner.com site focuses on agile planning and tracking software; Paul's comments on agile and CMMi were major enough that I asked him to contribute a sidebar around them.
Celeste Rosell bent her schedule out of shape on many occasions to help get article and page cross-references worked out in time. I'm not sure how she worked out correct answers, given (on occasion) nothing more than "???" in the middle of a paragraph.
The production staff at Addison Wesley / Pearson Education and their contractors were extremely helpful. Leslie Sweeney of Carlisle Publishing did an outstanding job of creating a clear and memorable diagram of decision flows in software development (Figure 1.1-2). Alan Clements of Pearson came up with the fabulous cover designs for both my Crystal Clear book and this second edition.
On a happy note, the Salt Lake Coffee Break still serves Turkish coffee and is open until 2 a.m. every night. I've relied on both in writing this second edition.
On an unhappy note, marketing squatters took the crystalMethodologies.org site when I unwittingly let it lapse, so that site no longer provides useful material and certainly is not connected with my work. I have created Alistair.Cockburn.us, which currently holds all my papers, talks, and other material. With luck, I'll be able to keep that site intact and useful.