Programmers practice a most abstract and incomprehensible art; yet, as end users, we take them for granted and demand perfection from the software they create. Rosenberg, a theater and movie critic turned technology columnist and founder of the Web magazine Salon.com,^B attempts to shed light on the day-to-day realities of what turns out to be a Herculean task: getting computers to do what we want them to. He spent three years following the course of development of a software program code named Chandler, a combination calendar, to-do list, e-mail manager, and personal database. This open-source project was the dream child of Mitch Kapor, the creator of Lotus 1-2-3, who envisioned a simple, elegant interface capable of easy storage and retrieval of any number of personal data. The practical matter of creation was another story, however, and we learn how the fits and starts of software engineering make even creating a "simple" program an arduous task. Although this is not edge-of-your-seat stuff, it is highly instructive for anyone planning on "managing" a software project. David Siegfried
Copyright © American Library Association. All rights reserved

"Technology people like to call complicated problems 'nontrivial.' Scott Rosenberg has taken an extremely nontrivial topic and made it accessible. He plainly admires the people who create code, but shows them as the complex, flawed beings we all are- and how human talents, quirks and passions become part of what people create. Dreaming in Code is stellar reporting and writing."
–Dan Gillmor, Director, Center for Citizen Media and author of We the Media

"We live in a world increasingly governed by the near-invisible logic of software, and yet most of us know almost nothing about that hidden world inside our computers. Dreaming in Code is a fascinating and sobering exploration of how the challenges of programming both inspire and undermine our human drive to create new tools. Beautifully written, it's a book for anyone interested in the roots of creativity and innovation, for coders and non-coders alike."
–Steven Johnson, author of Everything Bad Is Good for You and Emergence

"The great software genius Bill Joy likes to say that writing software is like building a cathedral: It's art, science, architecture, and manual labor all rolled into one. Dreaming in Code illuminates the truth of that metaphor in all its subtlety and fullness. It has drama, comedy, pathos, and poignancy- and its center, in Mitch Kapor, is one of the most fascinating and yet least understood figures of the digital revolution. It's also so smart and insightful on its subject as any book I know."
–John Heilemann, New York Magazine columnist and author of Pride Before The Fall: Trials of Bill Gates and the End of the Microsoft Era

"Dreaming in Code is the first true successor to Tracy Kidder's Soul of a New Machine, and is written with a combination of technical sophistication and narrative skill not seen in many years. Read it to understand what all these software wizards actually do."
–James Fallos, The Atlantic

"Dreaming in Code bravely goes where no nonprogrammer has dared to go before: into the whirlwind where human imagination struggles to become code. Scott Rosenberg brilliantly interweaves the story of a start-up software company with the history of our (endless) attempts to rationalize the process of programming. He brings the reader close to the programmers: to root for them, then worry for them, as their project begins falling into all the old traps. Yet Rosenberg's admiration for the visionaries and coders shines through, until one almost believes they will one day succeed at creating their own 'Key To All Mythologies,' their repository of all things digital. Most people do not understand what goes into the creation of computer software, but now they will."
–Ellen Ullman, author of Close to the Machine: Technophilia and Its Discontents

Our civilization runs on software. Yet the art of creating it continues to be a dark mystery, even to the experts, and the greater our ambitions, the more spectacularly we seem to fail.

Big software projects regularly crash and burn--just ask the FBI and the IRS, the Pentagon and the FAA, or any decent-size corporation. The software that runs our personal computers is just as trouble prone: The latest version of Microsoft Windows took years longer than planned, and it will still have mountains of bugs. Never in history have we depended so completely on a product that so few know how to make well.

Why is it so hard to bend computers to our will? Is creating a great program more like building a bridge or making a movie? Why do software projects display an almost metaphysical capacity for making time come to a stop? And will there ever be a bug-free program?

To answer such questions, Scott Rosenberg spent three years following a group of men and women--led by Lotus 1-2-3 creator Mitch Kapor--who are developing a novel personal information manager named Chandler (as in Raymond) meant to challenge market-leader Microsoft Outlook with elegant innovations. Their goal: to build something truly different--an application versatile enough to allow you to take emails, appointments, and notes and effortlessly transform one into another, organizing and displaying them as you please.

The team included legendary programmer Andy Hertzfeld, author of much of the original Macintosh operating system, and Lou Montulli, the Netscape cofounder who invented the Web browser "cookie." Chandler's first manager, Michael Toy, dreamed of speedy releases but found himself stuck in quicksand; its second, Katie Parlante, resolutely held together a crew of gifted but stubborn programmers--including John Anderson, a philosophical coder who frequently found himself chasing elusive bugs down "ratholes," and Andi Vajda, a database expert who once hacked open his high school's minicomputer and found his future inside.

Their story takes us through a maze of dead ends and exhilarating breakthroughs as they and their colleagues wrestle not only with the abstraction of code but with the unpredictability of human behavior, especially their own. Along the way, we encounter black holes, turtles, snakes, dragons, axe-sharpening, and yak-shaving--and take a guided tour through the theories and methods, both brilliant and misguided, that litter the history of software development, from the famous "mythical man-month" to Extreme Programming.

Not just for technophiles but for anyone captivated by the drama of invention, Dreaming in Code offers a window into both the information age and the workings of the human mind.

"Dreaming in Code is the first true successor to Tracy Kidder's Soul of a New Machine, and is written with a combination of technical sophistication and narrative skill not seen in many years. Read it to understand what . . . software wizards actually do."
--James Fallows, The Atlantic

"Technology people like to call complicated problems 'nontrivial.' Scott Rosenberg has taken an extremely nontrivial topic and made it accessible. He plainly admires the people who create code, but shows them as the complex, flawed beings we all are. Dreaming in Code is stellar reporting and writing."
--Dan Gillmor, director of The Center for Citizen Media and author of We the Media

"Dreaming in Code is a fascinating and sobering exploration of how the challenges of programming both inspire and undermine our human drive to create new tools. Beautifully written, it's a book for anyone interested in the roots of creativity and innovation, for coders and non-coders alike."
--Steven Johnson, author of Everything Bad Is Good for You and Ghost Wars

"Scott Rosenberg bravely goes where no nonprogrammer has dared to go before: into the whirlwind where human imagination struggles to become code. In Dreaming in Code he brilliantly interweaves the story of a start-up software company with the history of our (endless) attempts to rationalize the process of programming."
--Ellen Ullman, author of Close to the Machine

"Dreaming in Code has drama, comedy, pathos, and poignancy--and at its center, in Mitch Kapor, one of the most fascinating and yet least understood figures of the digital revolution. It's also as smart and insightful on its subject as any book I know."
--John Heilemann, author of Pride Before The Fall: The Trials of Bill Gates and the End of the Microsoft Era

SCOTT ROSENBERG is an award-winning journalist and the cofounder of Salon.com, where he served as technology editor, then managing editor, and is now vice president for new projects. His writing has appeared in the New York Times, Wired, The San Francisco Examiner, and other publications.

CHAPTER 1

DOOMED

[JULY 2003]

Michael Toy places his palms on his cheeks, digs his chin into his wrists, squints into his PowerBook, and begins the litany.

"John is doomed. He has five hundred hours of work scheduled between now and the next release. . . . Katie's doomed. She has way more hours than there are in the universe. Brian is majorly doomed. Plus he's only half time. Andy--Andy is the only one who doesn't look doomed. There are no hundreds on his list."

They don't look doomed, these programmers sitting around a nondescript conference room table in Belmont, California, on a summer day. They listen quietly to their manager. Toy is a tall man with an impressive gut and a ponytail, but he seems to shrink into a space of dejection as he details how far behind schedule the programmers have fallen. It's July 17, 2003, and he's beginning to feel doomed himself about getting everything done in the less than two months before they are supposed to finish another working version of their project.

"Everybody who has a list with more time than there is in the universe needs to sit down with me and go over it."

These lists are the bug lists--rosters of unsolved or "open" problems or flaws. Together they provide a full accounting of everything these software developers know must be fixed in their product. The bug lists live inside a program called Bugzilla. Toy's programmers are also using Bugzilla to track all the programming tasks that must be finished in order to complete a release of the project; each one is responsible for entering his or her list into Bugzilla along with an estimate of how long each task will take to complete.

"Now let's talk about why we're behind. Does anyone have a story to tell?"

There's silence for a minute. John Anderson, a lanky programming veteran whose title is systems architect and who is, in a de facto sort of way, the project's lead coder, finally speaks up, in a soft voice. "There's a bunch of reasons. In order to build something, you have to have a blueprint. And we don't always have one. Then you hit unexpected problems. It's hard to know how long something's going to take until you know for sure you can build it."

"But you can't just throw up your hands and say, I quit." Toy usually prefers to check things off his agenda fast, running his developers' meetings with a brisk attitude of "let's get out of here as fast as we can" that's popular among programmers. But today he's persistent. He won't let the scheduling problems drop. "We need to make guesses and then figure out what went wrong with our guesses."

Jed Burgess, one of the project's younger programmers, speaks up. "There's a compounding of uncertainty: Your estimates are based on someone else's estimates."

Toy begins reviewing Anderson's bugs. "The famous flicker-free window resizing problem. What's up with that?"

Officially, this was bug number 44 in Bugzilla, originally entered on January 19, 2003, and labeled "Flicker Free window display when resizing windows." I had first heard of the flicker-free window resizing problem at a meeting in February 2003 when the Open Source Applications Foundation (OSAF), whose programmers Toy was managing, had completed the very earliest version of its project, Chandler--an internal release not for public unveiling that came even before the 0.1 edition. Ultimately, Chandler was supposed to grow up into a powerful "personal information manager" (PIM) for organizing and sharing calendars, email, to-do lists, and all the other stray information in our lives. Right now, the program remained barely embryonic.

At that February meeting, Anderson had briefly mentioned the flicker bug--when you changed the size of a window on the Chandler screen, everything flashed for a second--as a minor matter, something he wanted to investigate and resolve because, though it did not stop the program from working, it offended him aesthetically. Now, nearly six months later, he still hasn't fixed it.

Today Anderson explains that the problem is thornier than he had realized. It isn't simply a matter of fixing code that he or his colleagues have written; its roots lie in a body of software called wxWidgets that the Chandler team has adopted as one of the building blocks of their project. Anderson must either wait for the programmers who run wxWidgets to fix their own code or find a way to work around their flaw.

"So you originally estimated that this would take four hours of work," Toy says. "That seems to have been off by an order of magnitude."

"It's like a treasure hunt," Anderson, unflappable, responds. "You have to find the first thing. You have to get the first clue before you're on your way, and you don't know how long it will take."

"So you originally estimated four hours on this bug. You now have eight hours."

"Sometimes," Anderson offers philosophically, "you just wake up in the morning, an idea pops into your head, and it's done--like that."

Mitchell Kapor has been sitting quietly during the exchange. Kapor is the founder and funder of the Open Source Applications Foundation, and Chandler is his baby. Now he looks up from his black Thinkpad. "Would it be useful to identify issues that have this treasure-hunt aspect? Is there a certain class of task that has this uncertainty?"

"Within the first hour of working on the bug," Burgess volunteers, "you know which it's going to be."

So it is agreed: Bugs that have a black hole-like quality--bugs that you couldn't even begin to say for sure how long they would take to fix--would be tagged in Bugzilla with a special warning label.

Shortly after the meeting, Toy sits down at his desk, calls up the Bugzilla screen, and enters a new keyword for bug number 44, "Flicker Free window display when resizing windows": scary.



Toy's fatalistic language wasn't just a quirk of personality: Gallows humor is a part of programming culture, and he picked up his particular vocabulary during his time at Netscape. Though today Netscape is remembered as the Web browser company whose software and stock touched off the Internet boom, its developers had always viewed themselves as a legion of the doomed, cursed with impossible deadlines and destined to fail.

There was, in truth, nothing especially doomed about OSAF's programmers: Several of them had just returned from a conference where they presented their work to an enthusiastic crowd of their peers--who told them that their vision could be "crisper" but who mostly looked at the blueprint for Chandler and said, "I want it now!" Though the software industry had been slumping for three straight years, they were working for a nonprofit organization funded by $5 million from Kapor. Their project was ambitious, but their ranks included veteran programmers with estimable achievements under their belts. Andy Hertzfeld had written central chunks of the original Macintosh operating system. John Anderson had written one of the first word processors for the Macintosh and later managed the software team at Steve Jobs's Next. Lou Montulli, another Chandler programmer who was not at the meeting, had written key parts of the Netscape browser. They'd all looked doom in the eye before.

Similarly, there was nothing especially scary about bug number 44. It was a routine sort of problem that programmers had accepted responsibility for ever since computer software had migrated from a text-only, one-line-at-a-time universe to today's graphic windows-and-mouse landscape. What scared Toy was not so much the nature of Bug 44 but the impossibility of knowing how long it would take to fix. Take one such unknown, place it next to all the other similar unknowns in Chandler, multiply them by one another, and you have the development manager's nightmare: a "black hole" in the schedule, a time chasm of indeterminate and perhaps unknowable dimensions.

Two months before, the entire Chandler team of a dozen programmers had met for a week of back-to-back meetings to try to solve a set of problems that they had dubbed "snakes"--another word Toy had salvaged from Netscape's ruins. A snake wasn't simply a difficult problem; it was an "important problem that we don't have consensus on how to attack." Snake superseded a looser usage at OSAF of the word dragon to describe the same phenomenon.

Black holes, snakes, dragons--the metaphors all daubed a layer of mythopoetic heroism over the most mundane of issues: how to schedule multiple programmers so that work actually got done. You could plug numbers into Bugzilla all day long; you could hold one meeting after another about improving the process. Software time remained a snake, and it seemed invincible.

This was hardly news to anyone in the room at OSAF. The peculiar resistance of software projects to routine scheduling is both notorious and widely accepted. In the software development world, lateness was so common that a new euphemism had to be invented for it: slippage.

Certainly, every field has its sagas of delay; the snail's pace of lawsuits is legendary, and any building contractor who actually finishes a job on time is met with stares of disbelief. But there's something stranger and more baffling about the way software time bends and twists back on itself like a Mobius strip. Progress seems to move in great spasms and then halt for no reason. You think you're almost done, and then you turn around and six months have passed with no measurable progress.

This is what it feels like: A wire is loose somewhere deep inside the workings. When it's connected, work moves quickly. When it's not, work halts. Everyone on the inside tries painstakingly to figure out which wire is loose, where the outage is, while observers on the outside try to offer helpful suggestions an...