on February 21, 2007
If you're in software development in any capacity, you'll find almost everything in this book is depressingly familiar: schedules dragging on endlessly, vague ideas standing for clear feature descriptions, and a remarkable lack of interest in learning from the past. You meet the indecisive manager, the frustrated designer, and the cowboy coder (the one who loses interest in things as soon as they finally work, determined to gut and rewrite thousands of lines of code for the sake of incremental "elegance").
"Dreaming in Code" really isn't a description of the act of programming, it's more about the difficulty of designing software with other people. It's not at all like designing, say, a new car, where real-world constraints (like the laws of physics and the behavior of materials) come into play. In software, nearly anything is possible. Unfortunately, or a "what comes out of that isn't usually the thrill of possibility, but the paralysis of choice. And that's what Rosenberg captures so well here: the endless, endless meetings discussing what *could* be done, how features *might* work, or what sorts of things "users" *might* want. And it is always that nebulous term "the user" here, the Chandler project never seems to bother thinking much about exactly who will use their product, under what circumstances.
It's not giving anything away to mention that the Chandler project doesn't end at the end of the book. Amazingly, the project is *still* grinding away after four years, releasing incremental versions of their calendar application (each loudly proclaiming that it's an "experimentally usable version"), even as they get passed by on a regular basis by new, fully-usable, web-based calendars like the one from Google.
It's not exactly a fun read--it is essentially a case study of a very long series of meetings--but it should be required reading for anyone involved in software development.
on January 23, 2007
Ansel Adams wrote, "There is nothing worse than a sharp image of a fuzzy concept." And such is the case with the Chandler project. After four years of development, they have delivered only a 0.6 release with no general availability in sight.
In Dreaming in Code, author Scott Rosenberg follows a group of programmers tasked with creating a new product over a three-year stint. Along the way the book explores disciplines in development (and the lack of), the history of computing (particularly its truths and folklore), and explains why software engineering isn't a science but an art. A common misconception even among developers is that software is similar to construction when, as becomes clear in the book, developing software is more like cooking. Programming methodologies are as plentiful as cookbooks but both are limited by the realities of artistry. A chef can make miracles from a pantry full of ingredients; a cook cannot.
If you're involved with a development team as a marketer, there is much here that will illuminate your team's dysfunction. Rosenberg reintroduces us to concepts that have been known since The Mythical Man Month and The Soul of a New Machine but apparently not understood, remembered, or believed. Strongly recommended.
on January 18, 2007
Santa came through this year with a slightly advance copy of "Dreaming in Code", which tries to do for software engineering what "The Soul of a New Machine" did for computer engineering, following a single project through to its attempted conclusion. Software development is a story that's very rarely told, considering how dramatically software has changed all of our lives in the last 30 years. Author Scott Rosenberg does a good job of conveying the difficulties in software engineering, and the inevitable headaches and drama that come with incomplete plans and shifting specs (and they're always incomplete and shifting).
Where Rosenberg went wrong, unfortunately, is his choice of project to follow. Mitch Kapor's Chandler is quite atypical of software projects: it's driven entirely by one man's quixotic vision, and never has to encounter the usual give-and-take with VC's or upper management that help to clarify a plan. Kapor comes off as an untethered idealist (Al Gore makes the obligatory cameo at the office), and his project is afflicted by the same we-are-the-world unseriousness as his politics. Most notably, Kapor decides there should be no central repository for data (because, hey, down with authority and all that): instead, every item will just be represented, Napster-style, across users' personal computers. It's a costly decision that I don't think would have been made if it were more than just Kapor running the show.
Actually, I think the strongest part of the book is when Rosenberg abandons the project entirely in the middle section to delve into the history of the programming discipline, noting everyone from Donald Knuth to 37signals' Jason Fried. It's a useful, lucid introduction to the field that contains stories I hadn't seen before.
To pick some nits, there are errors that betray Rosenberg as an outsider. "Foo" and "bar", for instance, usually aren't stand-ins for variable names, they're stand-ins for *values*; variable names are decided on almost immediately once the need for one becomes known. [UPDATE: this caused some controversy in the comments. It's true that "foo" and "bar" can also represent variables, but I still contend that it's only in theoretical discussions, when nothing is known about those variables. The book (p. 196) calls them "placeholders" during real-life coding, which I don't think is often true. FURTHER UPDATE, AFTER MORE COMMENTS: Okay, okay, I guess I was wrong. You guys win!] Rosenberg also, I think, makes too big a deal of software's need for precise language: many other engineering fields, and the legal profession, require precise writing, with small errors potentially leading to catastrophe. Rosenberg also overreaches when trying his hand at software philosophy, declaring "The only software worth making is software that does something new" (tell that to the OpenOffice people).
All that aside, this is an entertaining book with some interesting insights, and it would be a great read for anyone who's thinking of going into programming - hopefully it won't scare them off.
on February 3, 2007
I have been following the Chandler project with interest from the beginning and have checked out most of the releases. Even to a distant observer,the glacial progress of the project, the lack of clarity about the objectives, and the mid-course changes that have reduced it to a pale shadow of the original vision, have been baffling.So, when I noticed Rosenberg's book, I immediately bought it and devoured it in a couple of hours. While the main focus of the book is on Chandler, the OSAF and the personalities there, it does talk quite a bit about the difficulties of software development, history of programming languages, methodologies etc quite a bit. I suspect the actual Chandler & OSAF content page wise might be only 50-60%. The book's general approach is like this - start out with something that the OSAF tried to do, then drift into a related topic from the history of software development; it alternates between discussions of the development of Chandler and various other topics. Very rarely does Scott Rosenberg actually examine why certain development efforts at OSAF failed in depth - he relies too much on failures of 'proprietary' software projects as a way of explaining the failures of the Chandler project. There is no open source context to the discussion most of the time - examining some aspect of software development of an open source project and drawing parallels from the failure of some proprietary software development effort is confusing. One of the most puzzling things about a lot of software doomsday scenario books is that, in spite of the fact that not all projects fail, they never try to figure out what made the successful ones work. Maybe nothing sells like a disaster. This book is no exception. Except for an occasional quote from Linus Torvalds about the need to start out small and fulfill an immediate need, the book never poses the question - `Chandler was by no means the first open source project, Why did it fail when many other user-facing open source projects have managed to get traction ?' There several frank quotes from some of the hackers like Andy Hertzfeld about the lack of momentum and the chase for the illusionary perfect plan. The author tries to suggest that open source projects are not too good at coding from scratch, which I think is wrong. If you have been closely following the project, then there is very little that's new in this book, very little that any experienced programmer or a clued-in tech-industry person might not have been able to guess. The main failing of the book is it's attempt to spread the blame to several supposedly difficult aspects of software development, and make failure seem in projects of this kind seem almost intrinsic. The other main shortcoming is the way he soft-pedals on the OSAF leadership. This is exactly the kind of book you can expect from a very well-informed and clued-in author, but one who is not a practitioner of the craft of software, especially open source software.
on April 29, 2007
The catchy title does not reflect the book's content. This book explores the question of why developing software is hard. Is software an art or a science? Why can't software be built like bridges - is there an underlying physics? No one ever redesigns a bridge when it is half built, but software projects suffer this all the time. Why does adding more programmers to a project slow progress? All these questions and more are considered in this informative, entertaining and well written book. I learned a heck of a lot.
The book revolves around a case study of an open source software project called Chandler which started in 2001 and continued as the book went to press. This case study serves two purposes: 1) it is interesting in its own right and 2) it provides many examples and a basis for a broader examination of software engineering and practice, past, present and future.
The aim of the Chandler Project is to create a revolutionary new way to handle personal information. Although we learn a bit about the history of such tools and the current state of the art, this book is mostly about how the project was conceived and carried out.
The project was painstakingly chronicled by the author over a period of three years. Readers get a rare inside view into and blow by blow account of a contemporary software project. We experience the personalities, the exuberance of a powerful vision, and the reality of how hard it is to get there.
We learn how after many many months into the project, after countless discussions about requirements and design, there was little evidence of tangible progress. Things went round and round and no coding had even begun. Several $million had been invested; How could that be?
This was a high profile open source project, headed up by Mitch Kapor, who was independently wealthy after creating Lotus 1-2-3 and another successful company called On Technology. He had a dream, and Chandler was the vehicle for making it happen.
There was a flurry of excitement about the first release of code, and there were 15,000 downloads in the first day - but there was almost nothing there. People were very disappointed, and many lost interest.
The story is fascinating and at the same time shocking and painful to `watch' from our ringside seats how badly things were going. Amazingly, nearly two years in the trenches passed by before things started coming together. We learn how deadline after deadline slipped away further and further on the horizon. We relive the many painful meetings and discussions where reality set in, great idea after great idea was chopped out of the scope just to get something out the door.
Many people became discouraged, some left, new people came on board. It took about three years before there was anything that developers themselves could use and test (known as eating their own dog food). In the mean time, Microsoft was releasing Vista which contained some of the still-to-be-done innovations. Other innovative tools were coming on the market - did Chandler risk becoming irrelevant?
Launching from the Chandler case study, the author does a good job of exploring the broader issues of software engineering, putting Chandler in context. Horror of horrors, we learn that as bad as things went on the this project, especially in its early stages - it is no different from countless other projects.
We get a short history of various methodologies for building software that have evolved in the past 50 years, from the spaghetti code of Fortran GOTOs to structured programming, the capability maturity model, literate programming and more recent agile software development techniques such as extreme programming. We learn about the elusive holy grail of reusable chunks of program code that fit together like Lego blocks - and get some insights into why this is hard.
After I finished the book the first thing I did was go on-line and check to see if the project was still going on, or whether it got canned. I could use a tool like Chandler.
on January 28, 2007
...this book hovers somewhere between three and four stars. It's rambling, prone to interminable asides (but some of these asides are very good stand-alone essays!), and mostly written in a colorful, verbose style which makes it pleasant to read but dilutes the amount of information packed in every page (I was surprised at seeing this style from a journalist, who, I'd have assumed, would have mastered terseness and high info content), so it takes longer to read than it should. However, the book is worth the investment of time (for me - an engineer who does lots of software development and also coordinates/manages other SW engineers on a variety of SW projects), because it does contain a lot of information, quotable excerpts, and links (it's a real pity that there's no formal "bibliography" section, since the book quotes so many others). So, given the usefulness (and nudged by the pleasantness of the reading experience), I was tipped towards 4 stars rather than 3, despite the book's defects in terms of organization and structure.
If I knew as much as I do about developing and managing development, but wasn't actually involved in these activities as my job, I'd probably tilt towards 3 stars, because the usefulness aspect would then be missing.
on February 3, 2007
Unlike other reviewers, I picked this book up mainly to understand just what happened at OSAF. I was one of those who was enthused by the initial announcement. PIMs to me, are horribly demanding. I have never had the patience to add everything into one. And Mitch Magic seemed to be just what the doctor had ordered.
But, like many others, the coming years just kept disappointing. It was amazing to watch the project disintegrate. Vague catch all phrases instead of specific feature descriptions. An overzealous determination to build for every conceivable user, every conceivable need. And then, the entire Cosmo/Scooby debacle - to rescue a sinking product, they decided to make another 2 products!
Scott of course had no clue of the future when he started researching the project in 2003. Happenstance that he stumbled on to a project which broke every single cardinal rule of project management. And managed to document the same.
Had he chosen a "normal" project, the book would have been totally unremarkable. The happy/unhappy coincidence of his case study becoming a train wreck, makes this a fascinating read. You get to see all the emotions, analysis and fantasies that went into Chandler.
Unfortunately, the book clocks in at 400 pages, which is some where around 150 pages too long. By the 10th page, you kinda know what is coming. Now, it comes in color, and there is a lot of action. My jaw dropped more than once. But, in the extra 150 pages, you get tons and tons of "other" stuff - long winded histories of other products. Diatribes on computing "philosophies". It goes on and on ...
Simple advice - read the OSAF parts, and simply skim the rest - the extra 150 pages add little to the central thesis and can easily be avoided.
on March 16, 2007
Based on my 25 years of experience as a software developer, I found Dreaming in Code to be an astonishingly correct depiction of the current state of software development. In documenting the development of Mitch Kapor's "Chandler", a personal information management (PIM) project, Rosenberg reminded me of prior projects which went on interminably, as they slipped into "software time", where long-term software projects can get trapped in endless cycles of change, development, testing and fixes.
Every developer should find this story line distressingly familiar, even comical, as the "Chandler" managers make the same kind of project management mistakes which have doomed countless other projects. Although the "Chandler" story is easily recognizable to software professionals, Rosenberg tells this tale from an outsider's perspective, allowing him to simplify this highly complex subject and appeal to anyone who cares about computer software and its creators, managers and consumers. In a first for a journalist, Rosenberg interlaces his "Chandler" story with a brilliant tutorial on the history of software development, explaining this highly technical subject in ways which the non-technical can understand.
But Rosenberg goes much further than this. In clear terms that can be understood by practitioner and layman alike, he explains the sad truth that the majority of software projects can end up like "Chandler", with budget overruns, schedule delays, endless bugs and missed requirements. All these insights are strikingly similar to the ones in the landmark book by Frederick Brooks, The Mythical Man-Month, which described the difficulties found in IBM software projects 30 years ago. Despite all the technical advances since then, the problems with managing software development projects have remained largely unsolved.
My hope is that Dreaming in Code will become a resource which can help enlighten all stakeholders to better clarify and coordinate software projects for the betterment of all. Five stars for bringing such a difficult and important subject to the general public.
on September 22, 2007
This book has widely been touted as the software version of Soul of a New Machine. Being an erstwhile hardware hacker, I loved Soul, and because I've since been corrupted by the sirens of software, I looked forward to Dreaming in Code. But maybe because my expectations were too high (I was led to expect something like Kidder's book) I was disappointed.
Kidder's book was successful because it reached a definite conclusion: the new machine was completed, and it met its goal of beating the VAX. During the course of the book, the reader vicariously became a hardware hacker, a power tool, pedal to the metal - you could feel the tension, the long nights into morning, the frustration.
In contrast, Chandler does not get finished in Dreaming; it's barely alpha test. Of course that's not the author's fault, but in that respect alone this book falls far short of Kidder's. As far as style, Rosenberg is not nearly as effective as Kidder in conveying to the reader the mentality, the spirit, the verve, the je ne sais quoi that embodies the software hacker. And I say that as someone who has had to hack software to the wee hours. In other words, I know what it's like, and Dreaming didn't really bring back wistful memories of the quest to get it working; how would a lay person get it?
That said, Dreaming is a start, but IMHO it still leaves a void for a book that really is a software version of Soul of a New Machine.
on March 8, 2007
In the tradition of Tracy Kidder's "The Soul of a New Machine", Scott Rosenberg's "Dreaming in Code" captures the inner workings of visionaries, managers, and programmers as they attempt to create functionallity using complex and imperfect software development tools. The amount of access he is given to interview, participate in meetings, and generally act as a fly on the wall throughout the development process is phenominal. It provides a unique perspective, not only on the technical issues which abound when trying to create any new application, but also provides the personal insights involving leadership, communication, creativity, and productivity. Mr. Rosenberg is not so much interested in writing a 'How to' book on computer programming, but one which examines the inner workings of creativity in the complex and frequently frustrating environment of software development.
Mr. Rosenberg delves beyond the Chandler Project to put this effort into perspective with the history of computer software development. He tries to show why, despite the advances in computer hardware and the development of advanced programming languages, the creation of a new application remains so challenging. He shows how many of the challenges facing software development in the 1960's as outlined by Fredrick Brooks in "The Mythical Man-Month" are still facing programmers today, such as, adding more programmers to a project behind schedule, only makes it farther behind schedule. He explains why managing creative programmers whether it is for the Chandler Project, or for the FBI's Virtual Case File, is terribly difficult and can frequently lead to abject failure.
"Dreaming in Code" is written as much for the non-technicial reader as for a technical one. It does not require an advanced degree in computer programming to be understood. It is more of a epic travelog through the shoals of modern software development. Can the visionary communicate that vision to managers and programmers so they can bend technology to create the application? In some cases they can and in others the vision must be modified.
The reason I did not give this it's fifth star is that the story remains incomplete. This is not the fault of the author, but rather due to the delays in completing the Chandler project itself. A preview version of the Chandler application is due out this Spring, several years beyond the original schedule. But dispite this frustration, I would recommend this book to anyone interested in the complexity of technology development.