Customer Reviews

Perfect Software: And Other Illusions about Testing

5 star:		(7)
4 star:		(4)
3 star:		(1)
2 star:		(0)
1 star:		(1)

 

 

If you've been in testing long, you've probably heard the standard impossible questions:

"Why didn't you find that?"
"Why would we hire a human to test when we can automate our testing?"
"This NEEDS TO WORK. And we need it next week. Period"

If you drank the deep draught of testing, if you've struggled with it, wrestled with it, and gotten good at it, I have only one thing to say:

This probably ain't the book for you.

This book is for the people who are asking you the impossible questions.

Seriously. This is a nice evening read and will provide an executive, manger, or developer with enough information about testing to (A) understand some of the challenges of the role, (B) set appropriate expectations, and (C) communicate those expectations clearly.

In other words, it can be the difference between a sane life and an insane one - if only you can get the right people to read it.

Now, the style of the book is plain prose and folsky story - it's the kind of thing you can read in an comfortable evening, or perhaps, two. It doesn't have fancy metrics and graphs, but we all know what Mark Twain said about that.

So if you are a seasoned tester, this book might not be for you - it's for your boss, your bosses boss, the customer, the CEO, and The New Guy. I wanted to get my management team the book as a Christmas present, but somehow all those folks had read it before Christmas ... I suspect a co-worker.

Absolutely great for it's niche. Top Flight. Buy two copies to give away today!

A really useful book. Testing is the most misunderstood of software related activities. Even by software development professionals.

The book explains what we can expect from testing, what are the main challenges, and what is wrong with common practices and attitudes.

Short, well written (as always with Gerald Weinberg's books), easy to read, without technical details, this book is a good introduction to the realities of software testing, for every stakeholder of software projects and of software products.

I know Jerry Weinberg. I like him, treasure his advice, and have read and liked many of his books. This is another of his books that I like.

Jerry writes around testing software. I say "around" instead of "about" because this book isn't about the technical aspects of testing software. I find it to be about the personal and thoughtful aspects of testing software. More importantly, it is about information - communicating, considering, and applying information.

This all starts with Jerry's definition of Testing a System:


a process of gathering information about it with the intent that the information could be used for some purpose.


Aha! Gathering information that we can use. I agree. I have often pleaded with people on projects to understand this about testing.

I don't know how often I have heard screams of, "The test failed! The test failed!"

No, the test didn't fail. We learned something from the test, so it was a success. Perhaps you were disappointed by what you learned from the test, but the test didn't fail.

This is why I love Weinberg's book about testing. Its contents will last beyond the current and next three generations of test tools and techniques. It cuts to the heart of testing in a timeless manner.

Simple, right? Wrong. The second half of the testing definition is about using the information, and people use the information. Now we come to a huge hurdle. "Information is power." Some people crave information, and many people fear it, even to the point of hiding it or lying about it.

Weinberg discusses the fear at length, how to spot the fear, and how to work with the people who are afraid.

Yes, I recommend this book. I can give it to a manager at any level and trust that if they read it, they have the potential to be a much better manager. I can give the book to just about anyone - especially people who have nothing to do with software or testing software. It contains much about how people gather, communicate, consider, and use information.

Exhaustive testing of software is generally impossible; in all but the most simple of cases the number of possible paths is effectively infinite. By that I mean that while the number is finite, it is so large that only a tiny percentage could ever be tested. So in practice the number is infinite. Since humans are prone to error and the number of possible paths where a bug can be introduced into a program is also very large, the bug-free (perfect) program is an unattainable goal. It is no fun, but that is just the way things are.
Being driven by optimism and necessity, humans still create millions of lines of code every year and many of these programs are released into the untamed wilderness known as the customer base. Therefore, conscientious developers have no choice but to develop a testing program that will have a reasonable chance of identifying the most significant and likely bugs in the software.
As Weinberg points out, some people do make the choice to do very little testing, sometimes relying on an approach like "If we get lucky, no one will experience that bug." Others simply redefine the bug as a feature and move on. Neither of these approaches is ethical nor will it provide you with legal cover. In a very witty yet understated style, Weinberg describes some of the circumstances he has encountered where people simply denied there was a problem and if that failed tried to hoist the problem onto someone else.
As is the case in most areas of software development, the solution lies more in the realm of group psychology rather than technical expertise. Having authored the classis "The Psychology of Computer Programming" Weinberg is an expert at finding what solutions are available to this seemingly impossible problem. It starts with eliminating all illusions and delusions and accepting the inherent limitations of what software testing can do. After that, one develops a testing process that is executed concurrently with the development of the software. The first step is to test the modules by creating a set of reasonable coverage test values. Varying from module to module, this set covers some of the most likely entries as well as the bizarre ones and the ones on the limits.
Once the tests on the modules are complete, the next step is to integrate the modules into a functional unit. Once the unit is stable and can be run, an additional set of reasonable coverage test operations is created and executed. In each case, detailed knowledge of what the software is supposed to is used to develop the sets of test values.
Weinberg is also forceful in debunking the current state of the automated testing of software, a tactic that he dismisses. The strongest argument against using software to test software is grounded in human psychology, namely the easily adopted assumption that if it came from a computer then it must be right. This belief in the infallibility of technology has a strong tendency to dumb-down humans and that is to be avoided if at all possible.
As stated before, the quality software developer has no choice but to test their product. If you want to maximize your likelihood of releasing a quality product, then your first step is for everyone in your team to read and discuss this book. Doing so will clear a lot of the yucky stuff out of your path. So while you may still step in it, the pile you squish is either of your own making or one of the few that Weinberg does not cover.

Published in the online "Journal of Object Technology" reprinted with permission

As a software testing consultant, I have had the same mentoring session about one thousand times (or at least it seems) with different IT, development and testing managers. These conversions are on the same topics as Jerry covers in this book and the insights he gives are ones I will be adding in my consultations. The value of this book in my view is that it concisely cuts to the key issues in getting software right. Too many people in decision-making capacities hold misconceptions about what it takes to have a successful software project and/or product. The biggest misconception, which is why the book is titled as such, is that perfect software is possible.

Companies hire testers to "find all the defects." When they don't, the testers get blamed and perhaps fired. This book explains the reasons why we don't have perfect software, but gets much deeper. It's funny that when people want "to go deeper" in their understanding of testing, they want to learn about tools and techniques. This book beautifully explains the deeper issues of software testing by how we think and communicate about this very important aspect of software development. My copy is now tattered after carrying it from trip to trip, but I keep finding the gems of wisdom. It is easily readable, but will make you think about what we do in testing - and why we do (or don't do) things. I highly recommend it to testers and the people who manage them - all the way up to CIO!

Perfect Software is a high-level look at some of the "big picture questions" about testing, such as: Why do we have to bother testing? Do we have to test everything? What makes testing so hard? Is perfect software possible? Why can't we just accept a few bugs?

It's a small book (under 200 pages), and a quick read.

Here's an interesting example that will give you a flavor of the book.

In chapter 3 "Why Not Just Test Everything?", Weinberg has a section called "There are an infinite number of possible tests." He talks about a backdoor placed into a highly secure program whereby the ordinary password protection could be bypassed by typing W followed by three spaces, then M followed by three spaces, then J followed by exactly 168 more keystrokes without once using the letter L. Then he writes:

"Do you get the point by now? If you didn't guess that the number of tests required to exhaustively test software is infinite, or at least "a number greater than I could run in my lifetime", you didn't understand the point of this chapter. Now you do."

If you are looking for a "how to" book, you should look elsewhere. If you are looking for a "why" (and sometimes "why not") book, this might be for you.

I bought this book on the advise of a colleague. Part way through the first chapter I was hooked. I didn't really learn a whole lot about testing, but I did learn a lot about myself as a tester. Chapter 7 almost hurt me with the realization more than a few times of "I've done that", although also provided a deeper understanding of how to deal with people's defensive reactions.
I do recommend that most testers give the book a read, and then give it to your manager. Many developers, business analysts, test managers, and executives could learn much from this. I'm now planning on buying this book for a few friends on previous teams that still struggle with some of the points within.

Humans aren't perfect, so why do we expect it of our software? "Perfect Software: and other Illusions about Testing" is a guide for software developers telling them that perfection is an unreasonable goal when it comes to software. With charm, Weinberg entertains as he informs about how a good software development team moves forward and does what they can to make the product the best it can before they shove it out into public. "Perfect Software" is a must for any head of a software development team.

Quite useful for people who are not sure how testing fits into development lifecycle. It helps testers re-think their missions; helps managers to have a broader look on testing significance. It values communication within a team quite a lot.
Rather general book that the main aim is to make software folks aware of testing.
It views testing from many perspectives not neglecting the business and psychological ones.

Valuable and worth-reading although senior testers may find it obvious in large part

Like many who follow Jerry Weinberg's works, I love the humanity of this book. The ineraction and people seem like many I work with, facing these same challenges.


I love the book... and don't wish Jerry to change it.

I would like readers to consider these thoughts ....

Nevertheless there is a gap between this book and the realities of verifying the correctness of software. This area is one many software developers and many testers and quality leads are unaware of.


1) Proof: For example, It is a profound truth that like some logic, software logic can be proven correct. Some software can be proven correct mathematically (and indeed must be). The approach to prove correctness (or the lack of incorrect properties) is called formal methods. Some software developers write proofs of linear time logic which is converted to c or java. Software that meets all necessary properties and has no undesirable states could be considered "perfect" when proven so. (Example a dishwasher control program could be proven correct for all time, and meet all requirements for that system.) The barrier to reach perfection (satifaction of all requirements and the application cannot attain states).

2) Types of verification : What should the tester know about profiling and instrumenting software for reliable software.

3) Where the numbers are and where does software break: What in software systems provide metrics of performance and correctness in testing! Example testers should know that databases keep statistics on query performance. Testers should know that developers and tests can validate the lack of memory leaks and deadlocks. Testers should know how to keep performance history.

4) Tools for software quality: What are the types and how are they used. (Example what are static analysis tools (e.g. PMD for java). What are dynamic analysis tools.

5) Defending the test environment and test harnesses: Testers should know how to present the benefits of maintaining the testing environment and test tools.

6) When can something be tested. What is maturity in software. When does the many eyes model fail.

A good introduction is The Spin Model Checker by Holzmann.
The SPIN Model Checker: Primer and Reference Manual

Thanks