Customer Reviews

How to Break Software: A Practical Guide to Testing W/CD

5 star:		(10)
4 star:		(5)
3 star:		(1)
2 star:		(0)
1 star:		(3)

 

 

This book is part of the new wave of testing books that challenge not only the conventional wisdom about test process, but also challenge conventional wisdom about how to teach and write about testing. People who prefer testing textbooks that preach paperwork and process will be shocked, shocked, to discover that there are a lot of us who think it's a tester's job to find important bugs fast. We want books that give us strategies for actually finding problems. Paperwork and process help some, but not enough. We need something more. We need test-designer-sits-down-at-the-keyboard know-how.

As a test designer, myself (and a competitor of Whittaker's) I can certainly find things to nitpick about this book. But I won't do that here, because the big picture is far more important. That picture is simply this: if you are confused about what to do to uncover problems in software before it ships, EVEN IF you have no specifications to test from and EVEN IF no one listens when you rant about "quality assurance processes" they should follow, then there are only a few testing books yet published that will help you. This is one of them.

Don't let the title or description fool you into thinking this is a book about ad hoc playing with applications with a goal to break them. In reality the book gives a structured approach to finding vulnerabilities in software. These vulnerabilities are weak points commonly found in software, and should be included in any test suite.

The vulnerabilities are classified by a fault model, then the book systematically walks you through the procedures used to attack and break the software. Each vulnerability type is addressed:
User Interface
- inputs and outputs, with 6 attacks for breaking common input flaws and 4 for output flaws.
- data and computation, with 3 attacks against stored data and 3 against computation and feature interaction.

System Interface
- 3 media-based and 3 file-based attacks against the file system.
- how to test the application/operating system interface.

The book also comes with a Windows application that helps you to create the hostile environment with which to 'attack' the software being tested. Therein lies the sophistication of the book, which employs fault injection as a technique. This technique is not commonly used in any but the most advanced testing environments, which raises this book's credibility from ad hoc to a serious approach to software engineering. More importantly, it provides test professionals, especially those who are testing Windows applications, a catalog of common vulnerabilities to address. More importantly, it teaches test professionals to approach parts of the testing process from an exploitation point of view - after all, their job is to break the software.

My initial misgivings about this book vanished as soon as I started reading it, and were replaced by enthusiasm by the time I was finished. This book addresses a niche topic, but deserves a place in every software testing library.

If there is an area of software development that needs to be codified and formalized, it is the procedures for testing the software before release. With the exception of software that does only a few tasks, it is not possible to test all possible paths. The number of possible paths expands very quickly so that it is effectively infinite, which means that it is so large that it might as well be infinite. Furthermore, this problem will only get worse as software continues to increase in complexity. Finally, the testing phase of software is relegated to the last step and is often considered to be a menial task by developers. Given these conditions and the general pressure of meeting a release date, it follows that testing is often cut short.
With all of this as a background, it would appear that testing is a hopeless task. That is not the case if the testing is done in a systematic manner, which is what this book will teach you. Whittaker is a computer science professor whose area of expertise is that of testing software. He breaks the process into two broad categories: user interface attacks and system interface attacks. Each of these areas is then split into separate attacks, seventeen for user interface attacks and six for system interface attacks.
The attacks for user interface are:

* Apply inputs that force all the error messages to occur.
* Apply inputs that force the software to establish default values.
* Explore allowable character sets and data types.
* Overflow input buffers.
* Find inputs that may interact and test combinations of their values.
* Repeat the same input or series of inputs numerous times.
* Force different outputs to be generated for each input.
* Force invalid outputs to be generated.
* Force properties of an output to change.
* Force the screen to refresh.
* Apply inputs using a variety of initial conditions.
* Force a data structure to store too many or too few values.
* Investigate alternate ways to modify internal data constraints.
* Experiment with invalid operand and operator combinations.
* Force a function to call itself recursively.
* Force computation results to be too large or too small.
* Find features that share data or interact poorly.

The attacks for system interface are:

* Fill the file system to capacity.
* Force the media to be busy or unavailable.
* Damage the media.
* Assign an invalid file name.
* Vary file access permissions.
* Vary or corrupt file contents.

Each of the attacks is presented using the subsections:

* When to apply this attack.
* What software faults make this attack successful?
* How to determine if this attack exposes failures.
* How to conduct this attack.

This approach leads to a very thorough demonstration of how to perform rigorous software testing in a limited amount of time. If I ever teach a course in software testing, this is what I will use as a text.
The book includes a CD containing two software testing tools, one of which I wish was available when I was developing software. While it is running, you can move a slider to have it bind memory resources and learn the point of memory use where your software performance begins to suffer. This is very useful, and is much easier than trying to load up many other applications.
Software testing is a critical area of development that is still in the process of being codified into patterns for reuse. This book demonstrates many of the currently available strategies and should be read by all members of testing teams.

I think that this is an exceedingly useful book.

Most books that purport to be about testing are really about something else. They're generally about planning, or process, or mathematics, or graph theory. Often, they're about making models of software so that you can demonstrate that there are indeed jillions of paths through a given piece of software--hardly news to anyone who's bothered to think about it for a while. Sometimes they're about the underlying theory of the thing you're supposed to be testing, such as "Web applications" or "security". All of these are useful things to think about, to be sure. Many of these books are large, and this one is small. I would venture to say, though, that few books talk about actual bugs as much as this one does, and provide such entertaining, cringeworthy examples.

This book is about testing, and it's about thinking about testing. It provides a set of theories of error, and follows these with worked-out examples of using those theories of error to find bugs in real software. What a concept.

In some reviews of this book, you'll find pious pronouncements about process; you'll see one that complains that this book doesn't have anything about testing J2EE applications; or that this book somehow applies only to Microsoft software. Those reviews all represent valid points of view, equivalent to the valid point of view that Moby Dick is a book about a big fish.

Some of the information presented is quite basic. Mind, as a tester, testing trainer, and user of software, I've seen a lot of software--a LOT of software--not Microsoft products, some written in Java, built with well-defined process... but some pretty basic bugs. Mission to Mars, anyone?

Some reviews also seem to believe that there is One True Way to develop and test software. That may be true, though I doubt it. But either way, it's unquestionably true that the followers of The One True Way are in the extreme minority, and the rest of us testers have to live by our wits, work under pressure in chaotic organizations, and find important bugs quickly.

The book inspired me to think about the way that I approach a piece of software that I haven't seen before. I know some things about the underlying operating system (whatever it may be); I know something about the way data is represented in binary coding systems (whichever one might be in use at the time); I know something about the construction of programs (irrespective of the programming language); I know something about the way the program interacts with humans and other software. I also know something about the way programs and programmers can screw up--that is, I know something about certain risks. As a real tester in the real world, sometimes that and the program are all I have to work with. Nonetheless, I can use those things to find bugs effectively. Besides, even if I do have a specification, it's invariably incomplete, or wrong, or out of date, or so thick as to be unreadable in the time I have to test.

The book is fun to read, too--some of the fun is in the Microsoft-inspired schadenfreude, and some is the relaxed, conversational style of the writing. One nice notion expressed in the book is getting together with other testers and talking about bugs for fun. Good point--I believe that people learn more easily when they're talking to each other and having fun.

So this book helped me by providing an example of a taxonomy of software interfaces and theories of error, and ways of attacking the software based on those interfaces and theories. I have my own theories of error, and my own models, too; this book helped me to think about them and refine them. It's not the One True Way of Software Testing. Good thing, too: there isn't one.

Don't get me wrong: I would love to have a perfectly written specification to which the software completely conformed. If I were confident that such a thing were possible, I would never have to test; by that fantastic definition, the software would work perfectly.

A good book should help and inspire you to think for yourself. If your mind is closed to extending the ideas in this book (or any other), you probably won't like it much--but then you probably won't be able to function very well when you move to a different development culture. That is (sad to say) you won't be a very good tester when you leave your cocoon. In fact, if your mind is closed, you're probably not a very good tester now.

On the other hand, I believe that this book is very useful if you keep your mind open, accept its lessons and examples, and apply them to your own projects, your own environment, and your own thinking. We need more testing books like it.

"How to Break Software" will guide you through the art of breaking more than just software. You will break into the thoughts of how to find amazing bugs in some of the world's most used applications and along the way you will learn how to find amazing bugs in the software you test everyday.

The practical knowledge gained by reading this book will have developers listening to your insight the next time you file a bug report.

The text is interesting and informative. The text is short and sweet. There are examples, which is nice. That is as good as it gets.

Much of the text is based on Canned Heat, a test environment which the author claims works of Windows 2000. This claim appears to be bogus. I have verified that the software does not work on several perfectly healthy Windows 2000 workstations. The software does appear to work on Windows XP.

The problem is that so much of the text is based on Canned Heat, so if you can't get this to work, much of the text is not that useful. The text begins to sound like a marketing brochure for Canned Heat.

If you go to the website, you may be dissappointed (as I was) to never receive a response. Every form I tried returned an error message. I have verified this with other interested parties. Emails to the addresses provided have not been answered.

My best advice is to avoid "How to Break Software" and "How to Break Software Security." For what little you get, the books are overpriced.

If the author wants his readers to take him seriously as an authority on software testing, then he should spend more time testing his own software before shipping it out. Frankly, I find it hard to take him seriously if this is the best he can do.

This is an awesome book for software testers with less than two years of experience. If that's you, then you'll definitely get a lot of value out of it. You should buy it without hesitation.

What it does well is provide a clear understanding of what it means to "think like a tester." I recommend also that QA managers give it to their greenest team members. They will undoubtedly become better testers as a result.

However, if you're someone who has been in QA for several years, all of these attacks are going to be obvious--and ones that you almost certainly will already consider a part of your regular testing repertoire.

Even still, it's fun reading about some of the extant bugs in shipping Microsoft products, with step-by-step instructions on how to make them happen. I also like the freeware that comes with it--it's limited in scope, but still quite useful for certain testing situations.

This slim volume presents a series of testing techniques, dubbed
"attacks", that target common software errors. The list is based on an
empirical analysis of a large number of bugs found in commercial
software by the software testing labs at the Florida Institute of
Technology. Each attack is illustrated with an actual bug found in
everyday software. The analysis and the examples are mostly drawn from
Microsoft software.

James is one of the most engaging speakers to be found at software testing conferences; in part this is due the many rich experiences that he and his associates have encountered in their long tenure in this domain. The good news is that James has endeavored to distill the essence of his lively and effective presentations into a concise and easy read.

This book provides overviews on a series of generic attacks (or software testing techniques) that can be applied to virtually any software application. Although it should be noted that the application used to illustrate these techniques is frequently MS Office, and consequentially my give the false impression that the book is only aimed at GUI app's running on a Windows platform.

Chapters 2 & 3 describe a total of 17 different ways in which input data can be manipulated with a specific test objective in mind. For example, forcing an internal data structure to store too many (or too few) values. Chapter 4 lists an additional 6 attacks that focus on creating an unpleasant environment for the application to run in. For example, varying file access permissions, filling-up the free-space on a hard drive, or starving an application of CPU usage.
Whether you elect to think of these types of tests as "negative", "non-functional", "robustness", or some other category, they all have the potential to Break your Software, and as such are all worthy of consideration when you are determining how best to mitigate this possibility. In summary, I believe James has done an excellent job of describing a solid collection of techniques for unit-testing software.

In the vain of "full disclosure", you should know that I've known James for several years, and consequently I cannot be considered a completely impartial reviewer.

Great book covering orthodox and some unorthodox thoughts on testing methods and some specific guides. While the college trained tester or the person with many years of experience will probably find this too basic, it's still worth thumbing through for a new idea or two. The most common testing faults are missing common issues, such as testing for bad or negative input.