Customer Reviews

Best Practices for the Formal Software Testing Process: A Menu of Testing Tasks

5 star:		(5)
4 star:		(0)
3 star:		(0)
2 star:		(0)
1 star:		(0)

 

 

First, this book is not primarily for software test and QA professionals who are working in 'typical' organizations. As noted by others, the approach this book provides is best suited to organizations that are at least at CMM level 3. Moreover, unless software engineering practices across the organization are mature the approach will probably fail. However, that does not prevent even a Level 1 organization from selecting best practices and tasks set forth in this book and applying them. The net result will be an incremental improvement, and may be the catalyst for larger improvements with a small win.

That said, this book is invaluable to mature organizations that are committed to software engineering at the defined, managed or optimizing levels of maturity. It distills formal test practices drawn from a variety of sources and the author's experience into a succinct, process-oriented guide. The model itself is presented in IPO (Input-Process-Output) diagrams that start at a high level to describe the process itself, and drill down into successive levels of detail in level 2 and 3 IPO diagrams. This process-oriented structure gives a great deal of clarity to a complex set of processes that touch all milestones in any SDLC.

I like the fact that the model proposed is not rigid, but can be tailored to development life cycle approaches ranging from waterfall to agile approaches. Chapter 8 gives advice on how to accomplish the tailoring without breaking the integrity of the process. I also found the appendices useful, especially Appendix B (preferred practices) and the plans and templates provided, and Appendix C (testing processes evaluation questionnaire).

If your organization is pursuing CMM level 3 or above, or are contractually required to have a formal software engineering process or process capability, this book will address the software testing process areas of a larger initiative. However, do not overlook some of the small wins a chaotic organization can achieve by using many of the ideas in this book.

I purchased this book based on Amazon's information and the reviews. My purposes in doing so are different than most who would / should consider it -- to audit and assist US FDA-regulated companines in compliance, including the requirement for validating software in medical devices, or in manufacturing and data systems used to manufacture FDA-regulated products (devices and drugs). Given that caveat, Rodger's book is an excellent resource. He supplements his narrative with numerous diagrams which he defines as describing a process and a "set of tasks that can be used to implement or improve a formal testing program".

His stated assumptions (a pre-existing formal system in place at a company; specifically defined by the Capability Maturity Model / CMM 3-4+; with a separate reporting structure -- or, as he stated, "the full blown model described in this book details a full-featured formal testing process that is applicable to large programs and that would fully support programs deliverable to state and federal governments, or on programs delivering safety-critical systems or having significant impact on corporate profits" ). What he describes would fit well with the FDA's GMPs(Good Manufacturing Practices), a quality system similar to but more stringent than ISO 9001 / 13485, and various FDA /Agency guidance documents on software validation (a series of structured documentation and testing requirements).

Although presented for / geared to a large corporation w/ greater resources, I would argue that the basic principles he discusses, and the systems approaches recommended, are adaptable, and 'down-scaleable' to any size company. It also provides a model / target to aim for by any software developer / provider, including (especially) the small shop, a requirement trend that will probably only increase, and globally -- and providing such companies a competitive advantage, and enhance the Intellectual Property (IP) value of the resulting product. His strategic level and test level discussions also provide the basis for input to software portions of a company's documentation -- the Quality Manual, SOPs (standard operating procedures), and WIs (work instructions) for both engineering and testing / QA.

Certainly, the recommendations, systems, documentation and efforts outlined in this book, if followed in principle, would greatly reduce the problems experienced in software / hardware implementation projects, including some recent failures / delays receiving nationwide publicity.

As such, it has proven to be a valuable addition to my consulting library, and a useful reference in conducting audits, making recommendations, and developing validation protocols.

Testing large software projects is a very difficult task. Testing can only reveal the presence of bugs, not their absence and it is impossible to cover all possible pathways that the software can traverse. Furthermore, and what is the worst, "simple" changes made in software can cascade across many modules, requiring the re-testing of all affected modules. Therefore, any testing plan must incorporate repeating tests based on feedback. Finally, testing is something that must be done, so there is no choice in the matter.
The practices described in this book are all modeled using Input-Process-Output (IPO) diagrams, which are labeled state diagrams. The states in the diagrams are partitioned into three sections, input, process and output. Inputs are represented as labeled arrows, which can originate from another state, but do not have to. The process section describes what is to be done at that stage and the output section has labeled arrows exiting the state that then go to the next state. Multiple inputs and outputs are possible and the flow can loop back to a previous state.
Each state is described in the text, where the inputs for the state are explained in detail. Applicable feedback from all persons with a stake in the operation is discussed as well as feedback that this state can give to previous states. The process is described and then the outputs that the state will send to later states are explained. Feedback that may be received from states later in the sequence is then described.
What is most impressive about these modeling diagrams is the extensive allowance for feedback. The complexity of the testing process and the consequences of the results means that testing can form a feedback loop that exhibits many of the characteristics of chaos. A loop is chaotic when small changes can cascade into very big changes. The way to prevent this in any process carried out by humans is to incorporate damping mechanisms. These features reduce the impact of any result so that they do not grow beyond the bounds of the system to handle them.
When faced with impossible tasks, something that software testing has now become, the best that you can do is examine a subset composed of the most likely scenarios. By applying the models in this book, it is possible to raise the level of your testing quality to the point where you can be confident in your software

Rodger Drabick has written a useful book for those working on test efforts in formal environments. By "formal environment" I mean a CMM level 3 or above, SPICE, or ISO registered program, or one regulated by a government agency like the Federal Aviation Administration or the Food and Drug Administration in the US. There have been plenty of templates and standards floating around for years on what to write down for such tests, but precious little describing how to manage the formal testing process. This book fills that void.

The book has the following strengths:

1. Rodger provides a clear, complete roadmap for those new to testing in a formal environment. You could follow this roadmap, with the tailoring advice he provides, and do a competent job your first time working on such a project.

2. Rodger manages to cover a dry topic like formal processes in an engaging fashion. He includes useful "stories from the trenches" and lessons learned from his experiences, which bring the topic to life.

3. Rodger transcends and complements the IEEE 829 test documentation standard by harnessing a formal process model to the templates. Rick Craig's book, *Systematic Software Testing*, does this, too. However, Rodger's book is a good complement to Rick's in a more formal environment.

4. Finally, Rodger's book is browseable. You can skim sections, get the gist, and return later for a more detailed read.

The book has a few minor weaknesses, which I should mention:

1. The bibliography is a bit thin. The body of useful and interesting test knowledge extends well beyond what's shown there.

2. Rodger is careful to note that the processes he describes are for formal environments. So, the brief section on Extreme Programming struck me as somewhat of a non-sequitor. However, readers will probably simply skip this section if they aren't using XP or other agile approaches. If readers are using XP or some other agile approach, I'd recommend a different book on the testing process first.

In the domain and user community Rodger is addressing with this book, neither concern should dissuade someone from buying the book.

Anyone testing in a formal environment will likely benefit from Rodger's book. If you are testing in a formal environment for the first time, reading Rodger's book might well go from a good idea to a survival requirement. Formal environments are the world Rodger has worked in for decades, and no one else has brought his wealth of experience in that world into writing a book about the testing process.

Rodger Drabick has written a comprehensive and practical guide to formal software testing process. Everyone involved in software testing will benefit from his years of experience and his revealing insights. I've been in the testing field for more than 10 years, and I'm learning a lot from this book! This is a great textbook for new testers, a step-by-step cookbook for new managers, and a great reference book for everyone in the testing world. Rodger takes what can be a difficult and elusive process and explains it thoroughly, using graphic models as well as real-life examples. The best part is that he explains how to adapt the testing process in various situations, even Extreme Programming projects. He gives specific advice to testers at every level, most valuably for new testers and new test managers. Just a few of the things you can learn from this book: How to apply IEEE standards to your project, how to break a project into testing tasks, how a process model can be used as a training tool for new test engineers, how to apply the model to achieve a specific CMM level. Rodger's aim is to help the reader improve the testing process, thus improving product quality. He emphasizes that the testing and development organizations must work together throughout the software development life cycle - not a new idea, but not done nearly enough either. Don't be thrown by the technical-looking IPO diagrams and formal terminology - this is a common-sense approach that can be applied just about anywhere. Rodger doesn't expect you to run out and implement this entire model - he just wants to help you improve on what you do. What if your a tester who gets code to test but no requirements? Pretest and posttest meetings wouldn't be hard to implement, and they'd improve your process. This is the type of advice that makes this book golden. The appendices add even more value, with info on CMM, preferred practices, a way to evaluate your current practices, and a primer on test execution. The book's references to other works will let you explore other areas of testing.