Customer Reviews

Just Enough Software Test Automation

5 star:		(7)
4 star:		(1)
3 star:		(0)
2 star:		(1)
1 star:		(0)

 

 

This book is written for the in-the-trenches testing practitioner. Before describing the book and its strengths, I need to state that the authors' views of certain aspects of software engineering dramatically differ from kine. Specifically, they express some disdain for applying a life cycle approach to testing in general and test automation in particular, and also don't seem to see the value of maturity frameworks, such as the CMM. On the other hand, they are forthright about their focus on the practitioner, and are strong proponents of process. My views differ from theirs in that I see the value of bounding processes within a life cycle flow, and also see the value in measuring capability. While my perspective may not be meaningful to the practitioner who is actually doing the testing, it does take into account the realities of managing an IT organization.

Regardless of my opinions and views, the authors have put together a powerful, sensible approach to test automation. Key strengths include:

- Pragmatism, including compelling counter arguments to my own views (especially in the first two chapters titled "What Is Just Enough Test Automation?" and "Knowing When and What to Automate". I particularly liked the distinctions between processes, and life cycles and tools.

- Going straight to the critical success factors, such as requirements as the entire basis for test planning, and ensuring traceability throughout the development life cycle. In addition, the frank discussion of limitations of some testing tools, and the associated high maintenance associated with scripts, is illuminating. I also liked the way that the book shows what can be automated, and, more importantly, what cannot (or should not) be. It also reemphasizes the importance of developing a test strategy and test plans, and how automation tools fall short in some areas. An invaluable part of this aspect of the book is the discussion of test scripting languages and their strengths and weaknesses.

- Examples based on real tools, with an emphasis on Rational's TestStudio. Mercury Interactive's WinRunner is also used to illustrate key concepts of the Test Plan Driven framework that is discussed later in the book.

- Material that hands-on practitioners can use. While I have a high regard for the Automated Test Lifecycle Methodology that is proposed in an excellent book titled "Automated Software Testing" by Elfriede Dustin, Jeff Rashka, John Paul, that book is more for implementing and managing automated testing within the context of a life cycle, and isn't a topic to which the audience of this book will relate. Indeed, the real strength of this book is the fact that no other book on automated testing talks to the practitioners. In addition, the material covers unit, integration, and regression testing from the practitioner's point of view.

- Advanced topics including data-driven approaches to testing that ties into automated suites, hybrid approaches that combine manual and automated elements, and how to develop test plans and associated artifacts.

Despite my disagreement with some of what the authors views, I have to give this book my highest endorsement because, in my opinion, it's well thought out, provides one of the most thorough discussions of test automation at the practitioner level I've encountered, and is technically flawless.

As a software tester, I have been witness to several well-meaning attempts at software test automation end up in failure.

Usually the unsuccessful attempts end up at some point with a declaration similar to the following: We just do not have time to automate testing right now!

The automation tool becoming unused Shelfware usually follows this scenario. Meanwhile, the project moves forward without the many benefits of automated software testing.

The reasons for these failures have been many. I believe that every one of these reasons has been addressed in this practical hands-on guide.

This book stresses the importance of front-end test planning and test design activities.

It is especially effective at explaining why the traditional capture/playback model that testing tool vendors have championed for years just does not work most of the time in the real world of testing applications.

The case for data-driven automated testing is logically and thoughtfully presented. Not only is the data-driven approach presented, the book explains how to actually implement sound data-driven test scripts. The fundamental concept of separating data from function in the development of test scripts is clearly explained.

A very important section of the book offers practical advice on whether or not to automate, as well as when and what to automate.

Because the authors are actual practitioners they offer useful advice on test automation from the test automation developers/users perspective. It includes very pragmatic advice obviously gleaned through actual hands-on experience on what to do and what not to do when designing and implementing a test automation infrastructure.

I found the book to be really worth the read. I would advise anyone interested in the successful implementation of automated software testing to read this book.

"Purchasing a software testing tool suite does not constitute implementing a software process". Wise words from Dan Mosley and Bruce Posey in "Just Enough Software Test Automation"; maybe some development managers need to take heed.

Too many times have automated test tools become shelfware, or the cost of maintaining the scripts prohibitvely expensive. The authors of this book offer a simple and easy to use data-driven framework that can minimise scripts and human effort. They place their framework within the Rational Unified Process (RUP).

The book offers actual and detailed advice that goes all the way down to code and script templates. Based mostly on Rational tools, the book gives lip service to Winrunner and anything said can be translated to any automated tool.

Their open-source framework is the Control Synchronized Data Driven Testing(CSDDT). Data to be input, keywords to navigate through the application and actions to be performed are held in the spreadsheet. There are four main scripts: A Main script that reads and processes the records; a window selection script, a tab selection script, an action script and error handling script. Data input is held in an array and there is a comment field that documents the test record. Your application code is held in a switch statement, and it is highly conceivable that your project can have single figure script numbers. There is also a script that converts the spreadsheet data in a .csv file that is read by the Main script. There is detailed
information given on how to implement the CSDDT. It is a framework I use and am pleased with.

There are two interesting chapters on Unit and Integration testing. Like eXtreme Programmers they believe in automating unit tests that pass at 100% before submitting for build. They correctly argue that unit tests should be constructed before development code is written and they also point to the xUnit group of tools.

They make insightful points about the necessity of integration testing: Could you not help but identify with the following statements: "... We have seen two chronic problems: First, the build fequently does not install on system test machines. Second, the fact that unit and integration testing has not been done previously forces the system test team to do tests that development should have already executed." Again they also argue for automated integration testing else "it will not get done."

I feel however that Mosley and Posey's ideas need to be infused with agile values and practices. For example do we really need improved software requirements documentation, verbose Test Plans and meticulous test design when requirements change so much? Do we really need all these Rational tools and the time it takes to use and update them? Can we not make automated functional tests an integral part of requirements? What about Pair Test Programming? How are we going to increase oral communication? Is devolopment and test a false dichotomy? These kind of issues also need to be addressed as we begin to construct software in a radically different way.

I purchased this text while researching automated testing for my Master's paper. Mosley and Posey focus on the pragmatic aspects of implementing automated testing. Since the focus of my paper was on justifying software testing automation, I particularly apprectiated their straight forward arithmetic on tool justification. However, they go even further to address where the cost justification actually exists in areas where it may not be as obvious. This book, however, does not put a great deal of emphasis on the administrative methodology surrounding software development. It tells you this up front and addresses key success factors to implementing automated testing (ie don't automate the testing of something that's not completed or working.)

The only quirk I found in the book was the diatribe against the benefit of CMM or knowledge of other models. I understand their point, which is that these models don't really add value to the hands on aspect of testing or developing software. However, from personal experience, I have seen a greater tendency in developers to consider many of the points they make if their background includes an appreciation for the types of things that should happen in a mature organization.

"Just Enough Software Test Automation" written by Daniel Mosley and Bruce Posey describes test automation from a practical perspective gained from much experience by the authors with commentary and contributions from several well respected leading practitioners in the field. Key fundamental points are emphasized and explained throughout the book with supporting descriptions and concrete examples for using a data driven framework to implement and maintain software test automation.

While the book is well written and easy to read for someone who's familiar with software testing and who may have some experience with test automation, it assumes that the reader does have experience in the field.

The authors begin by reviewing important fundamental practices of software testing that are critical to effectively sustaining both manual and automated testing efforts. They provide recommendations on how to approach test automation for each phase of the software development lifecycle beginning with requirements through the final stages of testing. The authors present very specific recommended techniques and tools and offer many examples using a data driven framework with emphasis on Control Synchronized Data Driven Testing (CSDDT). Most often the tools mentioned and examples provided are those offered by Rational, Inc. as well as the use of Microsoft Excel. Frequently, automated tools from other vendors are referenced when they are applicable to the technique being discussed. They provide references to books and to several web links that offer sources of information on similar frameworks using other tools. The authors include useful information in the appendices such as a captured discussion on the subject of the data driven approach by leading practitioners, automated testing definitions, an example test automation project plan, and a test automation project work plan template.

Some of the key points in the book include the importance of identifying and documenting application and testing requirements as well as documenting test cases and conditions. They emphasize the importance of planning for test automation and implementing it similar to any other software development effort. This includes the separation of roles between test designer and test implementer. They urge that test automation be performed at most phases of software development including unit testing, but that it primarily be used for regression testing. The key success factor for test automation is the maintainability of test scripts. The authors point out that this is extremely difficult using a capture/playback method of implementation and that a data driven approach using modular scripts has shown to be much more successful in the long run. The authors do a good job of describing these key points and then making specific recommendations with examples on how to implement them.

As a practitioner of test automation, and reviewer of this book, I very much agree with these key recommendations and support the authors' intent to educate people implementing test automation as these key points can be the difference between failure and success.

My company has had an automated test team (for regression testing) in place for about 2 years now. Our job is to assist projects with their testing effort and to give added benefit (Side Note: All projects we have taken on have EVENTUALLY praised us for our efforts and relied on our team to take over their regression efforts.). Our team is a diverse collection, not necessarily programmers or even testers, so the initial effort is very impressive. However, as a newbie to the company (only 6 mos. but coming in with 12 years of testing experience but no prior experience with developing an automated test framework), I read this book and found it extraordinary. It recognized all of the pitfalls that we currently are experiencing with new applications coming on board (our current framework is no where near application-independent). Our newest tool is QTP, and I translated the code from the book (and website) into VBScript and made a sample for testing our most recent web application. It works like a champ and the framework is so easy. I have presented it to our automation team for review. I am confident that they will find this methodology should be the basis for each new project we take on. If you don't have a background in testing or any automated testing background, you will have nothing to compare this methodology to and so, it's strengths may not be easily realized. As well, if you do not have to support the testing of multiple applications, the same holds true. I am writing this basically hoping that Daniel can give me more insight into implementing this strategy into a QTP environment. Right now, I have a big main action that holds all of the "DDScripts". If this methodology could be translated into a QTP environment in a more structured way then I am sure to be the queen of all automated testing at my company!

If you are developing software test automation, this is a very useful book. Especially helpful are the chapters on using data-driven testing frameworks, which reflect lots of hands-on practical experience. The framework examples are the best hands-on guides I have seen published on this valuable technique. This book fills what has been a big hole in the software test literature.

Every time i see a "practical" book that contains a dozen pages about equivalence classes I want to scream. This book is no different. Don't know what black box testing is? Well, here you find it.
The authors explain how to report defects in Excel (is anybody out there still doing that?)! Nowadays one would certainly use Bugzilla, Jira or the like.
Test automation is always about tools (even if it is a self created framework) but this book is general.
Test scripts should follow standard programming practice (which is even explained!). Well, it is code, so that is no great surprise.

The one practical thing is the statement about where to start automation: "where it feels right". Yeah, thanks for that tipp.

Conclusion: Don't waste your time. You will probably be better of if you grab a book about JUnit testing, Fitnesse or the like.

Stephen

I find it hard to believe you even read the book. You are obviously a programmer not a tester. If you are like most, you really have no idea what testing is, including Unit testing. Our book is aimed at Functional, Integration, and System testing, although we have one short chapter on Unit Testing. Had you been a professional test engineer, you would have recognized that this is the focus. In Preface, we even advise the readers who are not interested, or who are already familar with the test data methods, to skip that chapter.

It is also obvious that you did not absorb the discourse on the fact that software testing is only as effective as the test data it implements. This goes for unit, function, integration, system, or regression testing. This holds true for test data developed in Junit, or any other automated test tool. A test script is just a delivery vehicle. The test data affect the application under test, whereas the test script does not. If the test data is not testing the correct aspects of the AUT, it is worthless.

Testing is all about the correct formulation of the test data. Putting down formal test case design methods shows testing immaturity on your part. If you do not implement formal testing methods, e.g. Black Box/White Box test methods, you will create useless ineffective test data. Your testing will be an exercise in futility. Furthermore, you will have wasted your company's time and money doing the testing.

From your review I concluded that there are several books, which would better fit your needs. They are "JUnit Recipes : Practical Methods for Programmer Testing" -- by J. B. Rainsberger, "JUnit in Action" -- by Ted Husted, Vincent Massol, and "JUnit: The Definitive Guide" -- by Derek Lane. Amazon sells all of these titles.

Dan Mosley