Implementing Automated Software Testing: How to Save Time and Lower Costs While Raising Quality [Paperback]

Elfriede Dustin (Author), Thom Garrett (Author), Bernie Gauf (Author)

Book Description

Publication Date: March 14, 2009 | ISBN-10: 0321580516 | ISBN-13: 978-0321580511 | Edition: 1

“This book fills a huge gap in our knowledge of software testing. It does an excellent job describing how test automation differs from other test activities, and clearly lays out what kind of skills and knowledge are needed to automate tests. The book is essential reading for students of testing and a bible for practitioners.”
—Jeff Offutt, Professor of Software Engineering, George Mason University

“This new book naturally expands upon its predecessor, Automated Software Testing, and is the perfect reference for software practitioners applying automated software testing to their development efforts. Mandatory reading for software testing professionals!”
—Jeff Rashka, PMP, Coauthor of Automated Software Testing and Quality Web Systems

Testing accounts for an increasingly large percentage of the time and cost of new software development. Using automated software testing (AST), developers and software testers can optimize the software testing lifecycle and thus reduce cost. As technologies and development grow increasingly complex, AST becomes even more indispensable.  This book builds on some of the proven practices and the automated testing lifecycle methodology (ATLM) described in Automated Software Testing and provides a renewed practical, start-to-finish guide to implementing AST successfully.

In Implementing Automated Software Testing, three leading experts explain AST in detail, systematically reviewing its components, capabilities, and limitations. Drawing on their experience deploying AST in both defense and commercial industry, they walk you through the entire implementation process—identifying best practices, crucial success factors, and key pitfalls along with solutions for avoiding them. You will learn how to:

•     Make a realistic business case for AST, and use it to drive your initiative
•     Clarify your testing requirements and develop an automation strategy that reflects them
•     Build efficient test environments and choose the right automation tools and techniques for your environment
•     Use proven metrics to continuously track your progress and adjust accordingly

Whether you’re a test professional, QA specialist, project manager, or developer, this book can help you bring unprecedented efficiency to testing—and then use AST to improve your entire development lifecycle.

Editorial Reviews

About the Author

Elfriede Dustin, Thom Garrett, and Bernie Gauf work together at Innovative Defense Technologies (www.idtus.com), which specializes in the design, development, and implementation of automated software testing solutions.

Elfriede Dustin has authored multiple software testing books and articles based on her many years of actual hands-on automated software testing experience. Elfriede leads IDT’s efforts in automated software testing research programs.

Thom Garrett has experience in planning, testing, and deployment of complex systems for DoD and commercial applications for companies such as Public Broadcasting Service (PBS), Digital System Resources (DSR), Inc., and America Online (AOL). Thom received a master’s degree from the University of San Francisco.

Bernie Gauf is the president of IDT. Bernie has been invited to participate in numerous DoD panels associated with the use of new technology, testing conferences, and as a guest speaker to share his insights on automated software testing.

Excerpt. © Reprinted by permission. All rights reserved.

Preface

Is your test automation strategy a losing proposition? Are you soured on the notion of automated software testing based on less than adequate past results? Are your test automation silver bullets missing their mark? Are you disappointed in your test automators? We at IDT1 have identified a boilerplate solution, strategies, and ideas, all provided in this book, that can help increase the chances of your automated testing success.

Given the arsenal of system and application software testing strategies, techniques, and solutions, automated software testing is one of the most effective practices that if implemented correctly can help increase testing efficiencies and ultimately reduce the testing cost while contributing to increased systems and software quality in terms of faster, broader, and more efficient defect detection.

This book is a guide that can help organizations implement successful automated software testing programs and efforts. The book does not provide gimmicks or magical solutions, as none exist, but it provides experience-based discussions and recommendations. It includes a thorough dissection of automation issues, such as in Part I of the book, where we describe what automated software testing is and is not; why a business case is required for successful automation, including step-by-step instructions for developing one; why to automate and when. Then we summarize why automation often fails and the pitfalls and blunders that can be prevented; we describe the tools that are available to help implement successful automation efforts, with a focus on open-source testing tools. In Part II of the book we present six keys to successfully implementing automated software testing. These are

• Key 1: Know Your Requirements
• Key 2: Develop the Automated Test Strategy
• Key 3: Test the Automated Software Test Framework (ASTF)
• Key 4: Continuously Track Progress—and Adjust Accordingly
• Key 5: Implement AST Processes
• Key 6: Put the Right People on the Project—Know the Skill Sets Required

IDT conducted two separate surveys related to automated software testing with approximately 700 total responses from test professionals all over the world, across organizations that were diverse in size and in what they do. The survey showed two very consistent themes:

• About 70% of survey respondents said they believe automation is high-payoff, but they are generally not sure why to automate and how automation applies to their project.
• Half of the survey respondents also said they felt they lacked the experience, time, or budgets to implement automation.

Most seem to agree: Automated software testing is useful, and an increasing need for it exists. However, the lack of experience seems to be the reason why automation is not implemented more often with a higher success rate. Finding people with the skills for the project is therefore important; a summary of skills required is provided in Chapter 10. For more details on the outcome of this survey, see Chapter 4.

Material Coverage and Book Organization

Part I: What Is Automated Software Testing and Why Should We Automate?

Chapter 1, What Is Effective Automated Software Testing (AST)?, describes what automated software testing is. The definition of automated software testing we use throughout this book is the “application and implementation of software technology throughout the entire software testing lifecycle (STL) with the goal to improve STL efficiencies and effectiveness.”

In Chapter 2, Why Automate?, we address this question that is asked so often. Here we discuss the challenges of software testing today and how the time and cost of software testing can be reduced. Reasons for why to automate, laying the foundation to help build the business case discussed step by step in Chapter 3, are presented here.

In Chapter 3, The Business Case, we define a step-by-step approach to defining the business case, which will cover the business need, the reasons for an automated software testing project, the business benefits (tangible and intangible), an analysis of the expected costs and timescales, an investment appraisal, and return on investment (ROI).

Chapter 4, Why Automated Software Testing Fails and Pitfalls to Avoid, clarifies some of the myths and realities surrounding automated software testing. The goal is for companies and organizations to review the lessons described here and not to repeat them during their automated software testing implementations.

Part II: How to Automate: Top Six Keys for Automation Payoff

Once management has been convinced by the business case that was laid out in Part I of this book and understands the pitfalls to avoid and the realities of automated testing, the next step is to determine how to automate. Part II of the book addresses how to successfully implement the various automated software testing tasks. We have determined that successful automated software testing can be achieved by implementing six top keys, described next.

Chapter 5, Key 1: Know Your Requirements, covers the importance of understanding the requirements before developing an automated testing strategy. Here we discuss approaches to determining the problem we are trying to solve along with how to gather information when requirements are not available.

Chapter 6, Key 2: Develop the Automated Test Strategy, discusses developing an automated testing approach in detailed steps, including test environment considerations, configuration management for automated test scripts, and related artifacts, among others. Here we also discuss what to consider when deciding what to automate and the importance of choosing the right tool, whether open-source, vendor-provided, or in-house-developed.

Chapter 7, Key 3: Test the Automated Software Test Framework (ASTF), covers the importance of understanding testing techniques and documenting test cases as part of automated testing. Automators often forget that documentation is still a vital part of the automated test program. The test case documentation serves as the blueprint for the automated software testing efforts. This chapter describes the importance of tracing test cases back to requirements; the content of the test cases, such as needing to include inputs and expected results; and how documented test cases become the basis for developing and implementing the automated tests.

Chapter 8, Key 4: Continuously Track Progress—and Adjust Accordingly, addresses the importance of tracking the goal that was set at the outset of the automation program. For example, during the discussion of business case development in Chapter 3 we explain the need for defining goals; in this chapter we discuss how peer reviews, inspections, and various automation and testing metrics can help measure and track progress against those goals.

Chapter 9, Key 5: Implement AST Processes, points out the need for a lightweight process. Some automated testing scripts can be implemented successfully without much process in place, but in order to effectively implement a large automated testing program a lightweight adaptable process should be in place. This chapter discusses a summary of this process, linking back to the details in various chapters.

Chapter 10, Key 6: Put the Right People on the Project—Know the Skill Sets Required, clarifies the skill sets needed for developing automated software testing, for instance, a skill set similar to that of the software development team, which includes requirements analysis, design, software development, and testing. Key 6 points out that although knowledge of testing techniques and analytical skills is important, effective automated software testing implementation requires software development skills. The skills described here parallel the automated testing process described in Chapter 9.

Audience

The target audience of this book is software test professionals such as test managers, leads, and practitioners. It is also geared toward all quality assurance professionals, QA leads, and practitioners. Project managers and software developers looking to improve the effectiveness and quality of their software delivery will also benefit from this book.

---

1. 1 www.idtus.com.

---

© Copyright Pearson Education. All rights reserved.

Product Details

• Paperback: 368 pages
• Publisher: Addison-Wesley Professional; 1 edition (March 14, 2009)
• Language: English
• ISBN-10: 0321580516
• ISBN-13: 978-0321580511
• Product Dimensions: 7.4 x 0.9 x 9.2 inches
• Shipping Weight: 1.6 pounds (View shipping rates and policies)
• Average Customer Review: 4.0 out of 5 stars  See all reviews (11 customer reviews)
• Amazon Best Sellers Rank: #166,958 in Books (See Top 100 in Books)
  • #68 in Books > Computers & Technology > Programming > Software Design, Testing & Engineering > Testing

Most Helpful Customer Reviews

19 of 19 people found the following review helpful

5.0 out of 5 stars Pragmatic and practical help for test automation May 12, 2009

By Lisa Crispin

Format:Paperback

Although this book is not oriented towards agile software development, it's still a solid resource for anyone new to test automation. It's pragmatic, practical, clearly written, easy to understand. I especially like the six "Keys" for automation payoff. The authors explain the reasons for automating - it might seem obvious to some but many newbies don't see all the potential benefits. The book also blows through automation myths. There's a lot of emphasis on ROI, which is often overlooked.

Where the advice I give on automation differs from this book is making it a whole team effort, rather than the test team only, but that's easier to do in an agile setting. Also, the authors do talk about things like interviewing stakeholders, and getting people with the right skills, these are all so important.

I wish the book had a section on continuous integration and automated build process. I think in another few years nobody will question the need for this, any more than people currently question the need for automated source code management. Whereas a few years ago nobody in my conference tutorials was doing CI, nowadays about a third of the people are. I think it's so critical to have a way to continually run all the automated regression tests every time new code is checked in. The book makes a passing reference to this, and it does mention test automation at different levels starting at the unit level, but it doesn't explain why you need a build process and how to set one up.

Nevertheless, it's a great resource, and will give readers a good grip on the fundamentals of test automation. I get so frustrated when people think it's impossible to automate, or that they have to hire some expensive consultant to get it done. This book will enable teams to be much more successful. It is a good overview of all the different areas where automation can help a team tremendously.

Just be sure to also buy a book that tells you how to set up continuous integration and automated builds, such as _Pragmatic Project Automation_ by Mike Clark, or _Continuous Integration_ by Paul Duvall, Andy Glover and Steve Matyas. Or _Ship It_ by Jared Richardson and William Gwaltney.

20 of 22 people found the following review helpful

3.0 out of 5 stars Strong on theory and planning, weak on practical implementation May 12, 2009

By James Holmes

Format:Paperback

This book isn't for everyone, but everyone can get some value out of it. What I mean by that rather confusing statement is that folks working in Agile environments will likely want to throw the book across the room while folks in more bureaucratic environments like CMMI or other waterfall environments will likely get a great deal of value from the book.

I'm an Agile fanatic and I had a difficult time dealing with book's approach which emphasizes spending large amounts of time creating documentation such as requirements traceability matrixes, detailed test plans, etc. My preferred approach is to have testers working side-by-side as part of a team, creating specifications from user stories/requirements and moving those right in to automated test suites via tools like Selenium, Cucumber, or RSpec.

That said, I did indeed get some good value from the book. I found the discussions on making hard evaluations on what to test very worthwhile reading: teams can easily vaporize large amounts of time creating large suites of brittle, unmaintainable automated tests. This book has several really good chapters on using business cases to drive return on investment (ROI) decisions for testing, understanding automated test pitfalls, and adjusting your testing as you progress through your project.

Additionally, one of the book's high points was on building the test team: "Put the Right People on the Project - Know the Skill Sets Required." This is a great chapter which emphasizes starting the search by focusing on how to interview test team members - and how those testers' skills are greatly different than other members of the team.

The book's very academic, dry tone makes for some difficult reading, and few concrete examples are used until very late in the book. Having spent a large number of years either in the DOD or working for DOD contractors, it quickly became apparent that much of the book seemed targeted to folks working in those environments - too many dry acronyms are scattered through the book, adding to the difficulty in reading.

The lack of examples using real tools frustrated me. While the appendices contain some examples of comparing various tools, the book doesn't actually show how a real world testing environment would use those tools. One appendix, eight or nine pages in length, is touted as a "Case Study" but falls short, in my opinion.

Overall it's a decent book. The dry tone and lack of real environments is balanced out by the excellent coverage of team skills and emphasis on selecting how and what you test.

6 of 6 people found the following review helpful

4.0 out of 5 stars first, automate the unit tests March 28, 2009

By W Boudville HALL OF FAMETOP 500 REVIEWERVINE™ VOICE

Format:Paperback

The authors give you a top level description of why automated software testing is highly desirable, along with detailed guidelines for doing so. The tone is very realistic, making you aware of many issues associated with the topic.

For one thing, you are cautioned to avoid the blandishments of a vendor who might suggest that her product will meet all your testing needs. In the authors' experience, there is no single tool that covers all major operating systems. The book also advises you to look at open source freeware. There is a surprising amount of good stuff freely available, that you might want to check first before considering proprietary products.

The book mentions many reasons for automation. These include manual tester fatigue. But also that some things are very difficult to test in a manual fashion. Often this could be because manual testing is at the GUI level. There could be bugs deep in the code, maybe in computational blocks.

Which also leads to the point that the "testers" for making automated tests often have a different skill set from manual testers. The latter might not be programmers. The former should be, with access to the source code [white box or grey box testing]. Because this gives them knowledge about what automated tests to write, that test critical aspects.

Naively, given the book's nature, we might expect it to say automate everything in sight. But the book's credibility is enhanced by it explaining that this is simply not economically feasible. The estimate is 40-60% of tests to be automated. Table 6.2 in the book is a list of questions that can be applied to each test, to suggest whether a test is suitable for automation. Roughly, tests that will be run often are a high candidate for automation.

The book also strongly recommends extensive unit testing. This is the lowest level of testing and bugs caught here have the best payoff in terms of minimising the cost to fix. A tight software development loop; "agile" as opposed to "waterfall"-like, though the book doesn't use these terms. Plus often unit testing might not be doable at the GUI level anyway, if the units are computational routines. So punting by not having automated unit tests and expecting manual tests to later find bugs in these units is very bad. Of coure, the book also describes higher level tests like regression and functional tests. But first do the unit tests.