Customer Reviews

Evaluating Training Programs: The Four Levels

5 star:		(7)
4 star:		(2)
3 star:		(0)
2 star:		(1)
1 star:		(1)

 

 

If you expect to interact with those in the training industry, you had best be armed with knowledge of Kirkpatrick's four-level model. For instance, if someone asks, "What are you doing about Level IV?" you need to come back with a robust response such as, "Well, we evaluate the impact of training by....". Kirkpatrick first presented his model in 1959 in a series of articles in the Training and Development Journal. In this book, the Godfather of training evaluation succinctly outlines his four-level model of training measurement: measuring reaction, measuring learning, measuring behavior, and measuring results. This book is an up-to-date enhancement of his model. It consists of corporate examples, which are always helpful, and survey templates. If you are looking for a model to increase training effectiveness, reduce cycle time of design and development of courses, or establish baselines, this is a good primer. I praise him for consistently encouraging formative evaluation. Less seasoned training professionals only conduct summative evaluation (Level 1-smiley sheets), but evaluation should not be seen as an event as much as a life-cycle process. The ISD process includes a formative evaluation approach, which is often over looked. If you read this book and measure all four levels, you will be on the right track to evaluation utopia.

If you are a serious evaluator or interested in calculating return on investment, a "must read" is Jack Phillips Return on Investment in Training and Performance Improvement Programs. Phillips expands Kirkpatrick's models to a fifth level, which is return on investment (ROI), and encourages ROI at Kirkpatrick's four levels, too. If you are interested in implementing Kirkpatrick's four-level model and/or ROI, you need to read Another Look at Evaluation Training Programs. This book is a series of "best practice" articles compiled by Kirkpatrick.

One final note: Keep in mind that Kirkpatrick's model is not the only one out there. For instance, there is: Kaufman's Five Levels of Evaluation, The CIRO (context, input, reaction, and outcome) Approach, Stufflebeams's CIPP (context, input, process, and product) Model, Alkins' UCLA Model, and The Phillips Five-Level ROI Framework. Choose a best-fit model that is based on your known organizational barriers and the decisions important to your stakeholders.

This book should be in every instructional designer's or training manager's library. Kirkpatrick's four-level model of training evaluation is frequently referenced in the training literature and is the standard for evaluating the impact of training. It is part of the common language used in the training industry. This book is also an excellent primer for some of the basic issues involved in evaluating the effectiveness of training programs. It contains numerous case studies from actual training programs. Almost two-thirds of the book teaches through case studies.

The book also contains examples of data collection instruments and other evaluation tools that can be adapted for a trainer's own needs. I wish the first part of the book had been expanded to provide more explanation of the four levels before launching into the case studies. Kirkpatrick has edited another book, Another Look at Evaluating Training Programs, which contains articles by other evaluators who discuss many issues related to the Kirkpatrick model and provide further case studies.

There has been much interest in return-on-investment (ROI) in the training industry. Kirkpatrick cautions against confusing ROI with results; they are not the same. Financial return is just one of many possible critical success factors. The focus on ROI can even mask true business results that encompass more strategic factors and involve a larger time frame than is typically analyzed in ROI analysis. Nevertheless, the reader may want to look at Return on Investment in Training and Performance Improvement Programs by Jack Phillips.

An indispensable work for anyone concerned with getting the most value from their training programs. What you'll find in this book:

* A concise but solid explanation of the four levels and the purpose of each level.

* Guidelines for measuring at each level.

* More than a dozen case studies that demonstrate how a variety of training organizations have implemented one or more of Kirkpatrick's levels, including sample forms and data analysis.

Kirkpatrick doesn't offer an "evaluation formula" with instructions to plug in an organization's name or training titles to a prefabricated shell. Instead, he presents a skeletal structure which an organization can flesh out to create an evaluation system.

Okay so this is an old and perhaps dated book, but it is still the definitive work on linking the training and development process to real knowledge, work performance, and business results.

Before Kirkpatrick (sometimes even today) Trainers and Training Managers would judge the effectiveness of a learning treatment by the opinion of the learner ("Level One" Measurement)...you know, those silly little evaluation sheets where we are always asked such banal questions as: "Did you LIKE the course," or the obligatory "Do you FEEL smarter," and so forth. Most in the educational field also understood the need of testing the learner in order to validate knowledge (This is "Level Two").

But in this book old Don K. gives us the Third and Fourth levels of learning. Level Three: Basically, if you cannot empirically demonstrate that the learning experience translates well into behaviors on-the-job, AND provide perceived value for the company (a.k.a. "Level Four"), then it really does not matter how much you actually learned, now does it spanky!?

As many might know, since the time of Kirkpatrick, ROI (Return on Investment) is now "the Fifth Level" of learning measurement. This ratio is defined as the Benefits of your learning program in dollars, Divided by the Total Cost of your program, Multiplied by 100, which gives you an ROI percentage. Anything over 100% suggests a positive return. Pffft! When will these so-called learning professionals learn to acount for the Net Present Value of cash?? I mean, even if you give a company a positive return because of your glorious learning management system, they still had to shell out massive capital upfront, right? So "Positive ROI" is not so easy as putting a dollar average on turnover percentages and then creating a ratio...

Anyway, please know that you simply have to understand the roots of Training ROI via Kirkpatrick before you can count yourself amongst the "Training Cognoscenti." : ) Enjoy!

Offers a how-to approach to evaluating training programs assessing student satisfaction, mastery of content, application of learned skills, and impact on the client's business. Includes detailed techniques and formulas for measuring success with specific illustrations. Useful and informative.

This is an excellent book for anyone charged with evaluating training. After a relatively brief and very readable introduction to the four levels of training evaluation, the detailed case studies will provide any training evaluator with ideas they can use in their own programmes. The book is worth it for the sample questionnaire forms alone.

I am an education designer, and am currently creating a program for 'evaluation of education'. This text is the primary reference for this design project, and has provided logical, systematic and concise descriptions of the different levels of evaluation. This is complimented by clearly stated rationale and relevant examples.

Where would a learning event be without smiley sheets at the end? Well, learning events be in a much better place if they follow the levels outlined by Donald Kirkpatrick decades ago. Kirkpatrick's Four Levels helps training teams and organizations understand that there are connections between good training and the success of a company.

Kirkpatrick outlines his four levels into satisfaction, knowledge, application, return on investment. By measuring each level, an organization better understands what to reinforce and use each level as leading indicators of success rather than just a piece of paper that gets scribbled on at the end of a class or learning event.

As I have reread this numerous times, I wish there were more about the transition from knowledge to application. It took many years to learn how to align the measurement and behavior to anticipated application...but, then again, that would have taken away from my own learning experience. Read the chapters once all the way through, then use the book as a reference guide as you create each level yourself. It provides good reminders of why you are measuring and not just what to measure.

Obviously careful thinking and a lot of reasonable stuff, but the whole thing taken overall reads a little like the kind of thing you'd expect Dilbert's manager to believe. Smarter than that, probably, but the point remains...

The evaluation information and logic in this book is inadequate. For example, the author notes that "being able to show results" is why we train (p. 65) and then offers up a chapter on "Evaluating Results" that is 8 pages long! One is a page of questions, another is a page of white space, and the few words that are presented are inadequate, at best.