Overcoming Barriers and Valuing Evaluation: An ASTD-I4cp Study Underscores the Progress Organizations Are Making in the Area of Learning Evaluation, as Well as the Gaps That Need Closure
Patel, Laleh, Talent Development
Findings from the American Society for Training & Development (ASTD) and the Institute for Corporate Productivity's (i4cp) study, The Value of Evaluation, highlight that progress is being made in measurement practices, as the pursuit of excellent learning evaluation continues. However, there is still room for development because few organizations think that they've mastered learning evaluation, and many admit to facing some ongoing challenges.
ASTD and i4cp partnered to explore the complex issue of learning evaluation and the value of learning. A total of 704 human resource and learning professionals completed an online survey during May 2009. The majority of them (85 percent) were managers, directors, vice presidents, or C-level officers, and the sample was evenly distributed in terms of workforce size, annual revenue, and industry.
Kirkpatrick/Phillips evaluation model
The findings revealed that the five-level Kirkpatrick/Phillips model of learning evaluation is the most common practice. The five levels include participant reaction (Level 1), level of learning achieved (Level 2), changes in learner behavior (Level 3), business results derived from training (Level 4), and return-on-investment from training (Level 5). While this finding was not shocking, there were some interesting results to additional questions, such as how frequently these levels of evaluation are used and how valuable they are to organizations.
The survey data answered these questions, showing that the majority of respondents (92 percent) measure their learning programs to at least Level 1 of the model. This isn't surprising given that Level 1 is the easiest metric to track, usually with "smile sheets." The use of other types of evaluation drops off as we move up each subsequent level of the Kirkpatrick/Phillips model, which is likely due to the increasing difficulty in obtaining the information at each level.
A little more than four out of five respondents said that their organizations evaluate at Level 2, where companies explore what knowledge was gained. Level 3 registered another drop in usage, with more than half of the respondents indicating that their organization evaluates learners' behavior. Level 4 measures an important area--results, which only 37 percent evaluate to any degree. Level 5 received the least consideration, with only 18 percent of respondent companies measuring the impact on ROI. The survey results indicate that translating the value of a training program into dollars and cents can be a difficult process.
The most used levels are not the most valuable
One of the interesting insights from the survey data was that the extent to which training evaluation occurs at the different levels of the Kirkpatrick/Phillips model does not tell us much about its perceived value. Although Level 1 is the most used form of evaluation, it was found to be the least valuable for organizations. Only 36 percent of respondents said that Level 1 had a high or very high value. By comparison, 55 percent said that Level 2 had high or very high value, and three-quarters said the same about Level 3 and Level 4.
At first, these findings may strike some as perplexing, but there is almost always more value in knowledge gained, behaviors changed, and results achieved than in participants' reactions. It therefore prompts the question as to why companies are not evaluating more programs at higher levels. Of the few companies that measured Level 5, nearly 60 percent perceived it to have a high or very high value.
The survey probed into the perceived barriers that prevent companies from using all of the evaluation levels. The barrier that looms largest is the difficulty in isolating learning as a factor that has an impact on results (52 percent endorsed it to a high or very high extent), which comes into play mostly for Level 4 and Level 5. …