Ever since Donald Kirkpatrick introduced the four levels of evaluation, academics and training professionals alike have scurried to research articles, read books, and attend seminars on evaluation and the return-on-investment of training. Despite ROI's popularity, the "ASTD 2001 State of the Industry" report notes that training and development professionals spend merely 6 percent of their time on evaluation. I'd hazard a guess that the figure is lower in Europe.
In the Netherlands, for example, university students do most of the research on the effectiveness and impact of training, and sometimes their efforts result in training policy changes. The real test comes after the students conclude their projects:
* Does the t&d department continue the students' efforts by measuring the results of training regularly?
* Does the department attempt to show ROT?
* Does the organization implement evaluation policies and procedures?
Typically, the answer to those questions is no. PricewarerhouseCoopers in the Netherlands is trying to change that. For more than three years, PwC has conducted evaluations using the methodologies of Kirkpatrick and Jack J. Phillips.
But here's the rub: For projects in the Netherlands and Germany, PwC has found that cultural and institutional differences affect the feasibility of measuring ROI.
Paul Boselie, Jaap Paauwe, and Paul Jansen, in their International Journal of Human Resource Management article, "Human Resource Management and Performance: Lessons From the Netherlands" (November 2001), provide support for our experience. They state that a focus on stakeholders--instead of shareholders--encourages a different system for measuring HRM results. A focus on shareholder value is a driver for measuring ROT; however, Dutch companies must also pay attention to such stakeholders as customers, employees, and trade unions.
Knowledge of stakeholder importance in training decisions has helped PwC tailor evaluation methods. That's a complex task, especially when you take into consideration that in the Netherlands, and to a lesser extent in Germany, the Rhineland economic model has been predominant for the past two decades. The Rhineland model depicts a state governed by close coordination of political partners, trade unions, and industry associations. That means that the relationship between business strategies, HR policies, and performance is moderated significantly by institutions and stakeholders, both inside and outside of the organization. Such collective bargaining agreements and labor laws prescribe, prohibit, and influence HRM practices and policies.
To further complicate matters, France, Germany, and the Netherlands have tax systems that reward companies when they invest in training. Those countries offer a tax refund based either on a percentage of payroll or on the number of days each employee spends in training--prescribed training that's based on a core curriculum rather than on individual performance needs. That reward leads companies to track training input instead of output.
Performance data is another factor that affects the feasibility of measuring ROI. Whether you work in the Netherlands or the United States, to measure the impact of training, you need data that supports change in knowledge, skills, performance, or business. While consulting in the Netherlands and Germany, PwC found that individual performance data is often missing.
But that shouldn't come as a surprise; you'll find no "Employee of the Month" in those countries because performance doesn't play a large role in their salary and reward systems. …