Newsletter: Issue 3, 5/13/11
CEOs Rate Learning Metrics
Submitted by Leslie Allan on January 11th, 2011
In this post, I continue my review of Jack and Patricia Phillips’ survey of CEO attitudes and practices around their organization’s learning investments. Here, I focus on their views on the learning metrics presented to them by their learning managers. The survey draws some interesting hints for where training managers need to focus their efforts in future if they are to enhance their value to the organization.
There has been much discussion and debate within the training industry in the last decade about how and when to evaluate the effectiveness of learning programs. Some learning professionals primarily rely on transactional measures, such as number of employees trained and cost per hour of learning. Others use output measures based on, for example, Kirkpatrick’s Four Level model or Jack Phillips’ ROI model.
Phillips’ survey asked CEOs how much value they placed on certain learning metrics and whether they were used in their company. Here is how they rated the utility of eight different measures, in order from (1) most important to (8) least important.
|1||Impact||increased sales volume; decreased defect rate|
|2||ROI||percent return on training expenditure|
|3||Awards||received from recognized industry body or government|
|4||Application||number of skills used three months later|
|5||Learning||percent participants passed; average test score|
|6||Inputs||number of people trained; total training expenditure|
|7||Efficiency||cost per hour; cost per skill; training hours per skill|
|8||Reaction||participant satisfaction level; number of positive testimonials|
When CEOs were asked whether they currently use a particular measure and whether they should do so in future, the results are illuminating. Here is what they said about each measure in the list:
|Use Now||Should Use||Metric|
Note how the two top rated measures in terms of importance (Impact and ROI) are reported by CEOs as having the lowest rate of usage. This no doubt is due to the difficulty and cost of collecting accurate data and the uncertainty around how much of the results can be attributed to the training program. Some consultants claim that many CEOs are wanting to see ROI figures for their training programs. The survey confirms this view, but to a limited extent. After some sixty years of the Kirkpatrick model and almost as long for Phillips’ ROI model, the uptake in organizations for Impact and ROI metrics is less than 10%. The want it seems is more like a wish than a demand.
Notice also how the use of Inputs metrics is encouraged by the survey respondents, although it is given a low importance rating. I suspect that this is because the relevant data is readily available and can be collected without much cost or effort.
Training participant Reaction data is not given the same latitude, with only one in five CEOs wanting to see its collection. This is in spite of it being the most widely collected form of data for those using Kirkpatrick’s Four Level model.
What can learning managers take away from this survey? I think it can give us an idea of where we need to concentrate our efforts in improving the visibility and importance of training in the enterprise. I have combined the difference between what is currently measured and what should be measured with the importance rating to give a score on the current effort shortfall. (Write to me if you would like the formulae I used.) The table below shows the results in ascending order of effort currently expended relative to CEO expectations.
The larger the negative value, the higher the effort required to meet the needs of the organization. Overall, organizations should spend more time and effort in developing and implementing metrics with large negative values. The larger the positive value, the larger the amount of excess effort currently put into this metric. Organizations should spend less time and energy collecting these data. A value of zero indicates the proper balance between effort and value to the organization.
Do not place too much significance in the absolute values in the above table. The survey anchor points are not calibrated and so you should take notice of the relative values only. Surveying the results, we can easily see that organizations need to put more effort into Impact and ROI studies and resources need to be withdrawn from Reaction data collection and analysis.
Of course, my analysis here relates to organizations in general and gives a picture of where the training industry as a whole needs to devote its efforts. For your own organization, I advise you to meet with your own CEO to discuss what is important to them. Initiating this dialogue and acting on what your CEO says is an important first step in getting a seat at the executive table and increasing the perceived value of training in your organization.
As the survey results demonstrate, learning managers still have a way to go in raising their profile. On average, the survey shows, Chief Learning Officers have two management levels between them and their CEO. In addition, on a four-point scale from very dissatisfied to very satisfied, CEOs rate their learning and development department less than “satisfied”. There is every reason then for learning managers to act on the lessons learned from the Phillips survey.
How Executives View Learning Metrics
Appendix C – Analysis of Survey Results
Is your CEO asking you to measure quantitatively the effectiveness of your training programs and demonstrate the results in financial and non-financial terms? Check out our comprehensive Training Evaluation Toolkit. Whether you are a novice or experienced professional, this guide and toolkit will walk you through planning your evaluation exercise, collecting all relevant data, isolating non-training factors and then analyzing and reporting the results convincingly to your key stakeholders. The toolkit is packaged with a full set of reusable and customizable Microsoft Word forms and Excel calculation worksheets for all of your measurement and reporting needs. Find out more about our Training Evaluation Toolkit and download the free introductory chapter today.