Learning: CEOs Rate Learning Metrics

Learning: Submitted by Leslie Allan on January 11th, 2011

In this post, I continue my review of Jack and Patricia Phillips’ survey of CEO attitudes and practices around their organization’s learning investments. Here, I focus on their views on the learning metrics presented to them by their learning managers. The survey draws some interesting hints for where training managers need to focus their efforts in future if they are to enhance their value to the organization.

There has been much discussion and debate within the training industry in the last decade about how and when to evaluate the effectiveness of learning programs. Some learning professionals primarily rely on transactional measures, such as number of employees trained and cost per hour of learning. Others use output measures based on, for example, Kirkpatrick’s Four Level model or Jack Phillips’ ROI model.

Phillips’ survey asked CEOs how much value they placed on certain learning metrics and whether they were used in their company. Here is how they rated the utility of eight different measures, in order from (1) most important to (8) least important.

When CEOs were asked whether they currently use a particular measure and whether they should do so in future, the results are illuminating. Here is what they said about each measure in the list:

Note how the two top rated measures in terms of importance (Impact and ROI) are reported by CEOs as having the lowest rate of usage. This no doubt is due to the difficulty and cost of collecting accurate data and the uncertainty around how much of the results can be attributed to the training program. Some consultants claim that many CEOs are wanting to see ROI figures for their training programs. The survey confirms this view, but to a limited extent. After some sixty years of the Kirkpatrick model and almost as long for Phillips’ ROI model, the uptake in organizations for Impact and ROI metrics is less than 10%. The want it seems is more like a wish than a demand.

Notice also how the use of Inputs metrics is encouraged by the survey respondents, although it is given a low importance rating. I suspect that this is because the relevant data is readily available and can be collected without much cost or effort.

Training participant Reaction data is not given the same latitude, with only one in five CEOs wanting to see its collection. This is in spite of it being the most widely collected form of data for those using Kirkpatrick’s Four Level model.

What can learning managers take away from this survey? I think it can give us an idea of where we need to concentrate our efforts in improving the visibility and importance of training in the enterprise. I have combined the difference between what is currently measured and what should be measured with the importance rating to give a score on the current effort shortfall. (Write to me if you would like the formulae I used.) The table below shows the results in ascending order of effort currently expended relative to CEO expectations.

The larger the negative value, the higher the effort required to meet the needs of the organization. Overall, organizations should spend more time and effort in developing and implementing metrics with large negative values. The larger the positive value, the larger the amount of excess effort currently put into this metric. Organizations should spend less time and energy collecting these data. A value of zero indicates the proper balance between effort and value to the organization.

Do not place too much significance in the absolute values in the above table. The survey anchor points are not calibrated and so you should take notice of the relative values only. Surveying the results, we can easily see that organizations need to put more effort into Impact and ROI studies and resources need to be withdrawn from Reaction data collection and analysis.

Of course, my analysis here relates to organizations in general and gives a picture of where the training industry as a whole needs to devote its efforts. For your own organization, I advise you to meet with your own CEO to discuss what is important to them. Initiating this dialogue and acting on what your CEO says is an important first step in getting a seat at the executive table and increasing the perceived value of training in your organization.

As the survey results demonstrate, learning managers still have a way to go in raising their profile. On average, the survey shows, Chief Learning Officers have two management levels between them and their CEO. In addition, on a four-point scale from very dissatisfied to very satisfied, CEOs rate their learning and development department less than “satisfied”. There is every reason then for learning managers to act on the lessons learned from the Phillips survey.

To Discuss how these Solutions will add value for you, your organization and/or your clients, Affinity/Resale Opportunities, and/or Collaborative Efforts, Please Contact:

Tom McDonald, tsm@centurytel.net; 608-788-5144; Skype: tsmw5752