Performance Metrics: How Executives View Learning Metrics

Performance Metrics: Jack J Phillips and Patricia P Phillips –  12/3/10

CEOs care about learning investments, but there are several areas chief learning officers can leverage to secure more executive-level interest and involvement.

---

The ROI Institute recently conducted a survey to obtain direct feedback from CEOs in large organizations. There are few if any significant data sets collected from this elusive group, though many one-on-one interviews have appeared as profiles in various media. These interviews rarely discuss specific results, however. A few surveys have attempted to target the executive level, but unfortunately, the survey is usually passed down to the CLO.

To obtain the executives’ views on learning and development, the institute sent a survey and a letter with instructions asking the CEO not to forward on the survey to the learning leader. The target audience was Fortune 500 CEOs and the top 50 privately held firms. Some 450 firms were included in the sample.

Special survey response-enhancing techniques were employed to achieve the response rate, including personal notes, assurances that survey responses would remain anonymous unless CEOs elected to provide contact data, and perhaps most importantly, the survey leaders knew someone employed at approximately 20 percent of the companies listed. In many cases, a middle manager was asked to deliver the study to the CEO directly.

Ninety-six individuals responded, representing 21.3 percent of the total survey pool. Executives chose to remain anonymous, and some did not answer particular points or provide comments. This response rate is especially significant when considering the difficult economic circumstances during which the survey was conducted. Taking a few minutes for a survey addressing learning and development is not at the top of the agenda for most executives during this time period.

Where Does the Money Go?
For one question regarding the rationale for setting learning investment levels, CEOs selected a strategy from a list. Investments in these companies range from $10 million to $640 million with an average of $138 million. Only 4 percent acknowledged that they try to avoid these investments, but based on what we have observed in our work, we suspect this number actually may be a little higher. Twenty percent state they only invest in the minimum. This may be due in part to the current economic times, where executives have had to trim activities perceived to be unnecessary. As expected, using benchmarking data was rated the highest with 39 percent. Because we asked that only one strategy be checked, we suspect many CEOs use a combination of benchmarking and other possibilities. We limited the choice to only one in the survey to determine the dominant approach for setting learning investment levels.

A significant number of CEOs, 10 percent, said that they invested in all learning and development needs. Although we worded this option so they would feel comfortable with this choice, this is probably overinvesting. In our own experience, some executives are proud they can invest in practically any learning request.

Finally, some 18 percent invest when they see value. We assume, however, that there are many different definitions of the term “value,” and it is not necessarily safe to conclude that this percentage of executives requires ROI calculations.

What’s the Relationship Between CEO and CLO?
It was important in this study to determine how close the CLO or head of learning and development is to the CEO. To that end, a 1 indicates the CLO reports directly to the CEO. A 2 means there are two levels between them, and a 3 means there are three levels between them. The average was 3.2, which means the CEO is at least three levels up. This distance is a little disturbing considering learning leaders as a group continue to expend considerable effort pushing this function to a higher-level position within their companies.

We asked the executives a general question regarding their level of satisfaction with their learning and development departments and then created a four-point scale to force them to take a stand. On the four-point scale, a 1 indicates very dissatisfied and a 4 indicates very satisfied. We were hoping to see a 3 or better. The results were 2.52, indicating some dissatisfaction.

It is often easier to gather data by having executives respond to particular checklists. We used this approach to determine which metrics currently in use are critical in Figure 1. We provided eight categories and mapped them into the levels of evaluation. The first two categories were “inputs” and “efficiencies,” which are process measures or inputs to the process, including volume, costs and speed — Level 0. The next two categories, “reaction” and “learning,” are typical measures — levels 1 and 2. “Application” is the extent of the use of knowledge and skills — Level 3. “Impact” — Level 4 — which is the business measure, created much interest, as did ROI. We included ROI — Level 5 — because of the abundance of information available about its use.

Finally, we included “awards,” which many CLOs are currently pursuing and reporting to senior executives, particularly in large organizations. This is an impact measure, one not converted to money.

The survey requested information to answer the following three questions:

1. What metrics are being reported to you now?

2. What should be reported that is not being reported now?

3. How would you rank these in terms of value?

On the ranking, an 8 score would indicate the bottom of the list, and a 1 score would indicate the top of the list. Inputs and efficiencies were ranked 6 and 7, respectively; input shows scope and volume, something executives need to know. This type of data is often reported. Reaction was ranked lowest, which may not be a surprise, though it is a No. 1 outcome measure reported to executives. This particular measure likely could be improved with more focus on strategic content development.

The awards category was rated higher than expected, but the highest two areas were impact and ROI, which should not be surprising because CEOs always want to see this kind of data, especially during these tough economic times. This reporting creates an important dilemma for learning and development executives. These are the least-reported outcome data sets, but they are of the most value to executives. Therein lies the challenge and the opportunity for the learning and development team.

CEO Involvement and Metrics
We also asked about a learning and development scorecard, and we discovered that only 22 percent of the CEOs said they had one. This is surprising given the effort learning and other executives spend to develop balanced scorecards in large organizations. Of course, a scorecard could be in place in some cases, but the information it contains may not make its way to the CEO.

On the positive side, this particular result indicates some executives are reviewing scorecards on a routine basis. For the most part, written comments were either negative or constructive. Only one person indicated that he or she was pleased with the scorecard currently in use in the organization. The other comments referred to scorecards as “inadequate,” “incomplete,” “doesn’t have all the data” and “doesn’t really connect to the business.” This leaves some great opportunities for chief learning officers to make improvement in this area.

A critical issue for learning and development departments is the extent of executive involvement. Most would argue that if executives are taking a more active role or are more involved with learning investments, more results will be achieved. As expected, survey results revealed the top area of involvement the CEO personally approves is the learning and development budget, which was indicated by 78 percent. Second on the list, 73 percent of CEOs said they review requests for major programs, while 61 percent review the results of those programs. A total of 24 percent use a learning scorecard to monitor progress and make adjustments.

Next, 29 percent open and close major programs, while 22 percent host or conduct periodic review meetings. Only 18 percent actually teach segments of major programs. Disappointingly, the two lowest survey results — holding periodic review meetings and being involved in teaching segments — can have the most impact on learning and development initiative success. Periodic review meetings represent an opportunity to evaluate progress and results and make adjustments. This is a great way to stay connected with the CEO, solicit feedback and demonstrate results, which can help to boost funding in the future.
Having CEOs and other senior executives get personally involved in teaching is also a powerful way to connect learning and development to the organization and deliver value.

Jack J. Phillips is chairman of ROI Institute Inc., and Patricia P. Phillips is president and CEO of ROI Institute Inc. They can be reached at editor@clomedia.com.

http://clomedia.com/articles/view/how-executives-view-learning-metrics/1%20

==================================================================

To Discuss how these Solutions will add value for you, your organization and/or your clients, Affinity/Resale Opportunities, and/or Collaborative Efforts, Please Contact:

Tom McDonald, tsm@centurytel.net; 608-788-5144; Skype: tsmw5752