Learning Analytics: The Buzz Surrounding ‘Learning Analytics’

Learning Analytics: 9/1; Republished from T+D Magazine September, 2003 (online issue), Via Knowledge Advisors

Learning Analytics: By Jeffrey Berk

If you’ve been following the learning industry recently you’ve probably heard the term ‘learning analytics’ thrown around. Learning Analytics is the new buzzword for the process by which learning professionals analyze critical indicators within their business to not only continuously improve but to demonstrate value to stakeholders and make better decisions to optimize learning investments. It is not just about cool technology. Learning professionals should be taking a very close look under the hood of the ‘analytics’ tools to ensure they are gathering the right data and leveraging the right queries that senior managers need to understand the value of learning investments.

The term analytics is part of a technology term commonly used as the engine for the complex and heavy number crunching that takes place in powerful measurement solutions. That technology term is referred to as OLAP – online analytical processing. OLAP tools are rapidly becoming the tool of choice to aggregate and parse through thousands of data points to serve up relevant metrics surrounding the learning organization.

However, it is extremely important to point out that OLAP tools, although powerful, are only as mighty as the queries they execute and the data within the queries. The most challenging elements of learning analytics has never been and will never be the technology, but instead are the processes that work behind the scenes to feed the technology.

First, the source data is absolutely vital. Feeding powerful OLAP tools the right data is extremely important. Activity based data such as number of employees trained, number of classes run, attendee to enrollment ratios etc. are nice forms of data. And, when combined with financial information fed from ERP systems, such as organization revenue or a training budget, they can make for interesting measures.

However, a senior manager who funds organizational learning may not find activity metrics all that meaningful. An activity metric such as number of employees trained per sales dollar doesn’t tell the manager how a multi-million dollar training investment impacted the job performance of the employees trained. Thus, you need to power your OLAP engines to collect performance-based data. This is data on learner reaction to training, quantitative evidence of knowledge/skill transfer, key ratios around impact to the job as observed by managers or experts or evidenced through participants when they are back on the job, linkage of training to key business results, and finally the financial return isolated to training relative to the cost of deploying the training. These performance metrics are significantly more powerful to proving the value of learning versus activity metrics. They are also harder to collect and many learning organizations don’t collect them because of the perceived complexity of doing so.

Unfortunately, the day is gone when the learning organization can shrug off performance-based metrics. Sluggish economies are prompting management to scrutinize training budgets. Performance metrics are quickly becoming a learning organizations best (and sometimes) only friend when they want to be at the table with management helping improve the business as opposed to on the table susceptible to cost cutting.

Building the right processes, using best practice measurement methodologies, and leveraging appropriate technologies can help the learning organization significantly streamline the cost and time to collect, store, process, and report performance-based data gathered from multiple stakeholders and systems as opposed to just activity data that is easy to pull from an ERP or LMS.

Once you have the best performance-based data to run through OLAP tools you then need to design the most optimal and meaningful queries so it can be leveraged properly. Here again, OLAP tools are wonderful enablers but without the knowledge of measurement and learning the query you create can have relatively little value. Solid queries based on tried and true learning measurement models such as Dr. Donald Kirkpatrick’s Learning Levels model that has been around for nearly a half a century and Dr. Jack Phillips ROI Process that is widely used around the world and endorsed by the American Society for Training and Development, are vital connecting points for OLAP queries. Programming a balanced scorecard of key performance indicators tied back to performance-based data linked to industry acknowledged measurement models are the queries you need to run your learning organization. Simply providing an OLAP tool and providing your stakeholders the flexibility to write their own queries can be dangerous and counter-productive.

Learning analytics is a powerful term and not one that should be taken lightly. The learning industry is on the verge of revolutionary change through the use of analytics. In other business processes such as the finance function, supply chain management, and customer-relationship management, analytics are common tools used day-to-day for information decision-making leveraging a balanced scorecard of metrics specific to those functions. As the analytics train pulls into the training departments, training practitioners must ensure that the data and queries powering the analytics are the most appropriate for making decisions to manage learning investments. A careful check under the hood of a learning analytics tool to ensure the right data and queries are being used is a best practice and obligation for any organization seeking technology to measure their learning investments.