Webinar: Measuring What Matters
Moving beyond basic LMS metrics to an outcomes-based measurement model
This week’s webinar featured two high-powered speakers, Dani Johnson, Vice President Learning and Development Research at Bersin by Deloitte and Axonify CEO Carol Leaman. Dani and Carol shared their substantial collective wisdom on the relationship between data collection and business outcomes in L&D and what high impact learning organizations are doing right.
Dani opened the webinar by describing research from Bersin that identified four levels of organizations, labelled according to their level of L&D maturity within the High Impact Learning Organization (HILO) model. Organizations at Level 1, the lowest level, tend to rely heavily on data and reports easily obtained from an LMS or other record system, Dani explained. They also focus on completion data and assessments. Organizations at the highest level, Level 4, use all available data to inform employee development and focus on using data to determine appropriate risks. Dani also pointed out that they tend to look at mistakes as data to effect where the organization is moving and look at what isn’t working so they can make appropriate corrections.
Bersin’s HILO research also revealed four major insights into the ways you can and should use data to better inform your L&D.
- Mature organizations use data for better design
The most successful organizations are not afraid of taking a hard look at the effectiveness of their learning programs. You shouldn’t be either. The research shows that 64% of Level 4 organizations measure how development experiences are being improved, double the number of Level 3 organizations and almost 6 times the number of Level 1 organizations. 82% also incorporate feedback data as they design future development activities, something that organizations at the other levels seldom do.
Data is also driving a change in the methods Level 4 organizations use to deliver learning. They are 49 times more likely than Level 1 organizations to use search software, 43 times more likely to use VR and 32 times more likely to use mobile apps. This demonstrates that L&D operatives at Level 4 are much better at understanding and meeting the needs of the entire organization.
- Performance data is playing a larger role in employee development
The end game for L&D is no longer simply learning. It’s now about performance and it’s critical that we collect data that allows us to improve that performance. Gone are the days when we simply collected data on how much people liked a course or what it took to create the course. Learning is work and work is learning, says Dani. This data can include everything from customer and client feedback, which 93% of Level 4 organizations are currently using, to feedback from managers and mentors which 94% of Level 4 organizations use. Simply put, mature organizations collect performance data from more sources and more often than less mature organizations.
- Mature organizations use a wider variety of data
It isn’t just about the amount of data collected, it is also about the source of the data. Level 4 organizations tend to pay attention to data collected outside of L&D and they don’t just measure the usual ROI analytics used to justify their existence. They align what they measure with what the company measures. If there is a problem with turnover, L&D should be asking what it can do to address that. Within many organizations, there is a disconnect between the impacts valued by business leaders and the ones most likely to be measured. L&D must care more about the outcomes and metrics that matter to the organization. This is exactly what Level 4 organizations are doing.
- Data Collection isn’t just something that happens at the end of a course
While organizations can and should collect data on effectiveness and efficiency they should also maintain their focus on aligning L&D with organizational goals. This might require L&D to simply get out and talk to people rather than rely on a surveys or assumptions. L&D also needs to maintain a focus on business outcomes.
“Data can either build credibility or it can kill it”, Dani says. We need to understand how our efforts are affecting performance and we should be able to speak to it and adapt as necessary. If everyone else is talking about product throughput or turnover and we are talking about learner satisfaction there is disconnect. Level 4 organizations are making the connections.
Outcomes -> Do -> Know -> Content
To better align with the business to drive measurable impact, Carol recommends working backwards from the end goal. Identify business outcomes first, advises Carol. Then determine what your people need to do to meet these business outcomes. If your outcome is to reduce safety incidents by a specific percentage, what do your people need to do (or not do) to achieve this? People do what they know, she says, and you may have to adjust what they know. This is where you develop content that provides the knowledge they need in order to meet the business outcomes you identified. Level 1 and 2 organizations tend to skip the “do” and “know” steps and simply adjust their content without truly understanding what their people need to do or know to be successful.
Carol refers to this as an Outcomes-First model where focused content results in measurable business impact. You can apply it to less tangible outcomes as well—with leadership skill development for example. Identify the outcome (leadership development) and then define the specific skills you want people to exhibit. Use that to identify key learning points, which you can then break down into micro topics to help develop your learning content. Continually collect data to identify learning gaps and leverage technology to help close them.
Measurement is the future. In this competitive world, organizations have to pay attention to data and mature, successful organizations are doing exactly that. Are you?
In case you missed this week’s webinar, you can watch it on-demand at any time.