Microlearning + Big Data + Machine Learning

Watch Video


Good morning. Good afternoon. Welcome to today’s webinar. My name is Alec, and I’m going to be here as usual today to help answer any of your general or technical questions, but before we get started, I’d like to cover a couple of things that you need to know about the session today.

There is no dial in number for attendees, so all audio will be streamed through your computer speakers or headphones, so please adjust your volume there accordingly. If at any point you lose audio, please try refreshing your browser first. If that does not solve your issue, please just relaunch the event.

You could submit questions for today’s speakers by clicking on the Q&A icon located on the right hand side of your screen, labeled here with this top arrow. Click that icon, type in your question, and click Submit. If it’s content-related, it may be pulled during the Q&A session, and if it is a technical question, I will have an answer for you that would appear on your screen shortly after you submit. You could follow along with the presentation by clicking on the Handouts icon also located on the right hand side of screen, labeled here with this bottom arrow. Click that icon, and you can pull down a PDF from there.

All right. Today’s webinar is being recorded. You’re going to receive the link to that recording in a follow up email. Please allow at least a couple of business days before that information is sent. In that same email from HTM Events you’re going to have a link to the PDF as well as your HRCI and SHRM certification codes for today’s event.

All right, so in today’s webinar, Microlearning + Big Data + Machine Learning: How to Finally Prove the Impact of Training, we have a couple of great speakers with us here today. First, it’s our pleasure to have Carol Leaman returning, the CEO of Axonify. Let me get it here. Okay. Axonify is a disruptor in the corporate learning space and an innovator behind the Axonify Microlearning platform, proven to increase employee knowledge and performance necessary for achieving targeted business results. Prior to Axonify, Carol was the CEO of PostRank Inc., a social engagement analytics company that she sold to Google in June of 2011.

With Carol, we have Chad McIntosh. Chad is the vice president of asset protection and risk management for Bloomingdale’s, and Chad previously worked for stores’ loss prevention and Macy’s Central, so it is our pleasure to have both Carol and Chad with us. With that out of the way, Carol, let’s go ahead and get started.


Carol Leaman: 

Thank you very much, Alec, and terrific to have you all on the webinar today. As Alec mentioned, we’re going to be talking about the impact of measurement and how measurement can in fact help you deliver some business outcomes. We’re going to also cover the emergence of microlearning data and machine learning, and give you a little bit of the underpinnings of that. There’s a lot of conversation going on out there in the learning world right now about microlearning and machine learning specifically, so we’ll talk just a little bit about that. We’re also going to show you what measurement looks like and can potentially look like in your organizations. Then it’ll be my pleasure to turn it over to Chad McIntosh to go through the impact measurement at Bloomingdale’s and some of the phenomenal results that they have been getting.

What I thought I would do first is just level set for those of you who aren’t familiar with Axonify. I just thought it would be really important to lay some foundation about what we are as a company and what we do, because it really does provide that basic understanding for the conversation that will ensue through the rest of the webinar. What I’ll say about Axonify is it really does play a pivotal role in business impact measurement. It’s quite transformative, because it blends the best of modern workplace learning principles in really meaningful ways, so, some of those ways.

The adaptive microlearning experience really drives continuous personalized learning that supports the needs of individual employees at the scale of a modern enterprise. We also apply evidence-based learning science principles, including spaced repetition, which many of you may have heard about, retrieval practice, the act of questioning as a learning modality, and confidence-based assessment to drive long-term retention and application of that knowledge in the workplace. There are lots of motivational triggers, including casual gameplay and an array of game mechanics, and they’re built in to the experience to really prompt sustained employee engagement and enjoyment with the learning experience, which are two key drivers in the modern workplace today. Continuous learning allows us to capture and apply data in really new and meaningful ways, which is going to be really the basis of the conversation today, and that includes insight, more knowledge, behavior results, all of the things that are very measurable and play into our business outcome. And finally, the continuous learning experience is available anytime, anywhere, via any internet-enabled device, which allows employees to build a learning habit, really, into their daily workflow.

With that, the first thing we’re going to start with is a quick poll. It’s a little bit of a different take on this question. We thought we’d ask not if you can measure the impact of learning or even whether you want to, but rather, as a learning practitioner, whether you fundamentally believe that you should be measuring the impact of learning on the business. So, take a look at. Do you think that organizations should measure the impact of learning? With the emphasis there being on the word “should.” We’ll give you all just a few seconds to answer yes, no, or not sure, and as soon as a good number of you have responded Alec will show us what the poll results are.

Measuring the impact of learning is becoming increasingly a conversation in the learning and development world, and as we know it’s difficult to do or has been difficult to do, so do you think that organizations should measure the impact of learning? I don’t know if none of you answered the question, but it’s a zero, zero, and zero, so I’m thinking maybe we didn’t get any answer. I don’t know, Alec, if that’s [crosstalk 00:07:04].



Hey, Carol. Yeah, I’ll jump in here. I’m not sure why you’re not seeing the percentages, but we did have quite a few people answer, and we ended up with 99% saying yes and 1% saying not sure, with 0% saying no.


Carol Leaman: 

[inaudible 00:07:20].



So an overwhelming majority said yes.


Carol Leaman: 

Okay. Well, that is absolutely terrific to know. Thanks for that, Alec. It is the age old unsolved question and really existential crisis for learning and development. Perhaps some of you listening in are at the top of the pack when it comes to impact measurement, which typically means it’s a one-time manual exercise for a handful of programs. That’s really what most of us have had available to us. Very limited, one-time sorts of measurements, and it hasn’t really been scalable until now, but what would it mean for your role if you could measure impact around metrics that really matter to the business? And really that’s the question we’re addressing today.

Before we get into the solutions, first let’s take a step back and look at the current picture, one that’s actually existed for decades. It just is the case and many of you will know this that businesses spend an extraordinary amount of money every year training on the same things over and over again. It is $140 plus billion a year, just an astronomical amount of money, but at the same time businesses lack insight into what’s working as far as training goes and in fact, the number one challenge for training professionals, when they’re surveyed, is this lack of being able to have insight into the value of what they’re doing. And not only that, the businesses themselves, so the C-suite, simply can’t understand the value of training either. There’s no way to demonstrate to CFOs, COOs, CEOs what the impact is on learning. It’s been that elusive activity. There’s lots of anecdotal evidence, lots of good feeling often around training, but the reality is 92% of CEOs cannot see the impact.

To highlight that point, the Brandon Hall Group said that the number one biggest challenge cited by learning professionals, as I said a moment ago, is their inability to prove impact on the business, and 38%, in fact, in the study they did, said they don’t do any form of business impact measurement. So it becomes really hard to justify the investment that an organization is making sometimes into learning and development when those professionals work their tails off to try to deliver a really impactful experience for the business, and yet be unable to really demonstrably measure and show the impact on top line revenue or bottom line expenses to what’s actually the result of all of the learning taking place.

Let’s take a deeper look at a really important concept as it relates to all of this, and that concept is attribution, so I’m going to talk a little bit about another kind of related example in the business about attribution, but we’ll start with this. Really, what if you could do these things? Measure and understand the impact of training programs on the business, and leverage that understanding to just continuously improve the effectiveness in real time on your training programs, and then very strategically use training as a lever to optimize business results on the fly. So know at any given point in time what’s working and not, and being able to change it instantly. Not only that, adapting to individual learning gaps based on what you know an individual knows or doesn’t know and being able to change that very, very quickly.

What if you could do it, and what if you could do all of those things at scale for each and every program that you deliver? So being able to do all of those things and really provide what we call learning attribution. The concept of attribution is something that has been around for quite some time, and in fact 20 years ago, if we look at a related field in corporate enterprise, and that field being marketing. The marketing team’s value to an organization 20 years really was not clearly understood. They were very much in the same position that learning professionals find themselves in today. They spent a lot money on things like billboards, on mailers to homes, on TV ads, on radio ads, and they really had no way to attribute each of those marketing activities to the impact on sales. Again, they were able to use intuition. They kind of had an idea sometimes of what worked and didn’t, based on timing or volume of activity, but the metrics that they did have, very, very rudimentary. Really were not as insightful as they could have been to the business.

But if you fast forward to today, the field of marketing has dramatically changed. Corporate marketing departments have fundamentally transformed their perceived value to the business over the last decade or so, and they really catapulted their organizational value and just even reputation within the organization from almost an overhead line item to a highly valued, data-driven, revenue-impacting strategic part of the business that’s capable of demonstrating a direct impact on virtually every marketing activity that occurred and tying that to a business result. It’s become known as marketing attribution and it’s now very common practice for every corporate marketing department. So for every dollar they spend on a campaign, they can actually attribute it to [inaudible 00:14:25] and they no longer just have the number of ads they placed or billboards they put up. They have very granular metrics that the business understands and values, and monitors, in fact, quite frequently.

So how did they get there? Well, they fundamentally reinvented the whole marketing function with the advent of the internet. It caused the emergence of the field of digital marketing, mobile, social marketing. We’re all subjected to that, and it provided the digital forensics to trace marketing spend with a high degree of confidence to a business result, like unit sales or dollars sold, and now they drive the business. They could clearly map every breadcrumb and action a customer takes, so things that we all click on, on the internet, for example, or emails that we open, forms we fill out. All of those things that we do on the internet, and they started to tie those directly to the end result, which was a purchase of something. So with all the available tools involved and in the delivery and tracking of corporate training, we are now in a position to give learning enablement professionals, one of those only functions left in the organization that can’t really … truly have not been able to justify their value. We’re able to give learning and development professionals the same kinds of tools and techniques that marketing very successfully used to become more than just a cost center, but a really business partner and business driver in their businesses.

The problem was in the fact, historically, that L&D professionals have been unable to capture and unlock the value of big data, which is what marketing has done. They really have lacked access to the valuable big data needed to tie training to a business metric, like increase in sales or decrease in expenses in any meaningful way. The problem can be attributed in large part to traditional ways of measuring, like test scores, attempts at answering questions or passing tests or completions of courses. All of the things that we historically have very typically tracked in the learning and development fields, and often true in LMS in the last decade. Those measurements unfortunately are taken at a single point in time, as we all know. They offer limited value in terms of the volume of data, the depth of data, the dimension of data. All of those things are really needed to catalyze the same type of digital transformation that virtually every industry and profession has undergone in the last decade.

So why haven’t we evolved past these things? Durations, completions, and grades? Which, by the way, only 5% of learning and development professionals when surveyed believed that those have any really meaningful value. Perhaps you’ve heard of the streetlight effect, which is highlighted here on the slide. It really is an observation bias that occurs when people only search for something where it’s easiest to look. I’m not going to go through the entire story of the streetlight effect, but essentially we have been looking for years where things are most visible. And now there are new tools and tricks that have just been out of the sight, in the dark, where we’re had all of this data collected that we have not had at our access previously, and we can now reach past where the light is shining and into those other reaches or other areas to find data that’s super, super important.

So to attribute learning to business results, these other forms of data that are now accessible to us, we like to call the five Vs of big data. They are velocity of data, variety of data, veracity of data, which is the accuracy of data, the value of data, and the volume of data. Those five things, which are now completely available as a result of technology and really create those big data reserves, form the basis of everything that is now possible for learning and development to do. It is now possible for L&D and business stakeholders to leverage those same big data principles and machine learning intelligence to digitally transform their functions and the industry, in fact. And we can now understand truly learning’s impact and do what marketing did, which is attribute learning to the activities of learning and development.

The key really is microlearning data, those small granular bits of data that really are the underpinnings of everything, so really the hero of the story is microlearning data, those small bits of data. What makes learning attribution or level four measurement, if you know the Kirkpatrick Model, happen at scale. These are the things that allow this reality. Most people know Axonify specifically as a learning company. What most people don’t know is that at the core it’s all about data and big data, and what I don’t mean are just attributes of an employee profile. I mean millions of rows of learning and interaction and knowledge data that are collected every day, and they’re really used to understand four things, which help to cater and individualize learning experience at scale. If you look here, not only is there demographic data just about an individual role or where somebody works, how long they’ve worked there, the team or the department that they’re working in, but there is also a tremendous amount of highly valuable data that is married with the demographic data.

Things like history, what learning history exists, the progress that the learner is making, their confidence in their knowledge, the recency with which they’ve been presented a key learning point, engagement. How often they are choosing to learn, and how frequently are they choosing to learn certain things? What’s the behavior that they exhibit? What are the results that the individuals are getting? So all of these very fine granular pieces of data provide massive volume onto which machine learning can be applied to create insight and be able to extract some really interesting things. So gone are the days of course completions as a measure on what people know and instead we now have a very, very rich granular dataset with which to work, and what you end up with is attribution at scale and adaptation, each moment, each learner, so that you can optimize the learning experience very specifically.

This is an expanded view of the micro data. It’s really the right side of the previous chart, and this is where the continuous microlearning approach combined with proven brain science principles really capture massive volumes of valuable data at scale. The starting point is at the top, that real base unit of knowledge, which is a question in time. That’s the center of the continuous big data that you can capture. Now with that is a starting point for each question that an employee answers, you can capture several critical pieces of information that give it depth and dimension, including question responses at multiple intervals, so not just one presentation, but more than one, at multiple degrees of difficulty, so think about Bloom’s Taxonomy. Confidence, self-assessed by the individual learner. Iteration, how many times have they answered question, their proficiency, and where the question really fits into their overall knowledge progression or universe.

This can happen continuously as employees engage in daily training, and what it allows you to do is all the time, continuously roll out knowledge into a view that tells you in real time what someone knows and what they don’t know, when a piece of information is forgotten, so you can immediately intervene automatically through an algorithm, and remediate that knowledge. And in some cases you can augment those particular data points with on the job behavior observations that really then add to the data pool to understand how the information is being applied. So this is real time learning that I’m talking about and the data that’s involved in it is big.

Just to kind of bring all of that down to something that’s a little more digestible and relatable, this will give you a little bit of perspective. What you see here in the background of this picture is a reflection of the Cincinnati Public Library. It’s been closed for decades, but really was an iconic piece in American history. I’m using this to give you a sense of scale for the learning data that can be captured with a microlearning platform during daily training sessions. The first two figures are critical to generating the volume of learning data, and this is in fact our experience across all of our customers when you use microlearning. 74% will participate on average every 30 days, which translates to over 100 sessions per employee, per year. So you can imagine how many times learners are engaging in learning. It’s about two to three times a week on average, which again allows that basis of huge amounts of data.

If there are, as we know, at least 10 data points that you can capture about the learner and what they’re learning with every question, you result in about 520 questions per year and 5,200 individual data points per employee, per year. And in an organization with 50,000 people that adds up to about 250 million discrete data points for an organization in a year, and that equates to 593,000 printed pages, which is what you see in the background of the picture. All of those books in total have about that many printed pages, so that’s just one organization and the volume of data that could be collected automatically to really then start to extract some really interesting insights, so you’ve got the depth and dimension needed, those five big Vs of data to really start to make learning measurement possible.

The microlearning data along with your business data, whether it’s sales data, safety or OSHA incident reporting data, transaction science data, there is an incredible amount of business data that you are already capturing. They can be fed into this continuous learning engine and used with learning data to extract some really interesting things. I’m not going to go through all of these points specifically, but essentially a machine learning engine which is technology, that takes all of those data points, and applies things like regression analysis to them, to scientifically and technically extract themes from the data. That’s the way that you can end up with actionable recommendations, so it goes through a process of data cleansing to really organize the data in ways that are meaningful, statistical correlation so that you can in fact start to look at what things are impacting other things. Is confidence or recency of seeing information or knowledge important, for example?

Then you get to measurement, which is really where the real value machine learning starts to flex its muscle, so you start to do those specific correlations though algorithms to start to tie together the business outcomes with certain learning data. A gap analysis will be able to flag where learning may be not having the impact that you expect it to have, and then those actionable recommendations results that talk to you about what parts of learning may be augmented, what parts may be going really, really well. So all of that data fed into the machine goes through this process and then results in some really interesting insights about what learning and development can do, and I’m going to show you a couple of slides in a moment of what that can look like.

Really quickly big data, just again to tie it back to other industries to make it relatable. Other industries have been transformed by big data. We all know in the retail space, for example, data is used to drive transaction size and volume. People are measuring what you’ve done online and tying that to whether you walk in the store and purchase something in store. Entertainment. Everything we do on Netflix we know is being tracked and archived and used to recommend stuff to us. Banking. Credit fraud is a huge way that big data is used, and in insurance, to mitigate risk exposure. So, many industries are on this trail of using big data to really, really drive their businesses forward.

We like to say that everything changes now. You can use microlearning to get a huge amount of data at a very granular level, apply it to business goals, tie it to business goals, which is really what we call the language, apply machine learning to all of that to extract intelligence, and what you end up with is learning attribution at scale. So it really is a new frontier from a learning and development perspective, and learning and development now has the opportunity to be a strategic partner to the business to really prove the value of what they’re doing in the organization. No longer is it that learning and development is creating lots of content, conducting lots of instructor-led courses, and really hoping that what they’re doing is going to be provable to C-suite folks.

Here’s a screenshot that really shows how learning practitioners can measure and understand business impact at scale. Armed with this information, they can demonstrate value to the business, but more importantly, unpack underperforming programs. All of it is in the context of the other factors that play a role influencing business results, seasonality, store promotions, but what you can see is really an isolation of what’s contributing to a business result and what things may not be contributing in ways that they should be. And allow you to identify them and adjust and see what’s going on over time. Similarly, we’re able to make sense of the data to make it actionable for frontline managers also, so this is an example of what frontline managers can see as it relates to their teams. They can see when a target is at risk of being met, for example, but it gets even smarter.

It serves up those actionable recommendations on the things that they need to do to get the business back on track to meet their objectives. On the bottom half of the screen we can see that this frontline manager, for example, has an opportunity to improve if they lift participation in training, training frequency, and the subjects they need their team to pay attention to, so it becomes very prescriptive in terms of which actions you can take to move the needle on the business. And finally, it self-heals through adaptive technology the gaps that are … How it does is by triggering training in the subjects that matter for the people that need it in a way that continuously optimizes business results and keeps the business on the path to business target achievement. I’m not going to go through the seven steps of impact. It really is, measuring impact is easy once you have the big learning data and machine learning in place, and if you want to learn more about this, you can certainly onto the Axonify website and download this.

With that, I am delighted to introduce Chad McIntosh from Bloomingdale’s. We’ve been partnered with Chad for several years now, and he played an instrumental role in helping us shape the Axonify Impact product doing this business measurement. He’s going to walk you through a big of the Bloomingdale’s story and the insights that they’re now seeing on the impact training on their business. With that, over to you, Chad.


Chad Mclntosh:

Thank you, Carol. I appreciate it. I appreciate the opportunity to kind of share the Bloomingdale’s story and really give everybody an understanding of how we use the tool, and what we’re seeing from the impact process and attribution. This is really one of the most exciting changes I think that we’re doing today with regard to learning and development in our organization.

As Carol said, I’m the VP of asset protection and risk management for Bloomingdale’s, and we’re a department store based in New York City, so I’m responsible for the safety of people, products, facilities at Bloomingdale’s. We’re 50 stores. My responsibility also includes security, shortage reduction, crisis management, and business continuity, so it’s a very complex job and it’s actually my dream job at Bloomingdale’s in New York City. I just keep saying New York City because I think it’s going to become important on the agility of the process going forward for us, one of the things that attracted us to Axonify. Thank you, Carol.

We started our relationship in 2012, as Carol mentioned earlier. I was challenged with getting a loss prevention message, a shortage reduction message, a safety message to our organization. My philosophy and my thought was in conflict with what our corporate learning process was, so we had all the traditional processes and programs, a new hire orientation that was a week-long series, kind of a fire hose of information to the associates. They would go into their stores. There would be store rallies, store meetings, an awareness program that was based on a poster campaign in the organization. So there was a real disconnect with what I saw as the business need. Some of the issues that we were having was inconsistency from department to department, store to store.

We’re a 50-store organization, and the method was actually to put out a training alert, send it to the general manager of the store. They would then interpret it and share that with the store family, if you will, so we were getting a lot of variances to the actual message we were trying to get apart, and then certainly never really understood who was getting it and who was adapting to the learning or what behaviors were we actually changing in our stores. So lack of engagement and appeal to the traditional approach, certainly who was paying attention, who wasn’t paying attention. We had no way of measuring the knowledge and the retention of our message and what we were trying to get apart, and poor compliance. Escalating claims certainly was one of the big reasons for getting into the process. How do we make the Bloomingdale’s environment safer than it was back before our relationship with Axonify? Then there was a stigma about training in general in our environment, and it wasn’t an agile process. As I said, it is a static poster at the coworker door that nobody was reading.

In today’s world in New York City I need to get information to the store family very, very quickly, and have them understand that information and that lesson so that they could adapt and be safer in the stores today. We knew that they weren’t remembering what we were applying because the claims were escalating, and it was a very expensive process for us, so we knew we needed to do something. Certainly we believed that a lot of the claims that we were experiencing with general liability and workers’ comp were preventable. Then there was a lack of understanding of LP, loss prevention, asset protection, risk management programs in general in the Bloomingdale’s environment. Again, I had no way of measuring the understanding of the store family and who was involved and who wasn’t involved in our message and our program. Thank you, Carol.

So in 2012 we selected Axonify as our microlearning platform. Certainly the flexibility that Axonify has offered us, when you look at the fact that we have 90% voluntary participation, so up until recently the process and the lessons and the microlearning process wasn’t required of an associate to be involved. So we knew that we were changing the culture of the company when we were hitting 90% voluntary participation. Of 15,000 employees, when you think about 15,000 employees, that puts us over 13,000 people involved in asset protection and loss prevention, safety in the Bloomingdale’s environment. That’s why we believe that we got to the 41% reduction in our safety incidence in our stores, knowing that our associates were better trained, more confident in being able to kind of handle anything that came along. The point of this, the 41%, really drove home with my boss and the EVP of stores, when they were walking one of our stores here in New York, and there was a spill in the cosmetics department. They stopped, and they were attempting to clean it up. One of the cosmetic associates came over to them and said, “No, stop. You’re not doing it. I learned about it on Axonify.” So we knew at that point that we were changing the culture of our business. Thank you, Carol.

This is the bottom line to it and the impact that we’ve had since 2012. Over $10 million. I will tell you that asset protection and loss prevention guys or gals are very adept at throwing things at a problem, multiple things at a problem, and really not understanding what was actually having the impact and driving the performance, but we knew that $10 million was something that we were very proud of and that we were changing the safety behaviors in our stores. Thank you, Carol.

In late last year we entered into the Axonify Impact early adopter program, which is very exciting to us because of the power of Impact, and as you’ll see in future slides … Well, here with attribution and Carol talked about attribution, so while I said we threw several things at the safety claims and experience in our stores, and got to the $10 million reduction, I now can go back to management and clearly tell them what we can attribute to the general liability reduction in our stores and what we can attribute to the workers’ comp reduction in our stores. So the process, the platform, and the training that we’ve done, we know that $3 million is directly attributable savings in this process for us, something again we’re very, very proud of. Getting to this kind of granular state is really important, and you’ll see that in another slide as well.

Here’s the other slide as well, as a matter of fact. When you think about proactively adjusting content based on what’s working and what’s not working, so Axonify can tell us now what’s generating the impact in our environment today. So we could see what the results are, tie the results to the lessons and then the content, and then we can either retire or replace the content based on what we’re learning and being much more proactive in the process today than we have been in the past. And then expanding the sets of business goals we’re measuring, so we want to expand the KPIs that we’re looking at, the business goals that my teams are concerned with, things like shortage and safety, of course. But also things that management and the organization is very concerned about.

For example, net promoter scores. A net promoter score is customer feedback on the expense that they had in the Bloomingdale’s store. There’s not an associate at Bloomingdale’s that is not reviewed on net promoter scores, so it’s something that’s important to everybody. So how do we tie this process? And we just started really measuring net promoter scores through the process, so how do we tie this process to identify what we’re teaching, what we’re training, and really proactively adjust the content based on the feedback and then enable leadership at all levels to act on these insights? One of the things that we really want to do as an organization, and Carol spoke about the frontline managers earlier, is arm them with the information not only to direct performance in the store, coaching in the moment, adjusting associate behavior, all those things that we know will drive confidence in the experience for our customers at Bloomingdale’s.

Ultimately, we see this as really making us much more agile than we’ve been to this point with all the savings that we’ve actually been able to accomplish with Axonify and the platform. We believe now we’ll be able to take a look at turnover in the organization, all those things that senior management in the organization and store management in the organization are concerned about, care about, so will really help the organization kind of drive performance and process improvement in all of our locations going forward. So, Carol, thank you.


Carol Leaman: 

Thank you, Chad. For anybody who is looking for additional detail, there are some resources available on the slide. You can see here impact measurement specifically, there’s a document that you can download, a guide to microlearning if you’d like to learn more about that and get on the microlearning bandwagon, and then of course the Bloomingdale’s story in more detail is available, also. We welcome you to go and take a look at those. With that, I’ll turn it back over to Alec for questions.



All right, Carol. Thank you, and thank you to Chad as well. At this point we do have some time for questions, so if you have a question for either of today’s speakers, click on that Q&A icon on the right hand side of your screen and click Submit. With that, we do have quite a few here in the queue, and we’re going to start with one that’s going to go to you, Carol.

This, it comes from William. William’s question is, is anyone looking into the behaviors and performance being modified by the training? We are currently attempting this ourselves.


Carol Leaman:

Yeah, so that’s a great question. There is actually a part of the platform called Behaviors and Inspections where in one case, this is just kind of the largest example of that in the data gathered, there is an organization that assigns individuals to observe behaviors being performed by employees, and then just very quickly accepting whether the behavior is correct or incorrect. So automatically capturing in the moment those behavior observations, which are tied specifically back to the knowledge that individuals have in those topic areas, so I’ll give you a lot of examples. You can capture, for example … As a sales rep, the behaviors that you would like your sales associates to exhibit are proper tone of voice, they know the products really well, they can deal with objections if a customer decides not to purchase something, all of those behaviors that you want a superlative sales associate to be able to demonstrate.

You can quickly watch and then assess correct or incorrect, tie it back to how did that individual do when it came to the module around customer service? Was their knowledge high? Was their knowledge low? And the algorithm will tie those behavior observations back to the actual knowledge of your learner set and automatically self-heal the knowledge and fill the gaps where knowledge is deficient. Interestingly, that one particular organization, they capture one million observations a month on average across their organization, and they discovered that behaviors is some areas were poor but the knowledge was quite high. What that told them was that the content that had been created to address knowledge and behaviors wasn’t the right content, so the content was wrong, knowledge of that content was quite high, but the behaviors exhibited were poor, and therefore it was the wrong content. And they were able to go back and adjust that very specifically.



All right. Perfect. We’re going to move on. We’re going to send this one to Chad. This one comes from Drew. The question is, with loss prevention and safety there are some clear metrics that can be used to measure the extent of employee awareness and behavior change. I feel that this type of training is a relatively easy sale, and does Bloomys use the platform for their own leadership development too?


Chad Mclntosh: 

We aren’t currently today, so we’re looking at how we expand the use of the platform, so we’ve added sales and service training and really follow up to the new hire orientation process that I spoke to. So we’re expanding the use of the tool, and now that we can be more granular about the information and the results and adjust what we’re doing, I believe it’s going to become a much more powerful tool for us in the future.



All right. Perfect. This one I’ll send to you, Carol. This comes from Emeron. How do you leverage the supervisor manager engagement in this whole entire journey? I think this could be a challenge.


Carol Leaman: 

That’s another great question. We designed a part of the platform called Leader Zone, which allows leaders of people to basically look at how their direct reports, those people that they supervise, are doing. What do they know? What do they not know? What topics are they struggling in? How often are they participating? Are they taking extra training? You get a very granular set of metrics, person by person, across your team to allow you to coach individuals and they’re alert. For example, if I was a new employee and I didn’t know a whole lot and my confidence in my knowledge was quite low, something else that gets measured with every learning point, my manager can see that I’m struggling and I’m not making the progress in the first month or two of my employment with the organization. And then allow me to be coached personally, one-on-one with very specific information.

So you seem to be doing well in these topic areas, but not so well in these others, and is there something at the fundamental disconnect that I can help you with? Or you seem to know a ton of things. Why is your confidence low? One of the things we know is that when an employee lacks confidence in their knowledge, even though they may be a very knowledgeable person, they will not act because they doubt the level of knowledge that they have, so those key metrics really give tools in the hands of leaders to effectively manage their teams and the knowledge and performance of their teams at a much more granular level.



Perfect [crosstalk 00:52:43].


Chad Mclntosh: 

In that, Carol, if I could add something to that from our perspective, that we use the Leader Zones in all of our stores today, and the sales managers are all very competitive. We use a lot of data to drive the business, so we use the information here to really talk about participation in management meetings in our stores and really get them involved. Nobody wants to be on the bottom of the list with regard to participation in the store in the store managers’ meeting when the store manager is standing in front of the group talking about how this tool is working, getting everybody involved in the process and making the store better. So, it is an important tool, an invaluable tool, and it’s something that has become part of our culture.



Great. Yeah, thank you for that, Chad and Carol. I believe this one here came in and this is from Bruce, and I’m going to start with you, Chad, and we may have to bounce it to Carol if this isn’t for you. But the question is, have you tried trying your approach with a progression certification program for the employees?


Chad Mclntosh: 

A progression certification program for the employees.



I’d say probably just a regular certification program with your approach is what they’re asking about.


Chad Mclntosh: 

In the platform and the learning process we actually … as an associate learns more about a particular topic, they go through kind of an expert status, so they are being certified, in essence, with regard to whatever the topic is and they become an expert in that topic. And we go back later on in the process to check in with them, with that reaching that expert level, to be sure that they’re retaining and they’ve still got a firm grasp of the information or the lesson we were trying to teach. Carol, you may have a better answer than that one.


Carol Leaman: 

Just to augment what you’re saying, Chad, so there is actually a certification module on the platform that if you have employees who require regulatory certification by a specific date, it allows you to weave those questions into the daily learning experience. And the customers that are using certifications generally find their entire employee population complete certifications much more quickly than they otherwise would, because it’s just part of that natural daily experience. So you can set your dates and basically just weave it into the daily learning experience, so it isn’t that big, heavy, watch all these modules, answer all these questions once a year to maintain or acquire a certification. It just becomes part of the natural learning workflow and is very, very effective at certifying quickly and across the board much more prevalently than a traditional environment.



All right. With that, I think that’s going to be all the time we have for questions today. Carol and Chad, thank you both so much. Carol, it’s always a pleasure to have you on this, and Chad, it was great having with us today as well.

If you enjoyed today’s presentation, please take the time to fill out our post-event survey, which is going to appear right as this webinar ends. Your feedback is always important to us and it does help us improve our events moving forward. Also, I had a few questions roll in about this webinar being recorded. This webinar was recorded. You’re going to receive that in a follow up email. I had a couple of people asking for their certification codes. You will receive in your follow up email as well for both SHRM and HRCI. I want to thank again Carol and Chad for that great information shared, I want to thank Axonify for sponsoring today’s event, and everybody else on the line.

Thanks again, and we will see everybody back here for our next Workforce Magazine webinar on Thursday, June 14th, and that is titled Staying Ready for What’s Next: Outcomes vs Learning. Thanks again, everybody, and have a great day.