Recently Administrate’s CEO, John Peebles, presented on the topic of ‘Business Intelligence for Training: What It Looks Like, How to Get There’ at the HR and Future of Work event. Run by the HR Exchange Network, this online event was designed to help HR professionals once again find their footing in these changing times while bringing together HR professionals who are successfully addressing the issues of diversity, personalization, and adapting to the future of work.
You can read John’s talk in full below. You can also head to our Learning Analytics page for view the video for John’s talk. You can also view the webinar directly below.
Transcription
(0:00)
Mason: Hello everyone. Welcome to the second presentation of the day. If you’re just joining us, welcome to Day Two of HR and Future of Work. Alright, I’m honored to introduce our second presenter. He’s also returning speaker Administrate CEO John Peebles, and his session is entitled ‘Business Intelligence for Training: What It Looks Like and How to Get There.’ John, always fantastic to have you. Welcome back, the floor is yours, sir.
John: Thank you, Mason. It’s a real privilege to be speaking again. I’m broadcasting to all of you from my home office here in Edinburgh, Scotland, because, like many of you, we are under quarantine now for COVID-19. But the show must go on. So I’m really pleased to be able to present what I think is a really critical topic at any point, but particularly in unprecedented times like right now, how business intelligence reflects and relates to training is more important than ever.
Just to get started, this is that obligatory slide we’ve all seen in million times where there some stat that somebody will bring out about how much data is created every day and we all get it, we’ve all seen it, hundreds and hundreds of times. I guess the important point is, “That’s great, but how is this impacting our work? How does data becoming information that relates to what we’re doing, particularly in this field of learning development and train education?”
I just wanna put a thought out there for you briefly, which is we only think about infrastructure when it’s not required, right? Or it doesn’t exist. Here’s a bunch of power grids out there, we really all think about electricity and the power grid when it’s not working. So hold that thought, and we’ll come back
(2:00)
to it in a little bit. First, I wanna take you back through history to 1853. This an engraving of Commodore Matthew C. Perry from the US Navy who sailed to Japan to engage in a bit of gunboat diplomacy, which is a term for you right there. The idea was to open up the Japanese nation to trade with the United States and to basically open up that market. This is Matthew C. Perry; he sails into the Tokyo harbor and basically fired what is known as the Paixhans gun, which is the newly installed on his collection of ships, and at the time, this was the first canon that was ever designed to fire explosive shells. He did that from the harbor of Tokyo, and it was such in a tremendous display that basically what happened is it caused the Japanese empire to realize that they were completely out-matched and had completely missed the boat, if you pardon the pun, in terms of technology and their engagement with the modern world.
This is a quote by one of the senior advisers of the time. Basically, “If we take the initiative, we can dominate; if we do not, we will be dominated.” Effectively this kicked-off this display of technical superiority and dominance kicked off the Meiji Period within Japan that culminated in the early 1900s with this picture of the 1907 Tokyo Industrial Exhibition.
Today, when we think of Japan, we think of modernity, we think of building, skyscrapers, technical powerhouse, and so on. It all relates back to this show of gunboat diplomacy in the early 1850s.
Today is an
(4:00)
invitation to think differently about how we’re engaging with the data and with the systems that we have within our learning and training environment. There’s kind of two ways to think of technology. On the left-hand side we have the space race landing on the moon. This is an example of pushing towards a technical outcome. The United States decided that we’re gonna go to the moon and we did that. We put a lot of resources behind it, and so on, it was a conscious decision.
On the right end side though, is a cover Time Magazine from the early ‘90s about the advent of the internet and the picture this Jerry Yang, the founder of Yahoo!, and maybe somewhat relevant today, and the bottom of it talks about hot stocks and brash billionaires. This is an example of the type of the type of technical revolution that pulls us along. We didn’t go out and decide to invent the internet like we did decide to go to the moon, but it’s going to affect all of us and have affected all of us over the last 20 or 30 years, no matter what.
I’ll just say that this is basically how our perspective is relative to business intelligence and the training in L&D space, and that is the playing field has changed, it’s inevitable, and it’s up to us just to figure it out how we are going to respond to this.
Today, I want to talk about business intelligence and transformation. How do we transform our thinking? How do we incorporate business intelligence into our decision-making? And how do we get there?
I think, for starters, it’s very critical to view learning and development as a stand-alone business. By that, I mean, who are the stakeholders? How are you investing? What is the return on that investment, and what is the P&L? A lot of times, if we’re in an internal training leader, we say, look, my stakeholders are my manager, or what have you? We would really encourage you to think beyond this and think,
(6:00)
who are the stakeholders in terms of employees — maybe their managers, maybe it’s the company as a whole? Who are these audiences? What do they need to understand? How do you serve them as if you are a stand-alone business?
That then leads you to figuring out what kind of investment you would need in order to serve them. What the return would be on that, and then, the profit and loss statement. Somebody somewhere is keeping that within your organization. It might as well be you.
As we go through, a lot of times you will say, “Well, I don’t need to do that. I don’t want to do that.” But the cost of ignoring this is high, and we cannot say the best case of ignoring the information that is within your organization is at best career-limiting, but at worst it’s company-limiting, and what we’re all trying to do as educators is to support the companies and the organizations that we’re part of.
So, what does this mean? What are the questions that we want to be asking as leaders? Things like, what are my constraints? Or, could I do more with more? What are my economics? How does this impact my profitability? All these are questions that we should be able to ask fluently within our learning and development organizations.
But, if you’re like, most of the people that I’ve talked to, most of the customers, most of the folks that we interact with, it’s really actually, a hard problem. But why is it so hard? The reason is this is a typical environment that you find within a training organization that is they’ll be a classroom operation with all the spreadsheets and calendars and whiteboards that exist in terms of figuring out what the resources are and how to manage them. That is somewhat separated from the eLearning operation, and that usually involves one or more learning management systems, and then you have the rest of the business, which has commercial systems, operational systems, and servicing to other servicing and support systems.
(8:00)
All these are segmented from each other, in many cases or at best the rest of the business is all operating within one system, and then the eLearning and classroom portions are divorced.
That is a very difficult challenge to solve, and it reminds me of a customer of ours who was talking to us about how spreadsheets get created, and they came to us, and they said, you know, we asked the Salesforce administrator, “Can you add the field the dietary requirements to the CRM?” Because as you know, this can be very important for folks to showing up to get the right meal, and the answer came by very quickly, and it was “No,” because that would gum up the rest of the organization’s view of that particular record, and right away, that creates a spreadsheet.
Similarly, the same customer, software and service as a business selling to large enterprises and they sell multi-year, year-long contracts, and the renewal team was wondering which renewals are at risk? Somebody had a great idea – those customers that take more training are probably at least risk of stopping their subscription, and yet, in order to get access to that information, it created another spreadsheet that had to be emailed into the renewals team on a scheduled basis.
The root problem of all of this, we believe, is that there’s no system of record that is designed for learning and development. That might be a surprising statement for some of us today, but when we think of a learning management system, I tend to think of the good old VCR. The learning management system traditionally tends to be like a web-enabled VCR — it is very content-focused, and you have your content as your videotapes you and you insert that content, your press play, your student goes through it, at the end of the day, they have completed that course.
(10:00)
What that means is that because it’s so focused on delivery and it’s so focused on just the fact that the content needs to be delivered to the student or to the learner, we’re basically trapped by isolated activity metrics, and these isolated activity metrics is to who did what, when, are supposed through canned report.
What does a system of record really look like? It’s a little bit technical, but effectively we think about a system of record in terms of it can hold arbitrary data. It can hold data that goes beyond whatever the stock data set is, or the stock data points are about the learner or the learning interaction. It can extract and transform and load different data from different systems, and it will ingest this data and also share the data outward with other systems.
There is a platform strategy that would underpin the system of record. Basically to support existing requirements today but also future needs, and it can be operationally integrated whether it’s the system that you’re working in day-to-day or that it’s feeding to and from the system that you’re working to day-to-day that’s really really key.
And lastly, the creating of all that data that is coming in and out or moving though, it’s very important and a key component of what a system of record looks like. Data manipulation and introspection, whether it’s the ad hoc querying like an API or reporting facility, that’s really key.
Coming back to this infrastructure that we only ever think about when it’s not available.
We here in the Administrate, we spend a lot of time thinking about learning infrastructure. The reason is because we believe that the ultimate global we all share is better serving the learner while advancing organizational goals. If we think about that, one of the best ways to do that, to really serve the needs of
(12:00)
organization and the needs of the learner is to use data to drive decisions in decision-making.
Really, what we’re talking about today is your technology, and your tech stack that you have at your disposal as an L&D or training leader should help you create a narrative that is supported by data.
When we talk about what that means, is reports exist, they should exist, to answer questions. When you ask questions, what tends to happen is you get the answers back, and those answers will create more questions. To introduce driven decision making is only possible when you’ve got a short question and answer cycle that you can iterate through within seconds or minutes. This is the key test for us when we’re talking with folks that are interested in this topic is I’ll just say, “Is it possible for you to ask a question, run a report, get the answer back, and ask another follow-up question within minutes or seconds.” If the answer is “No,” if it takes hours, if it takes days, or there’s no ability to do any of that, then we’re not really within the realm of being able to use data to drive decision making or at least not do so efficiently enough.
We wanna see this type of cycle be able to happen over and over again because as many of us aware one of the key problem-solving methodologies that are out there is called the Five Why’s, where you start with why, you ask a why again, and so on. You keep going until you get the answer at the root of the problem or the question you are trying to solve.
Without this ability to iterate through the question and answer chain, you’re really saying don’t ask any questions. You have your stock activity reports; that’s it. That’s not
(14:00)
really gonna support you long term.
When you think about the situation that we’re in today, all around the world with COVID-19, there’s questions like, “How’s my training operation going to be affected? If it is affected, how do I need to compensate for different training modalities? Maybe the training that I was doing last week or three weeks ago in that modality of classroom-only is not going to survive in the next few months. If that’s the case, where do I need to reinforce or compensate for specific modality? Based on the data that are existing in my systems today, how can I see where students struggle in a specific modality and alter that or change that? Who are my best instructors? What is the definition best mean as we switch within a modality?” This is just one example of a bunch of questions that customers of ours have been asking themselves for the last few weeks, and it’s been interesting to see the results.
When we talk about business intelligence, it’s these three ingredients that we like to think about. On the left-hand side is that infrastructure piece, the system of record, but in the middle, it’s the humans that are interacting with this system of record. Today it’s relatively uncommon to see a data analyst as part of the L&D or training team. That probably needs to change over the next five to ten years. We would expect to see folks on the team that are well-versed in these techniques of analyzing and querying data that have the right systems at their disposal, to then deliver the third piece, which is that actual analysis and data-driven decision-making on the right-hand side.
What we’re trying to talk about today at high-level terms, but always have to be delved down to more detail with you, is an evolution to the art of education. 20 years ago, education meant standing up in front of a class and delivering
(16:00)
the lecture and so on and we’ve evolved considerably since then, but we needed to go beyond what it is in term of not using this data because when we can access the data, and we can access these questions and answers, we can actually get back to personalizing that education interaction at scale, which is what many of us are trying to do today.
I think the key point here is that I don’t know your questions, you will have the questions that are specific to you, your challenges, your organization, your initiatives, and so on, but we do know how to ask them. So, if you have questions as to how you’re gonna figure that out within your organization, this is what we do, this is what we think about a lot, and we’d love to talk to you about that. I guess there’s a time for Q&A. If there any questions?
Mason: Definitely. One to kind of start – what are the attributes of training beyond measurable, reportable, repeatable, relatable, or to ROI?
John: Yeah. There’s a lot of different attributes that can be considered here. For us, what we believe is the starting point is that system of record and that ability to query intensively in a short feedback loop to ask what those questions are. That’s key piece number one.
The second piece is that integration focus, which is how integrated are your systems with the rest of your organization?
And lastly, is just making use of that information. A lot of times we see training teams that are just so busy, they can’t actually spend time to engage and reflect with what’s going on within their environment. It’s those
(18:00)
three things that we really look for.
Mason: Another question in terms of data, and using it to drive decision-making. And I know this is a hard question because it is individual based on organizations. What kind of metrics, generally speaking, should organizations be in tracking, keeping tabs on, looking at, that help drive at this decision-making?
John: It is an open-ended question. I think it’s one that will change depending on the organization, depending on the goals. Obviously, if you are out there trying to sell training or make money from your training, the questions and the analytics that you’d be tracking would be different than if you’re focused more on outcomes.
A lot of folks that we talk with are doing both; they’ll have the outcome-based training that in education they are trying to do, and they also have the profit-based training. It really depends on what the goal of the organization is. What I can say is that, by having the data that you’re collecting accessible, that is the most important thing because the questions will change as you get the answers back.
I know that sounds a little bit like I’m avoiding the question, but the reality is if you have a goal or specific initiative, one example is the example of is the example provided earlier which is we want to keep our renewals and our subscriptions active, we don’t want to see customer’s cancel. The way that you interrogate your training data to solve for that question or that goal
(20:00)
is a lot different than if you’re just asking students or learners how they felt about a particular piece of learning. You’re gonna look for things like engagement, you’ll look for things like repeat engagement, you’re gonna look for things like additional interaction with content, and so on, that might pertain to new releases of functionality that you have just done. It’s really just about having that clear goal to find at an organizational level and then aligning your metrics and your reporting to answer the question of how do we achieve that goal on the best possible way.
Mason: I think that’s important to consider. And to your point, it is open-ended, I don’t think you are avoiding the question at all. I think that’s a question that’s based on situation and based on goal, and that’s how you kind of have to answer without, maybe that example of what are we trying to address it’s very hard to give specifics.
Well, you have another question – we’re just getting started with building a data structure for our program. What advice would you give to an organization that is brand new to this thinking, and what data points should we pursue first to build a good foundation?
John: Basically, in any organization today is already going to have a collective of systems and a tech stack out there at play, and it might be simple, it might be complex. I think the idea is to find a way to articulate what it is that you wanna be tracking on a day-to-day and activity basis, and then how to relate that information to this other systems and the rest of the tech stack in your organization.
This is all pretty technical stuff. You might be sitting there thinking, this seems very out with the realm of
(22:00)
L&D’s remit in terms of I just wanna deliver great education or training to my stakeholders. But, I would just mention that this is where we’re at L&D and training today is very similar to where marketing was 10 to 15 years ago before the marketing automation system came about. That is, you had folks that were focused mostly on content, mostly on creative, and they didn’t have great systems to demonstrate ROI. With the advent of the marketing automation platform, what that gave marketing teams was an ability to begin tracking and thinking through what it was that they’re doing and the investments they’re making a systematic way, and that has really transformed the marketing to what it is today.
We’re on a similar curve with L&D, and that is, we got a lot more data and a lot more activity metrics and so on, but more and more particularly when the economy is kind of going through ups and downs like it’s right now today, leadership is gonna be asking for a very, very clear impact analysis on what training is doing and where we’re finding the positive impacts. That can only be done if you are really looking beyond the activity metrics and focused more on, for example, a business problem like how do we get our customers to renew, or how do we know if they’re not going to renew and prevent that.
Mason: Alright. One question from the audience – many organizations have online training and other learning systems in place that do not give enough time to the new employee to absorb the content, due to constraints, and so on. Do you have any solutions for that?
John: In terms of supportive absorption and retention
(24:00)
I think one of the things that we like to talk about is, you have an employee that gets onboarded, it’s a pretty common use-case, they blitz through a whole bunch of stuff, and then three months later they’re like, “I don’t really remember any of that.”
What you can do, and sometimes really powerful is feed learning on a cadence or a path that meets the individual where they are in their journey through the organization. A lot of us have heard of learning paths, and so on, it’s a really powerful feature. I think the other thing that you can do is, again, work with that person or that individual’s team or management to understand where they are on their journey within your organization. Something that’s very popular with our customers is the idea that managers can request training on behalf of their teams, and so, they’ll go in, and they’re kind of take a look at the catalog and so on, and assign their teams to the training that makes sense at specific times that work well for them.
That’s great because the training team can focus on the content and delivery and so on, but there’s that kind of avenue where management can be involved in solving the problem of retention and making sure that the employee that they’re closest to is getting well-served.
Mason: How do you take quantifiable data and get qualitative information out of it that can then become actionable and insightful to our training strategy in our greater business strategy?
John: I have a saying that I don’t know where I can found this, I’ve heard it before, but we use a lot internally here in Administrate when we’re kind of trying to figure out our own solutions to a business problem,
(26:00)
and that is, “You wanna torture the numbers until they confess, right?” That’s a bit of a silly thing to say, but it’s important, right? Which is, you’ve got to be able to pull out those data points and then look at them, and try to understand, begin crafting the story in your head and what are the numbers telling me, and then run the numbers again and try to disprove your theory.
It’s kind of like a scientific method and keep combing through and keep figuring it out. It’s an iterative approach, understanding what that story is like and getting the collaboration around the data. Often we’ll pull reports and I’ll say, “Hmm… I wonder if this my world view,” and we’ll discuss with the team, and a person will say, “Well, if that’s true, you’d expect to see this within the data.” And so, we’ll run the report again. And we’ll look and see if that’s true, and sometimes it is and sometimes it isn’t, and I think it’s that idea which is your bringing your organization and your team around to focus on the numbers and what you’re seeing there as opposed to just crafting the story that’s in isolation, that’s not supported by anything. That breeds confidence, that breeds transparency, and it also breeds accuracy in your decision-making and that’s really what we’re talking about, I think.
Finally, we have been talking a lot with customers around the world about their response to COVID-19 in kind of a podcast radio
(28:00)
series that’s on our blog, so check that out, it’s free. It’s been really, really interesting and fun to do.
And if you are suffering from COVID-19, and your operations are being impacted; we do have a way to get your classroom-based training online in a matter of days. If that’s something that you’re struggling with and we particularly in countries like Italy and so on, better little bit further ahead on this issue, we’ve had some really great success there, and we’d love to help you out if that’s a challenge for you.
