Measuring Learning Success With the LTEM-Framework

Ideally, further training measures lead to an increase in employee competence. But how can you measure learning effects in concrete terms? The Learning Transfer Evaluation Model helps you to determine the success of training.

Every training measure results in monetarily measurable effects that can be mapped with the ROI formula. Concrete learning effects, such as an increase in employee competence, an improved mood among the workforce, or the competitiveness of the company, on the other hand, are difficult or impossible to quantify in monetary terms.

Nevertheless, you can measure them in your company – and you should!

After all, this can provide crucial insights into whether and to what extent the training measures are contributing to the company's strategic goals.

But how and where do you start? The Learning Transfer Evaluation Model by Will Thalheimer from 2018 provides orientation and an overview. It is a further development of the Kirkpatrick-Katzell model from the 1950s.

>>>Download Free Handbook "Return on Learning"<<<

Measuring learning success with the Learning Transfer Evaluation Model (LTEM)

The Learning Transfer Evaluation Model (LTEM) provides you with a helpful overview of which metrics, quantitative and qualitative evaluation concepts you can use in different phases of the learning process. It helps you to obtain well-founded conclusions about the return on learning in terms of the learning effects for employees.

The Learning Transfer Evaluation Model (LTEM)

Origin: Origin: Developed by Will Thalheimer, PhD et al. 2018. Benefits: The framework enables organizations to ensure that their evaluation methods are effective and provide valid feedback. History: The LTEM is a further development of Kirkpatrick-Katzell's four-stage model from the 1950s for measuring learning effects. Functionality: The LTEM divides the learning process into eight stages and identifies the relevant assessment aspects for learning success at each stage. The model is designed to be relevant to all learning methods, including classroom learning, e-learning, mobile learning, workplace learning, and self-study learning.

Thalheimer divides the learning process into a total of eight phases. He has created separate assessment categories for each phase. Similar to the Kirkpatrick-Katzell model, the measurement of success becomes more complex with each level and the significance with regard to the learning effects increases. 

Compared to the Kirkpatrick-Katzell model, however, the LTEM represents an even more detailed and comprehensive method for measuring the effectiveness of training.

How to measure learning effects in eight phases

The first four phases relate to the learners' interaction with the learning materials. Measurements are not so much used to assess learning, but above all provide important information on motivation, interests, and the quality of the learning program. 

It is only from phase 5 that measurements can identify competence gains and provide well-founded conclusions about the effectiveness of learning. For the first five levels, learning platforms can provide helpful data; for the last three, the L&D teams themselves are required to collect data through qualitative interviews, quantitative surveys, or observations.

1. Attendance

Although mere participation in learning opportunities does not yet provide any insights into actual learning success, it is an critical indicator of "learning appetite". This is particularly relevant when learning is not a compulsory task (as in the case of compulsory training on data protection or compliance), but takes place on a voluntary basis.

2. Engagement

The engagement phase examines how attentively learners participate in training. In the seminar room, instructors can assess whether learners follow the training attentively (and ask questions). In online learning, alternative indicators can be whether learners skip parts of lessons, click on links from lessons, or download secondary materials offered.

3. Learners Perception

How do learners evaluate the content? First, surveys can determine whether they liked the course in principle and whether they would recommend it to others.

However, Thalheimer points out that well-rated courses are not automatically courses that actually help learners in their day-to-day work. Secondly, learners should therefore be asked whether they were able to follow the content of the course and whether they believe the content will help them in their everyday working lives.

Online learning platforms can continuously evaluate perceptions and then suggest relevant follow-up courses to learners.

4. Knowledge of Facts

Can knowledge and terminologies be retrieved after the course and are they still present after a longer period of time? Knowledge quizzes are used to check knowledge retention. As a rule, they achieve the best learning effects when the questions are neither too difficult nor too easy (more on the Yerkes-Dodsen-Effect in our Trend-Report). Compared to classroom training, learning platforms have the advantage that courses and knowledge queries can be repeated for refreshing after a longer period of time, as they can be accessed flexibly from anywhere and at any time.

5. Decision Making Competence

In contrast to reciting knowledge, decision-making competence presupposes that what has been learned has also been understood and that a transfer to new, unknown situations and scenarios is possible. 

For Thalheimer, the review of decision-making competence is particularly relevant: Whether or not training actually has an effect in practice is shown by the extent to which learners actually make different decisions in everyday work than they did before.

6. Task Competence

Task competence shows the extent to which learners are not only able to make a specific decision correctly, but also to lead independently through an entire scenario. Thalheimer explains this using the example of leadership training: It is no longer just a question of whether or not a supervisor involves her team in a strategic decision. Rather, the decisive factor is how the manager arrives at this decision and how he or she implements it.

In order to determine task competence, colleagues, mentors, lecturers, or supervisors are needed who can meaningfully evaluate the qualitative response to the tasks, for example on the basis of homework.

7. Transfer

Transfer means that learners can apply their knowledge in a real-world context. Thalheimer points out that this can involve the work environment, but also areas of application outside of work. Companies should consider both forms of knowledge transfer. If learners usefully apply their acquired knowledge outside of work, this will lead to greater appreciation of the employer and higher employee retention.

The application of learned knowledge happens per se outside a learning platform and can be determined through observations, self-assessments and qualitative surveys.

8. Transfer Effect

The transfer effect examines the impact of a change in behavior. An example: A manager implements a new approach to feedback culture in the team. The knowledge transfer takes place, the new method is applied.

What positive or negative (economic or social) effects this leads to in the team or company can be recorded in observations, in discussions between managers and employees, or in surveys.

This in turn also allows conclusions to be drawn about the quality of the learning content – an aspect that should not be underestimated. Finally, poor-quality learning content can also lead to no or negative effects, whether through opportunity costs or a deterioration in the feedback culture as a result of the methods learned.

LTEM in Practice: How to Use the Learning Transfer Evaluation Model

The LTEM is a roadmap to find out where HR professionals stand in terms of learning evaluation and to identify missing phases where evaluations make strategic sense.

In a sense, it serves as a map for your training landscape and can provide guidance to help you find the right evaluation methods in places that are relevant to your organization. E-learning providers, such as Masterplan, already build assessments and evaluations into their learning solutions (especially in LTEM levels 1 to 4) and thus provide L&D managers with tailored reports with important insights into the learning behavior of their employees.

Read our interview with Digital Learning Expert Anke Wygold from Masterplan client E.ON and learn how the group systematically identifies aspects such as learning time, learner evaluation of learning opportunities, and employee learning transfer.

Learning providers also benefit from this and can continuously improve their offering on the basis of this. In this way, the foundation for learning success is laid.

In-depth evaluation methods in phases 6 to 8 are particularly suitable for determining the return on learning for strategically critical, cost-intensive learning measures that are deployed over many years.

Systematic education controlling can ensure that the measure actually pays off in terms of strategic goals.

Download Handbook "Return on Learning"