Masterplan now available onAndroidiPhone
Back to news
Insights
9th January 2023

Measuring Learning Success With the LTEM-Framework

Masterplan TeamMasterplan Team
Measuring Learning Success With the LTEM-Framework

Ideally, further training measures lead to an increase in employee competence. But how can you measure learning effects in concrete terms? The Learning Transfer Evaluation Model helps you to determine the success of training.

Every training measure results in monetarily measurable effects that can be mapped with the ROI formula. Concrete learning effects, such as an increase in employee competence, an improved mood among the workforce, or the competitiveness of the company, on the other hand, are difficult or impossible to quantify in monetary terms.

Nevertheless, you can measure them in your company – and you should!

After all, this can provide crucial insights into whether and to what extent the training measures are contributing to the company's strategic goals.

But how and where do you start? The Learning Transfer Evaluation Model by Will Thalheimer from 2018 provides orientation and an overview. It is a further development of the Kirkpatrick-Katzell model from the 1950s.

Kirkpatrick-Katzell-Model
The Kirkpatrick-Katzell model was developed in the 1950s by Donald Kirkpatrick and James Katzell. It is one of the early models for evaluating training programs and has since been used by many organizations and companies.

Jack Phillips eventually added a fifth layer to this model in the 1980s: return on investment (ROI).

>>>Download Free Handbook "Return on Learning"<<<

Measuring learning success with the Learning Transfer Evaluation Model (LTEM)

The Learning Transfer Evaluation Model (LTEM) provides you with a helpful overview of which metrics, quantitative and qualitative evaluation concepts you can use in different phases of the learning process. It helps you to obtain well-founded conclusions about the return on learning in terms of the learning effects for employees.

The Learning Transfer Evaluation Model (LTEM)
Learning Transfer Evaluation Model (LTEM) Overview
  • Origin: Origin: Developed by Will Thalheimer, PhD et al. 2018.
  • Benefits: The framework enables organizations to ensure that their evaluation methods are effective and provide valid feedback.
  • History: The LTEM is a further development of Kirkpatrick-Katzell's four-stage model from the 1950s for measuring learning effects.
  • Functionality: The LTEM divides the learning process into eight stages and identifies the relevant assessment aspects for learning success at each stage. The model is designed to be relevant to all learning methods, including classroom learning, e-learning, mobile learning, workplace learning, and self-study learning.

Thalheimer divides the learning process into a total of eight phases. He has created separate assessment categories for each phase. Similar to the Kirkpatrick-Katzell model, the measurement of success becomes more complex with each level and the significance with regard to the learning effects increases. 

Compared to the Kirkpatrick-Katzell model, however, the LTEM represents an even more detailed and comprehensive method for measuring the effectiveness of training.

How to measure learning effects in eight phases

The first four phases relate to the learners' interaction with the learning materials. Measurements are not so much used to assess learning, but above all provide important information on motivation, interests, and the quality of the learning program. 

It is only from phase 5 that measurements can identify competence gains and provide well-founded conclusions about the effectiveness of learning. For the first five levels, learning platforms can provide helpful data; for the last three, the L&D teams themselves are required to collect data through qualitative interviews, quantitative surveys, or observations.

PhaseDimensionDescriptionMeasurement of success
1.ParticipationThe learner enrolls, starts, participates in, or completes a learning opportunity.
  • Course completion rate
  • Average learning time
2.EngagementThe learner participates in activities related to learning.
  • Time spent on a text lesson
  • Are the videos fast-forwarded or watched in full?
  • Are questions asked? (in person)
3.EvaluationThe learner evaluates:
  • a) whether the course was found helpful (learning effectiveness),
  • b) whether the course was liked (not an indicator of effectiveness)
  • 5-star or 10-point rating
4.Knowledge acquisitionThe learner answers content-related questions (knowledge quiz)
  • Multiple choice test or open questions with a free text field
5.Decision-making competenceThe learner applies acquired knowledge in realistic scenarios to make decisions
  • Multiple choice test or open questions with a free text field
6.Task competenceThe learner solves realistic tasks
  • Homework or case work related to the course
7.TransferThe learner applies the acquired competencies in their work routine
  • Observation by supervisor, colleagues, or 360-degree feedback
  • Self-assessment
  • Quantitative survey on the learning program or as part of the regular employee survey
8.Transfer effectThe application of the new competencies has a positive effect on the learner's work or the company's environment
  • Observation by supervisor, colleagues, or 360-degree feedback
  • Self-assessment
  • Quantitative survey on the learning program or as part of the regular employee survey

1. Attendance

Although mere participation in learning opportunities does not yet provide any insights into actual learning success, it is an critical indicator of "learning appetite". This is particularly relevant when learning is not a compulsory task (as in the case of compulsory training on data protection or compliance), but takes place on a voluntary basis.

Masterplan's Solution
  • The average completion rate of Masterplan courses among all users and all courses is over 80%.

2. Engagement

The engagement phase examines how attentively learners participate in training. In the seminar room, instructors can assess whether learners follow the training attentively (and ask questions). In online learning, alternative indicators can be whether learners skip parts of lessons, click on links from lessons, or download secondary materials offered.

Masterplan's Solution
  • Text lessons from Masterplan are actively read by the vast majority (more than 70%). Only in 11 % of all lessons used were parts of the learning videos skipped (skipped).

3. Learners Perception

How do learners evaluate the content? First, surveys can determine whether they liked the course in principle and whether they would recommend it to others.

However, Thalheimer points out that well-rated courses are not automatically courses that actually help learners in their day-to-day work. Secondly, learners should therefore be asked whether they were able to follow the content of the course and whether they believe the content will help them in their everyday working lives.

Online learning platforms can continuously evaluate perceptions and then suggest relevant follow-up courses to learners.

Masterplan's Solution
  • The average rating of all master plan courses is 7.79/10.
  • Masterplan continuously evaluates course ratings and incorporates this data into the recommendation algorithm: First, learners are shown the top-rated courses in the dashboard. Furthermore, individual courses are suggested in the "Recommended for you" section based on usage and rating behavior. In addition, Masterplan analyzes particularly well and poorly rated courses (including comments) on a quarterly basis in order to further optimize the content offering.

4. Knowledge of Facts

Can knowledge and terminologies be retrieved after the course and are they still present after a longer period of time? Knowledge quizzes are used to check knowledge retention. As a rule, they achieve the best learning effects when the questions are neither too difficult nor too easy (more on the Yerkes-Dodsen-Effect in our Trend-Report). Compared to classroom training, learning platforms have the advantage that courses and knowledge queries can be repeated for refreshing after a longer period of time, as they can be accessed flexibly from anywhere and at any time.

Masterplan's Solution
  • Each Masterplan lesson contains a knowledge query mostly with changing question types.
  • Masterplan allows saving courses and automatically lists all completed courses in a separate learning area. Learners can then more easily review the knowledge and associated quizzes.
  • Masterplan's lessons are short and can be accessed on any device. Courses can be saved in the app and retrieved even when no Internet is available. This increases the likelihood that knowledge will be refreshed.

5. Decision Making Competence

In contrast to reciting knowledge, decision-making competence presupposes that what has been learned has also been understood and that a transfer to new, unknown situations and scenarios is possible. 

For Thalheimer, the review of decision-making competence is particularly relevant: Whether or not training actually has an effect in practice is shown by the extent to which learners actually make different decisions in everyday work than they did before.

Masterplan's Solution
  • Many Masterplan lessons contain so-called scenario questions in the subsequent quiz, the answers to which are not derived directly from the learning content but require a transfer of knowledge. This means that learners do not merely repeat definitions and terms, but are required to apply the learning content to new situations.

6. Task Competence

Task competence shows the extent to which learners are not only able to make a specific decision correctly, but also to lead independently through an entire scenario. Thalheimer explains this using the example of leadership training: It is no longer just a question of whether or not a supervisor involves her team in a strategic decision. Rather, the decisive factor is how the manager arrives at this decision and how he or she implements it.

In order to determine task competence, colleagues, mentors, lecturers, or supervisors are needed who can meaningfully evaluate the qualitative response to the tasks, for example on the basis of homework.

Masterplan's Solution
  • Some Masterplan lessons include homework assignments and case studies.

7. Transfer

Transfer means that learners can apply their knowledge in a real-world context. Thalheimer points out that this can involve the work environment, but also areas of application outside of work. Companies should consider both forms of knowledge transfer. If learners usefully apply their acquired knowledge outside of work, this will lead to greater appreciation of the employer and higher employee retention.

The application of learned knowledge happens per se outside a learning platform and can be determined through observations, self-assessments and qualitative surveys.

Example from E.ON (Masterplan customer)
  • Employees have a personal development plan, coupled with annual target discussions (e.g., target: "Become a Scrum Master").
  • Interim meetings to agree on goals and regular tasks serve to check whether the learning content, e.g. from the Scrum Master training, can be applied in practice.

8. Transfer Effect

The transfer effect examines the impact of a change in behavior. An example: A manager implements a new approach to feedback culture in the team. The knowledge transfer takes place, the new method is applied.

What positive or negative (economic or social) effects this leads to in the team or company can be recorded in observations, in discussions between managers and employees, or in surveys.

This in turn also allows conclusions to be drawn about the quality of the learning content – an aspect that should not be underestimated. Finally, poor-quality learning content can also lead to no or negative effects, whether through opportunity costs or a deterioration in the feedback culture as a result of the methods learned.

LTEM in Practice: How to Use the Learning Transfer Evaluation Model

The LTEM is a roadmap to find out where HR professionals stand in terms of learning evaluation and to identify missing phases where evaluations make strategic sense.

In a sense, it serves as a map for your training landscape and can provide guidance to help you find the right evaluation methods in places that are relevant to your organization. E-learning providers, such as Masterplan, already build assessments and evaluations into their learning solutions (especially in LTEM levels 1 to 4) and thus provide L&D managers with tailored reports with important insights into the learning behavior of their employees.


Read our interview with Digital Learning Expert Anke Wygold from Masterplan client E.ON and learn how the group systematically identifies aspects such as learning time, learner evaluation of learning opportunities, and employee learning transfer.


Learning providers also benefit from this and can continuously improve their offering on the basis of this. In this way, the foundation for learning success is laid.

In-depth evaluation methods in phases 6 to 8 are particularly suitable for determining the return on learning for strategically critical, cost-intensive learning measures that are deployed over many years.

Systematic education controlling can ensure that the measure actually pays off in terms of strategic goals.

Download Handbook "Return on Learning"

More Blog Posts