How to assess eLearning effectiveness?

By Andres Cabezas
Account Manager

Have you wondered how you can assess e-Learning effectiveness? Whether investments are supporting business results or not?

Based on e-Learning advantages such as scalability, consistency, cost reduction, time reduction, improved productivity, profitability impact, and others, it is essential for training managers to know how to assess e-Learning effectiveness to prove that it is indeed a value-added project and thus guarantee a successful e-Learning strategy throughout the corporation.

There are several approaches to assess training delivered through classroom instruction or e-Learning. After studying several authors on this topic, an article by Judith Strother published in the International Review of Research in Open and Distance Learning offers quite a comprehensive approach on this topic since it is based not only on the classical Kirkpatrick model, but it also adds a further assessment level related to the Return on Investment or ROI. Therefore, we will use her study as the main benchmark.

“Hall and LeCavalier summarized some firms’ economic savings as a result of converting their traditional training delivery methods to e-Learning: IBM saved US $200 million in 1999 providing five times the learning at one third the cost of their previous methods. Using a blend of Web-based (80%) and classroom (20%) instruction, Ernst & Young reduced training costs by 35% while improving consistency and scalability. Rockwell Collins reduced training expenditures by 40% with only a 25% conversion rate to Web-based training. Many other success stories exist. However it is also true that some firms that have spent large amounts of money on new e-Learning efforts have not received the desired economic advantages”. www.icp.org

A classical model for those working in the corporate training business is the one developed by Donald Kirkpatrick many years ago and that makes it possible today to assess both traditional training delivered by an instructor in a classroom, as well as e-Learning that can be delivered using the Internet, Intranet or CD- ROM. This model is composed of four progressive levels:

Levels Steps Follow-up
Level 1 Reaction How do learners feel?
Level 2 Learning What did they learn?
Level 3 Behavior Are they transferring the learning to their job?
Level 4 Results How much does the training affect the company’s bottom line?

According to Strother, there is an added level (level V) related to the Return on Investment or ROI, which should include a cost-benefit analysis. To this regard, she mentions that using the evaluation data, the results are converted into monetary values and then compared with the cost of the training program in Level V. Let’s analyze these levels in detail.

Level 1. How do learners feel?

In this first level, students evaluate the training at the end of the program. The main objective of this measurement is to find valuable information and not only to find out whether the individual liked the course or not. He/she could be asked whether the course met his/her expectations, if he/she found enough exercises, and whether he/she liked how the topic was handled, etc. It is difficult for you to measure the learning at this point; what you can find out is simply the students’ reaction. Some firms carry out online surveys if it is Web-based e-Learning, or they include surveys at the end of the courses, and ultimately they hand out paper surveys to users.

Level II. Learning results

This is the level to measure learning results: whether students absorbed knowledge, skills, and attitudes as stated by the learning objectives. To achieve this, in her books Kirkpatrick recommends a pretest and a posttest to compare results and observe the first training effect. For researchers on the topic, this kind of evaluation is not as common as Level I, but it is still used. Regarding the information resulting from this evaluation, Strother states that “while some studies show greater benefits in favor of face-to-face delivery, research results consistently demonstrate superior benefits of e-Learning in general.”

Level III. Workplace behavior

When a Human Resources Manager or training director chooses an e-Learning program, one of the most important questions to ask is whether the use of this kind of training will help the trainee transfer the knowledge and skills to his/her daily job. To this regard, the Kirkpatrick model seeks to find out if student behavior changed as a result of the learning experience.

In her bibliography Kirkpatrick recommends the evaluation to be done a few months after the training so students can apply all their knowledge in their workplace. According to researchers, the most common assessment tools at this level are observation surveys or student, supervisor, or even customer surveys. The bottom line is to prove a behavioral change resulting from the training.

Some sample questions to assess a training program aimed at customer service could include:

Did the agent use proper non-verbal communication when greeting the customer?

Did the agent offer the customer this week’s promotion?

Did the agent respond properly to customer complaints?

Level IV. Business results

The last level from the Kirkpatrick model is to assess the results of training as it directly affects a company’s bottom line – a challenging task for trainers. According to Kirkpatrick, the number of variables and complicating factors make it difficult to evaluate the direct impact of training on a business’ bottom line. This is just as true for e-Learning as for traditional training programs. However, depending on the kind of course, some variables could be measured. For instance, if the sales force is trained, firms can measure the change in the sales volume and a reduction in the sales cycle or sales profitability. If the training is related to the use of a given information system, results can be measured in the reduction of the number of calls to the technical support staff or reduced time to complete reports or transactions.

Level V. ROI

The Return on Investment calculation is a level added by Strother in her article to the classical Kirkpatrick model. As stated by Strother, “Using Level IV evaluation data, the results are converted into monetary values and then compared with the cost of the training program to obtain a return on investment.”

ROI studies have shown a positive return for companies implementing e-Learning even though most studies show a positive return based only on cost reduction. It is also essential for ROI studies to include a benefit analysis. Therefore, cost reduction and improved benefits could be identified.

Conclusion

Evaluating e-Learning training program effectiveness is important for corporate training managers because it makes program scope, benefit, and improvement analysis possible. The Kirkpatrick model is a systematic way of measuring the effectiveness of any training composed of four progressive levels: Reaction, Learning, Behavior, and Results. In her article Strother concludes, “Until a more solid research methodology is developed for measuring e-Learning results, we can rely on the mainly qualitative feedback from corporations that are using e-Learning to deliver their training. Firms praise online training as a cost-effective, convenient, and effective way to deliver corporate education. Early studies seem to demonstrate that e-language-learning in business is a win-win proposition for all – the learner, the corporation, and the customers served by the corporation.”