How to evaluate training

There is no way of getting away from the full extent of the 4-level system.

It is the only complete and effective way to evaluate.  Only this system answers the questions:

1. Has our money been well-spent?

2. Was the training worth while?

3. Did it have any impact on the workplace?

4. Have we had a positive return on the investment we made?

5. What is this return?

You will never get answers to these questions from a course reaction sheet.  Course reaction (Level 1) is no evaluation at all.

What the four level system requires you to do

Here are the things you have to do before you even start training.

The cost of tracing, databases and evaluation must always be included as a standard cost of training.

Training costs are NOT just the cost of the trainer.

Training  costs:

1. Needs analysis

2. Impact assessment before training

3. ROI design

4. Development

5. Training materials

6. Quality assurance

7. Accreditation

8. Training materials and conduct.

9. Down time while learners are taken off the job.

10. Training accommodation and meals

11. Assessment and moderation

12. Database and tracing

13. Impact assessment after training

14. ROI calculation and evaluation report

Training quotations that do not include all these elements are suspect, unless they are specifically not required by the client or the funder.

Contracts with learners

If the learners are already employed, and the training is at the expense of the employer, then there is no need for any special contract or tracing system

If however the learner is at a school or training institution, is freelance or not yet employed then the learner must sign an undertaking to disclose employment particulars including remuneration for a fill year after the end of the training.  This includes learners who pay their own fees.

If the training is funded, then this undertaking must be signed as a condition of undergoing funded training.

Course reaction Sheets.

Design reaction sheets that gather information about what the learner’s think they gained from the course.  They must be made to assess what they knew before the course and assess what they can now do.  The course must be interactive enough to allow this to happen.  They must be able to make this assessment on practical exercises and experiences during the course and other formative assessment instruments that they experienced.  The reaction must be made on the basis of what they experienced, and not on how comfortable they felt and what they thought of the food.

Assessment guides.

Assessment guides, fully compliant with SAQA and OETD compliant, must be designed.  They must ne administered by the accredited assessors and moderated by accredited moderators.  The learners can be assessed only as “competent” or “not yet competent”.

Impact instruments

Impact cannot be calculated earlier than three months after training.

There are only four dimensions to workplace impact:

The quality signifiers you agree on for measuring quality before training must be same as those used after training.

ROI calculation

There is only one way to calculate return on investment:

[(monetary benefits - cost of the training) ÷ cost of the training] x 100

Production and productivity

The monetary benefits can be anything.  In the case of sales they are easy.  It is simply value of sales after training, less value of sales before training.

In the case of manufacturing it is just as easy.

Production

In the case of contribution to a TV production, it is not as easy.  But it can and MUST be done.

The training manager and the client, (or employer), decide on dimensions that they will use.  You can even use opinion questionnaires, where you gauge the productivity from the both supervisors and peers.

Management

In the case of commissioning editors, it can be done on the value of content commissioned over a fixed period after training less that before training.  This must also be weighted against no of errors (the top score being “no error”.)

In the case of managers, it can be done on the basis of “cost of errors before training compared to cost of errors after training.  What is the cost of the errors?  It doesn’t matter, as long as you use the same formula to calculate before training as after.

However you agree to do it, you must reach agreement before the training is contracted, and you must not change the ROI calculation criteria.  You can make it as simple as you like, using fixed costs or as complex as you like (using the Repertory Grid Technique). As long as you do it in the way you agreed to do it at the beginning.

Funded training

This is the easiest of the lot.

Supposing the learner was earning R5 000 a month prior to training.  After six months, his earning jumped to R10 000 a month. For the whole of the second year he earned R15 000 a month.

Thus the total earnings after training were:

If we take an agreed estimate that this person will pay an average of 18% in tax, then the total tax paid after training is R270 000 X 18% = R48 600.

Supposing the government paid R25 000 towards the training, then the return on investment to the government after two years is 194%.  Sounds like a bargain to me!