Cool research: happy sheets hopeless, training failed…..failure-led, spaced practice worked

This astonishing piece of research by Daniel Bilton & Charles Gluck sums up almost everything I believe about good and bad training.

Over nine months, 500 people in Booz Allen were initially given three types of training:

1.       Placebo
2.       Page-turning
3.       Interactive

All three groups were then given surprise:

Three simulated phishing emails with remedial help if they failed i.e. spaced practice, learn through failure exercises.

Results

Both trained groups seemed to know what to do if they received ‘phishing’ emails:

·         87.8% of static trained

·         95.6% of interactively trained

Happy sheet evaluations for both were through the roof.

However….

Based on actual simulated attacks, they discovered no significant difference between training and no training!

WTF!

They then implemented ‘Failure-Triggered Training’, like the “Secret Shoppers” used by the retailers. Phishing emails simply dropped into your in-box, three different phishing emails on spaced intervals. Each user’s response/action was tracked. What they found was that these simulated emails resulted in a huge learning effect as error rates plummeted. The authors concluded that two main factors were at work:

1. Spaced practice

The researchers attributes their success to the spaced practice approach where the simulated emails put them to the test and with each of the three iterations more and more trainees became competent at dealing with these dangerous, fiscal requests.

2. Learn by doing

The researchers also saw gains from learning at the point of realisation, by doing something relevant at that moment, not on some disassociated training course. It was the failure-triggered training, delivered on teh back of unannounced, blind exercises, combined with immediate tailored remedial training, provided only to the users that “fail” the exercises, that did the trick.

Conclusion

So happy sheets were hopeless, straight text based and interactive e-learning was not significantly better than a placebo BUT spaced practice delivered as learn by doing worked magnificently well. This testimony from a learner about sums it up:

“I learned about the CIRT team through the phishing training email sent out a couple months back. It really stuck with me, since I ‘failed the test.’ ”