Monday, January 21, 2013

Cool research: happy sheets hopeless, training failed…..failure-led, spaced practice worked

This astonishing piece of research by Daniel Bilton & Charles Gluck sums up almost everything I believe about good and bad training.
Over nine months, 500 people in Booz Allen were initially given three types of training:
1.       Placebo
2.       Page-turning
3.       Interactive
All three groups were then given surprise:
Three simulated phishing emails with remedial help if they failed i.e. spaced practice, learn through failure exercises.
Results
Both trained groups seemed to know what to do if they received ‘phishing’ emails:
·         87.8% of static trained
·         95.6% of interactively trained
Happy sheet evaluations for both were through the roof.
However….
Based on actual simulated attacks, they discovered no significant difference between training and no training!
WTF!
They then implemented ‘Failure-Triggered Training’, like the “Secret Shoppers” used by the retailers. Phishing emails simply dropped into your in-box, three different phishing emails on spaced intervals. Each user’s response/action was tracked. What they found was that these simulated emails resulted in a huge learning effect as error rates plummeted. The authors concluded that two main factors were at work:
1. Spaced practice
The researchers attributes their success to the spaced practice approach where the simulated emails put them to the test and with each of the three iterations more and more trainees became competent at dealing with these dangerous, fiscal requests.
2. Learn by doing
The researchers also saw gains from learning at the point of realisation, by doing something relevant at that moment, not on some disassociated training course. It was the failure-triggered training, delivered on teh back of unannounced, blind exercises, combined with immediate tailored remedial training, provided only to the users that “fail” the exercises, that did the trick.
Conclusion
So happy sheets were hopeless, straight text based and interactive e-learning was not significantly better than a placebo BUT spaced practice delivered as learn by doing worked magnificently well. This testimony from a learner about sums it up:
I learned about the CIRT team through the phishing training email sent out a couple months back. It really stuck with me, since I ‘failed the test.’ 

3 comments:

Lars Hyland said...

This is a great piece of research, Donald, and exemplifies the power of the spacing effect coupled with a considered learning design. Harnessing these benefits in a way that is scalable and cost effective can be difficult and time consuming, which led me to develop the Retenda system. While revisiting the design of existing training is often advisable, identifying key learning points and using the spacing effect to reinforce these post delivery can have a positive effect on long term retention and transfer. Training companies and corporate L & D departments that offer open training courses can quickly add value and demonstrate improved business benefit. Further and Higher Education providers can improve student results through personalised spaced revision support.

I've found that there is still so much time and energy expended in delivering "learning experiences" which simply don't stick. Having a simple to use system that is easy to adopt and can yield improvements to long term retention with minimal administrative overhead helps us all on the journey towards more effective learning design.

Unknown said...

Great article and the research results are truly amazing! How does a small company (20 people) marshal the resources needed to build the secret shopper army? It seems like it would take a large investment of secret shoppers to give experiences that are meaningful and feedback that is timely.

Pete said...

Another argument for in situ medical simulation!