Cool research: happy sheets hopeless, training failed…..failure-led, spaced practice worked

This astonishing piece of research by Daniel Bilton & Charles Gluck sums up almost everything I believe about good and bad training.

Over nine months, 500 people in Booz Allen were initially given three types of training:

1.       Placebo
2.       Page-turning
3.       Interactive

All three groups were then given surprise:

Three simulated phishing emails with remedial help if they failed i.e. spaced practice, learn through failure exercises.

Results

Both trained groups seemed to know what to do if they received ‘phishing’ emails:

·         87.8% of static trained

·         95.6% of interactively trained

Happy sheet evaluations for both were through the roof.

However….

Based on actual simulated attacks, they discovered no significant difference between training and no training!

WTF!

They then implemented ‘Failure-Triggered Training’, like the “Secret Shoppers” used by the retailers. Phishing emails simply dropped into your in-box, three different phishing emails on spaced intervals. Each user’s response/action was tracked. What they found was that these simulated emails resulted in a huge learning effect as error rates plummeted. The authors concluded that two main factors were at work:

1. Spaced practice

The researchers attributes their success to the spaced practice approach where the simulated emails put them to the test and with each of the three iterations more and more trainees became competent at dealing with these dangerous, fiscal requests.

2. Learn by doing

The researchers also saw gains from learning at the point of realisation, by doing something relevant at that moment, not on some disassociated training course. It was the failure-triggered training, delivered on teh back of unannounced, blind exercises, combined with immediate tailored remedial training, provided only to the users that “fail” the exercises, that did the trick.

Conclusion

So happy sheets were hopeless, straight text based and interactive e-learning was not significantly better than a placebo BUT spaced practice delivered as learn by doing worked magnificently well. This testimony from a learner about sums it up:

“I learned about the CIRT team through the phishing training email sent out a couple months back. It really stuck with me, since I ‘failed the test.’ ”

Krug Don’t make online learners think

Steve Krug has a background in writing computer manuals. He saw that this was not the solution to most problems and moved on into usability design. He has had a huge influence on web design through his best-selling book Don’t Make Me Think (2000). He also provided a method for quick and effective testing in his second book Rocket Surgery Made Easy (2009). The huge success of these books meant that user interface design was taken more seriously in terms of good practice and the need for testing with real users. Interface design, or UX design, has gained importance as part of general online learning design.

Interface design

Krug asks a simple question, ‘How do we really use the web?’ We glance, scan and muddle through. We don’t read pages, we scan them, choose the first reasonable option, and because we’re lazy, we meander through content. This is important and, if excesses in design are to be avoided, it has to be understood when designing web sites and online learning. His theory is based on real practice and positive results on real web sites. Krug’s first law of usability is to strive to make things self-evident or self-explanatory, hence the title ‘Don’t Make Me Think’.

Users and learners want the interface to be an unthinking act, easy and unobtrusive. The more they think, the more likely they are to stop and go elsewhere. Forget instructions. People don’t read them and don’t want them. He recommends that we design out the need for a tutorial or instructions.

Design recommendations

Sensitive to the needs of the internet as a medium in itself, he emphasises the importance of the Home page. This leads to reflection on the importance of the ‘Big Picture’, namely the essential purpose of the site or online learning programme. He recommends tag lines that capture the essence of a site or web experience. Mission statements he hates, as they rarely tell you the real story and usually miss the Big Picture. 

Navigation hierarchies

Taking his lead from newspapers, always an interesting source for screen design, he recommends carefully designed hierarchies. He hates navigation that breaks down when you get past the second level. The solution, he thinks, is persistent global navigation at the same position on every page with a home button and tracking. He loves fixed menus. He also makes the useful distinction between navigation, utilities (print, search and so on) and content. It is always a payoff between ‘wide and deep’ hierarchies. 

Be conventional

Following on from Norman and Nielsen, he stresses conventions. Don’t play fast and loose, make things easy and consistent. Use conventions, such as shopping carts, standard video controls and icons. This is sound advice. Conventions are more than just objects of convenience, they are part of the grammar of interface design. Designers often refuse to use conventions as they crave creativity and innovation – this, in his view, is rarely useful. Pages should also be broken up into carefully defined areas, clickable areas should be obvious and every attempt made to minimise ‘noise’, again a Mayer and Clark principle in online learning.

Half the number of words and half again

True to his belief that screen readers are different from readers of print, he has strong views on writing for the screen. Less is more and so he exhorts designers and writers to omit needless words. In his own words, “Half the number of words and half again”. 

Search

Krug was among the earliest evangelists for search on websites. Search is a window into the collective mind of your users. It tells you what people really want, look for and what is most likely missing.

Interface design in online learning

In interface design, as Steve Krug says in the title, the point is NOT to make people think. In learning, the whole point is to make people think. Yet many of his recommendations are applicable across learning experiences. His advice on drastically reducing text has been confirmed in research by Mayer and others in online learning design. The importance of search has also come to the fore in Learning Experience Platforms used for learning in the workflow.

Krug’s prescriptions are even more important in online learning than in web design, as learning’s great enemy is cognitive overload and dissonance. If learners have to work hard to understand, navigate and read online learning, they have less sustained attention for retentive learning. Most online learning, like most offline learning, is too long winded and needs to be seriously edited to avoid cognitive overload. Keep navigation simple and consistent, use de facto conventions, avoid deep hierarchies and write for the screen not the page. And don’t forget to test – a few iterations with experts.

Usability testing

His second book Rocket Surgery Made Easy, shows how to do modest, low budget testing. His starting point is that designers can’t see the mistakes they make as they get too close to the design and as the navigation has come from their own heads, they lack objectivity. You need other fingers and eyeballs, guided by experts, using voiced testimonies. 

Krug, like Norman and Nielsen, is a strong believer in a specific form of usability testing. Following Nielsen and Landauer he takes the view that a few good, experienced testers and a few iterations are all you need. Forget the large-scale focus groups and massive testing, which suffer from the law of diminishing returns. His practical experience shows that just one, or a few testers early on are more effective than a large number at the end. 

He recommends evidence gathering with a camcorder and facilitator who asks questions and gives tasks, especially ‘Get it’ tasks where you probe the user for their understanding of the point of the experience, how it works and how it is organised. The point of the facilitator is to probe and ask them not only what they’re looking at but what they’re thinking. Listen, keep an open mind and take lots of notes.

An underlying point, made many years before by Dewey and Heidegger is that technologies work best when they hide themselves in things and tasks. Technology is at its best when it is invisible. This is the consistent theme in all good usability theorists and practitioners. The task of the designer, to make the delivery mechanism as invisible as possible.

Krug understands the different roles of specialists in design teams and the tensions that arise between them. His solution is to objectify the debate through testing, not with the mythical average user, but with real users. His is a useful, practical and prescriptive approach to good usability through good design.

Influence

Krug’s Don’t Make Me Think has sold over 600,000 copies. Rather than depend on academic literature, this popular book, in a sense he practices what he preaches. It is readable and does exactly what the user expects, give concrete advice on design and testing. As user design has grown as a practice Krug’s work is still relevant for its lasting recommendations.

Bibliography

Krug, S., 2009. Rocket surgery made easy: The do-it-yourself guide to finding and fixing usability problems. New Riders.

Krug, S., 2000. Don't make me think!: a common sense approach to Web usability. New Riders.

Mayer & Clark – 10 brilliant design rules for e-learning

Richard Mayer and Ruth Clark are among the foremost researchers in the empirical testing of media and media mix hypotheses in online learning. Their e-Learning and the Science of Instruction (2003) covers seven design principles; multimedia, contiguity, modality, redundancy, coherence, personalisation, and practice opportunities. Clear explanations are given about the risks of ignoring these principles - with support from worked examples and case study challenges. It should be a compulsory text for online learning designers.

Media mix is not mind rich

Their precise studies have confirmed that our media mix (text, graphics, audio, animation, video) in online learning is often flawed, resulting in cognitive overload and dissonance. Perhaps their greatest contribution has been in identifying redundancy as a serious problem in screen-based learning but they are known for research that produces clear practical recommendations that do not pander to those who think that media rich automatically means mind rich.

Less is more

If you were asked to sum up the psychology of learning in three words, it would be ‘less is more’, that’s also Mayer and Clark’s mantra. In one study, Mayer, et al (1996) presented 600 pieces of scientific learning and found that briefer versions, which were concise, coherent and co-ordinated, resulted in far more effective learning. They are precise in their recommendations, ‘There is a clear pattern in which the more words added to the core verbal explanation, the more poorly the student does in producing the core explanative idea units. These results are consistent with the idea that the additional words overload verbal working memory, drawing limited attentional and comprehension resources away from the core verbal explanation.’ The lesson with text is to cut it ‘til it bleeds! Bullet points, simple writing, highlighted keywords and short paragraphs are all useful screen writing techniques.

Avoid eye candy

They are critical of gratuitous graphics which are added to simply fill slots on pages of text. This is not uncommon in e-learning where designers simply take a noun within the text and slam in an associated image. This does nothing, according to Mayer and Clark, than add cognitive load and slow up learning.

Avoid ear candy

Background music and environmental sounds create unnecessary cognitive load and distract from, rather than increase, learning. Indeed, music, over longer periods of time can be incredibly annoying. Note that this also applies to sounds, such as beeps or applause, that reinforce right and wrong answers. This may be appropriate in a games, but not for most online learning. Ear candy is as bad as eye candy.

Text and graphics good

They argue that ‘text and simple relevant graphics’ can improve learning as they use separate cognitive channels. They are not absolutist on these rules, as text within graphics can be useful when explaining a process or in labeling.

Beware of text and animation

‘Text and animation’ which both use the visual channel, cause cognitive dissonance and often confuse rather than achieve learning. Animation, like video, should use audio narration, rather than accompanying text.

Beware of text and audio

They claim that words in both text and accompanying audio narration can hurt learning. This is interesting as it is often assumed that one needs both to cover accessibility issues. In other words, they argue for using ‘audio and graphics’ without screen text. According to Clark and Mayer (2003), ‘audio or text on their own’ are better than ‘text and audio together’. This is confirmed by another study by Kalyuga, Chandler and Sweller (1999) where the group with audio scored 64% better than the group with both text and audio. They claim that one or other is redundant and will overload the visual and aural channels.

Redundancy

A review of studies around this concept, known as the redundancy effect, by Sweller et al (1998) cites a list of research studies that all point to the damage done to learning when redundant material interferes with the efficacy of the learning. For example; they illustrate a point about leaving out extraneous or distracting graphics in media with an experiment, conducted by Harp and Mayer (1997), in which students were given a text to read on lightning strikes. Students who read the passage accompanied by elaborate colour photos with additional captions - as opposed to the text with simple graphics - showed 73% less retention of knowledge and 52% fewer solutions on a transfer test.

Keep it close

Mayer (1989), Mayer Steinhoff Bower (1995) and Moreno and Mayer (1999) in five separate studies compared graphics with text close to the graphics, and graphics with text below the graphics, at the foot of the screen. In all five studies, learners who used the co-located text and graphics improved their problem solving by between 43-89%. Similar results have been found by Chandler and Sweller (1991), Pass and Van Merrienboer, (1994). Making the learner’s eye jump from one part of the screen to another is disruptive and reduces the effectiveness of the learning. E-learning has also introduced heavy doses of rollover text which is displaced away from the item over which the cursor rolls so that the pop-up text appears elsewhere on the screen at a distance from the item in question. The research confirms that this is to be avoided in learning programmes.

Personalise

Backed up by the work of Nass and Reeves at Stanford (subject of my next post), they recommend a more conversational style, using first and second person language. This is not to say that it should be over-friendly or condescending. It should feel like a dialogue, not a lecture. They also recommend the use of an on-screen coach or agent. Note that they absolutely recommend self-paced user control, as well as frequent practice and context setting through interactions.

Lessons for production

Their research explains why broadcast TV and web design companies often fail to produce good online learning. They are drawn to techniques that entertain rather than educate, often adding media that unintentionally degrades the learning experience. This is why Nielsen and others were so critical of Flash, as it encouraged, the unnecessary addition of animation. On the other hand it confirms the use of short video lessons, with images and audio, as a form of instruction.

Conclusion

Clark and Mayer were among the first to seriously research the use of media in e-learning and have come up with empirically tested conclusions, often repeated by others, which suggest that many common practices in e-learning design are, in fact, wrong. They actually result in harming rather than helping the learning process. They call for simpler, less gimmicky use of media. Animation and audio do NOT necessarily lead to better learning and may, in fact, degrade the learning experience.

Bibliography

Clark, Ruth and Chopeta Lyons (2004).Graphics for Learning: Proven Guidelines for Planning, Designing and Evaluating Visuals in Training Materials. Jossey-Bass Pfeiffer

Mayer R E and Clark R, E-learning and the Science of Instruction (see p61 for multiple references), Pfeiffer, 2003

Richard Mayer (2001). Multi-Media Learning. Cambridge University Press

Clark, Ruth (1999). Developing Technical Training: A Structured Approach for Developing Classroom and Computer-based Instructional Materials. ISPI

Mayer R E, Systematic Thinking Fostered by Illustrations in Scientific Text. Journal of Educational Psychology, 81, 240-246.  1989

Mayer R E, and Gallini J K. When Is An Illustration Worth a Thousand Words? Journal of Educational Psychology, 88, 64-73. 1990

Mayer R E  and Anderson RB. Animations Need Narrations. Journal of Educational Psychology, 90, 312-320. 1991

Mayer R E  and Anderson RB. The Instructive Animation: Helping Students Build Connections Between Words and Pictures. Journal of Educational Psychology, 90, 312-320. 1992

Student cliff – 7 reasons for plummeting student numbers

Lots of angst has appeared around what is now being dubbed the ‘student cliff’. The Vice-Chancellor of the University of Worcester says there will be 10% drop in the proportion of UK students starting degrees this year, the steepest fall in 30 years. To be precise, he predicts a 70,000 drop in applications. We have already seen a 53,000 student shortfall. Then there's the catastrophic statistics on foreign students. UK universities get a third of their tuition fees from foreign students, yet Indian student numbers fell by 23.5% overall with a 28% drop in postgrads and Pakistani students by 13.4%, with a 19% drop in postgrads. Non-EU students coming to Britain for postgraduate courses has dropped for the first time in 16 years. The UK is now relying on Chinese students for growth but even these have dwindled and there’s every sign that China is doing everything it can to build their own capability. For the first time, no one is disagreeing about the falls. What we need to understand are the causes:

Changing demographics

Demographically, the number of UK 18-year-olds will decline over the next ten years by 11%. This will have a long-term effect on University applications. Projections released this week from the US Department of Education show enrolment figures for 2010-2021 falling miles below the 46% growth experienced between 1996-2010. This is already playing out in California, where there is a demographic shift to negative growth, combined with crippling state debt. Overall, a new report has shown that the projected ‘pupil cliff’ will result in the “death of the growth agenda in the US”.

Commoditisation

Scarcity creates value, commoditisation destroys value. Youth and graduate unemployment has gone through the roof in some countries and is still rising across Europe. The spectre of a high-cost degree with a low-salary future is starting to bite. A degree is no longer the goal but THE degree from better brand institutions. Because degrees have become commoditised, employers are also less interested in their value. Music, newspapers, retail in general, have all been commoditised through Napsterisation. The same thing is happening in learning.

Steeply rising costs

We have seen student costs soar in the US, UK and elsewhere, way beyond inflation and house price rises, yet the deliverable remains much the same. Remind you of the property bubble? Student loan costs have risen well above that of credit card debt in the US. When faced with student fees in the UK, many have chosen not to apply. Yet little has been done to lower the cost of HE which is still rooted in a high cost model based on low occupancy buildings, one intake a year, long vacations and inefficient teaching. Cost is pushing more young people to reject HE.

Debt

‘Debt’ is a dagger of a word that now strikes fear into people. It alone almost led to a global meltdown, is tearing apart the European Union, has led to massive youth unemployment and is NOT going away any time soon. To take on debt now, is to take on a massive risk. It will affect your ability to buy a flat or house, have children, sustain a credit rating, with no guarantee of a job with golden prospects. Potential students have wised up to this fact.

Relevance

The medieval model of the University as somewhere that provides a huge breadth of courses, with a focus on research rather than teaching, has led to parking the relevance argument.  We have seen dramatic drops in students applying for Universities in the UK, when they have been asked to pay £9000 a year for that privilege. Foreign student income (one third of all tuition in UK) is not geared towards this breadth but towards business and STEM subjects. Economic relevance is not the only aim of higher education but neither is the abandonment of relevance. We have to face up to the fact that relevance has become a greater factor in student choice,

Groupthink

Universities struggle with rapid, innovative change and governments are still stuck in the mindset of more degrees as an intrinsic good. Peter Thiel identified this as one of the fundamental symptoms of a bubble – groupthink. ‘The nth degree’ problem is the simplistic idea that the more degrees we fund the better. Forget the fact that the world has been brought to its fiscal knees by graduate bankers or that many of the skills we require are not taught, or taught badly, at our Universities, we need to reduce costs through the mass adoption of cheaper solutions, such as online learning. This, especially in the UK, has hindered innovation.

More options

The race is now on. Different models are emerging that lower costs and increase reach and access. This draws students away from traditional HE. MOOCs, with accreditation (Signature Course from Coursera), are now available. Other models are emerging, such as separate online departments within Universities (UDOL – University of Derby Online), outsourcing to online delivery companies that have multiple yearly intakes, low costs and no VISA problems (Interactive Design Institute). These are, at present, modifications to the existing model. But there are other more radical, tectonic shifts at work here. There’s a genuine thirst for shorter, faster courses, that are available when you want them, more relevant apprenticeships and high quality workplace learning, and not just the 18 year old undergraduate meander through a 3 or 4 year degree course.

Conclusion

Unlike the fiscal cliff, there is no sign of any immediate solution to this problem, other than taking the pain. There’s no way politicians can do a 180 degree (sic) turn on this but that’s what’s needed. After decades of expansion, the whole system has ballooned out of control with quality, and now quantity, falling. The danger is in behaving like lemmings heading towards the student cliff without adequate planning.