Practice - mostly forgotten yet where most learning takes place...

This is Mathew Syed, Britain’s No. 1 table-tennis player for ten years. Not only that, the next five best players all lived within a few streets of him. How come? Well, they had a great coach, a club that was open 24 hours a day (they all had the keys) and a group who practised, deliberately and mercilessly to get to the top. It was nothing to do with natural ‘talent’ and everything to do with effort and practice. It is all in his excellent book ‘Bounce’ (Syed 2011) where he explains not only his sporting journey but the psychology behind that journey, in particular his discovery of deliberate ‘practice’. 

In learning, learning ‘practice’ experiences (LPXs) are likely to be far more important than the original learning experience. They will play a much more powerful role in refining, reinforcing, recalling and ultimately in acquiring knowledge and skills. One simple finding that often surprises teachers is that the simple act of recall by a learner may be a stronger reinforcement event that the original teaching experiences. Practice is therefore the learning that takes place after the initial teaching has been delivered.

When I spoke to Tim O’Reilly, a name everyone who has done any coding will know, as he’s published a zillion text books in the subject. He explained how textbooks (online or offline) are only the start of a learning journey and how coders need both structured, learning experiences at the start, as well as opportunities to learn by exploring and doing. Most will use a tool such as ‘Jupyter’ a structured environment where one can practice how to code for real. It is rare for these environments to exist in other knowledge and skills domains and we have much to learn from technical training on that front.

Of course, practice environments, as simulations, have been around for decades. Pilots use them because they need to practice skills and situations that are rare in the real world. In the Mayo Clinic last year, I got a tour of their simulation facilities. It was huge and full of practice opportunities that could be supervised, without the need for endangering actual patients. Learning practice experiences matter, as it may be too expensive or dangerous to have such experiences in the real world. VR offers an expansion of such opportunities.

Beyond these formal practice environments, in most cases what is learned is reinforced and turned into an automatic skill by actual application in the real world. This takes time but has the advantage of using learning experiences within real contexts, real experiences. Few learning experiences beat real experiences for actual skills acquisition. Most of us learn this way. We may learn how to type, use PowerPoint and many other skills by doing it and, over time, getting better at it. Apprenticeships allow for practice under supervised conditions, work shadowing, internships and other placement techniques can also be useful. Even these may not be enough, so more formal programmes of on-going coaching in sport or formal practice techniques may also be necessary. In sport, you may spend far more time training than applying your skill. Whatever the methods of formal or informal practice is adopted. Practice makes perfect or at least moves you steadily in that direction.

Of course, practice may eventually make perfect but that is not the whole story as practice is a multi-faceted and complex process. One area of practice that has over 130 years of research behind it and uncovers much of this complexity, is spaced-practice.

Spaced-practice

Formal spaced-practice, at its simplest it is the spacing of practice eexperiences over time into the future to embed learning. It uses recall, rehearsal, revision, application or deliberate practice and recall of knowledge or skills spaced over time to reinforce and consolidate them in long-term memory.

Experienced learners know that spaced practice; repeated practice, really does matter. Good learners do their homework, develop revision techniques, repeat and rehearse in their heads, take notes and so on. This is often without any real guidance from the educational system, but they eventually realise that it is what leads to success. Similarly with practical skills, where the complexity of the real world provides additional learning experoinces tha move learners from being novices to experts.

So, given that millions of teachers, lecturers, trainers, coaches and instructors are employed in the learning game, it is perhaps surprising that little or no attention is paid to the idea of spaced-practice, in the actual application and professional practice of teaching and learning. One could argue that without knowledge of this principle and its causes, those who teach are missing a key set of learning experiences in the process of learning. This a little unfair, since most traditional learning has been in fixed courses in training rooms, classrooms or lecture halls and once the student has walked out the door, they have gone. All attempts at practice, revision and application is down to the learner.

But one thing has happened that changes everything – technology. We now have the ability to design, develop and deliver spaced-practice online, to our smartphones directly to the learner.

What is spaced-practice?

Ebbinghaus, in 1885, published Uber Das Gedachtis (On memory), published in English in 1913 (Ebbinghaus 1913), a groundbreaking work, which laid the foundations for the practical science of memory. Not only did he give us the application of the scientific method to the measurement of memory, this also resulted in some startling findings. First, the famous forgetting curve, that even now, has the ability to startle those who first encounter the precipitous nature of forgetting. He also explored chunking, an essential principle in memory and learning theory, a practical response to the severe limitations of working memory. Then there are his findings on primacy and recency, showing that we have a tendency to remember the first and last things in a learning experience.

Let’s focus on the forgetting curve. This applied to the recall of short strings of letters, and not all of the evidence for forgetting is as pronounced as this. Nevertheless, it is certain that most learning experiences lead to some, and usually substantial, forgetting. Although decay rates are variable this should not detract us from the task at hand, which is to increase the productivity of learning through increased retention and recall from long-term memory.

Although students often perform better immediately after ‘massed practice’ (single bout of practice), they forget quicker and perform poorly in later tests than ‘spaced-practice’ students (Keppel 1964). This is why much end-point assessment is short-term and short-sighted. 

Forgetting is initially steep and show that memories are lost very quickly then more slowly, so forgetting is not simply proportional to the passing of time (Wickelgren 1976). One solution to this predictable process of forgetting is spaced-practice which works, as Ebbinghaus discovered, through the spacing effect, the separation of learning events over time (Dempster 1988). As forgetting is a curved descent, so methods that combat forgetting (remembering) need to be spaced across a curved ascent. 

A solution to the problem of the failure to elaborate and shunt learnt knowledge and skills from working memory to long-term memory is to repeat, retrieve, review, revise, rehearse, recall and practice at spaced intervals in the future. Evidence suggests that the periodicity of these intervals matters but it is also important that it involves active recall and not just the recognition of answers. Whatever profile the forgetting curve has, and almost all learning results in a quantifiable fall, the cure is clearly to do more to consolidate the cognitive gain beyond that initial experience. If most of what we learn is forgotten it should be an imperative to slow the forgetting curve. The science suggests that this one technique has the greatest chance of substantially increasing productivity and performance in learning.

Science of spaced-practice

Memory theory is one of the most developed areas of experimental psychology and learning theory, yet the learning industry, schools, further education, higher education, corporate and adult learning have taken little practical advantage from these theoretical advances. So what does the science tell us?

All ‘learning’ comes down to encoding into memory with performance being the retrieval of that stored knowledge and skills (Anderson 1994). Yet, much learning has been shown to play to short-term memory with rapid and on-going decay. This cramming or sheep-dip learning plays to short-term memory and recall, and does not have high retention value (Dempster 1988). 

Encoding matters, so chunking of the original material to prevent cognitive overload, along with techniques to grab and sustain attention really do matter. But to consolidate memories, repeated active practice pushes knowledge and skills from working to long-term memory then consolidates these memories to make them more permanent. Spaced events, combined with repeated retrieval, consolidate memories and improve accurate recall. In practice, the repeated spaced-practice intervals get longer over time.

We have known for over a century that memories decay over time. We also know that memory is better encoded the more times it is actively learnt and that the same amount of active learning is better if distributed over time. This is important as spaced-practice gives you better performance with optimal effort. Additionally, memory is a process of reconstruction and the more we recall a learnt item, the more recallable it becomes in the future (Bjork 1988).

Memories are encoded (Anderson 1994) into long-term memory and can be consolidated (made stronger) by repeated, active spaced-practice. Spaced-practice also increases the probability, speed and performance. The promise is that the learner will be more likely to recall something quicker and better. This is why the retrieval through spaced-practice is so important. It prevents memory loss (Bjork 1975).  The more we recall, the more recallable memories become.

But the efficiency of recall can be made all the stronger by the use of ‘cues’, namely stimuli that help you recall knowledge and skills. Cues are important, as it is cues that activate the memories for recall. They’re like the handles on suitcases, which you can use to haul out stored memories. Cues can be mnemonics, contextual and there is evidence Kuiper & Kirker (1977) to suppose that they are strong when self-referenced i.e. the learner creates their own cues when encoding things to be remembered. 

Memory systems used by high performing memory champions use ‘memory palace’ techniques that are remarkably efficient in aiding cued recall. Typically you would place the items to be remembered along a well-known street you know or in the rooms of your own house, then use that known place as a cue-rich environment you are already familiar with, to recall new knowledge.

If courses are chunked, and cues deliberately provided, so that each chunk has cues or encourages the learner to create their own cues, this is useful in the construction of spaced-practice, as it is the cues and not just bits of content that can be spaced and used for recall. Later practice events can be contextual cues,, where a contextualised scenario is presented.

Active recall

Rather than simple repetition; reading, watching or listening in spaced-practice, active recall, pulling something out of memory, not just recognising something from a list or multiple choice question, improves future performance. This is something we have known for a century (Gates 1917). The act of active recall develops and strengthens memory. It also improves the process of recall in ways that passive recall – reading, listening and watching do not. In practice, it is active recall that really matters in knowledge and skills, not recognition. An additional advantage is that if we learn in a way that mimics the conditions of future recall (rarely just recognition), recall is all the more certain (Morris et al 1977).

Let’s get more precise on active recall. Professor Roediger at Washington University in St. Louis has researched this in detail. In this study students had to memorise pictures. Group 1 were simply asked to remember as many as they could, Group 2 were given a booster quiz where they were asked to actively recall as many pictures as they could, Group 3 were given spaced-practice recall exercises. No extra study was provided in any of these exercises. In a test a week later there were clear differences in performance. 

A weakness of this experiment is the charge that there was still more ‘learning time’ through the recall tests, so Roediger split students into two groups. Group 1 read a science essay and were allowed to reread that essay. Group 2 did the same but rather than re-read they were asked booster recall questions. A recall test 2 days later showed a clear difference and that difference was even more marked after 1 week,

We can take advantage in a formal, pushed system to push out cues and active recall, on knowledge that users have shown they don’t know, don’t know well or don’t feel confident about. 

Active recall and/or practice generally produces much greater performance benefits, especially repeated and spaced testing. These have been shown in many trials to be superior to the re-presentation of content. Allen, Mahler, & Estes (1969), Hogan & Kintsch (1971) Nungester & Duchastel (1982) Cuddy & Jacoby (1982) Kuo & Hirshman Izawa (1992). When this active practice is accompanied by feedback, it has even greater benefits. 

But it is the combination of spaced-practice with active recall experiences (Landauer and Bjork (1978), with repeated sessions, as well as greater gaps, that leads to optimal retention and recall. The timing of this practice is important. We know that there is a point, soon after the learning experience, where it is essential to practice but as decay slows over time, the practice sessions can be increasingly spaced out over time. This typically follows a minutes, hours, days, weeks, months pattern.

Agency

All of the above encourage student agency, where the learner plays a role in determining what they need to practice more. Agency, as a feature of memory, is important in terms of improving performance. This self-awareness of one’s own learning and processes, such as memory, is called ‘metacognition’. This metacognition (Tulving & Madigan 1970) can be used to control and improve learning (Nelson & Narens 1990). We also know that, if learners’ metacognition skills are low, they often fail to plan their study so that optimal learning and retention takes place (Nelson & Leonesio 1988). This can work to the student’s disadvantage. Zechmeister and Shaughnessy (1980) showed that metacognition of massed repetitions give learners a false view of their ability to recall knowledge. 

An additional feature of spaced-practice is the self-awareness of the learner in relation to their confidence that they know something.  However, simplistic metacognition, the students’ knowledge of their own learning is a double-edged sword. Self-perception of ability can both help and hinder learning. The bottom line is that one can study too little or too much (Nelson and Leonesio 1988). This has led to spaced-practice systems that allow the learner to express a rating on their feeling of confidence about their ability.

Spaced-practice can therefore be selective in that items that are clearly known can have less weight than items which the user is not confident about, had difficulty in learning or clearly doesn’t know. Here spaced-practice can involve algorithmic inferences that use performance data about each individual learner, then route that learner through a series of items, or network of knowledge, based on optimising their learning and spaced-practice.

It should also be noted that the use of spaced-practice methods improves a learners metacognitive skills and makes them better learners. Applying a formal method can lead to more informal methods and habits being adopted.

Blended learning

To give spaced-practice a real, practical, performance context, let us consider its place in blended learning. Blended learning has become an acceptable shorthand for learning experiences that are sophisticated in that they are designed around the real needs of the learners, types of learning and resources you have at your disposal, along with costs. Note that blended learning is not blended teaching, where you simply slam together a bit of offline and online (sometimes known as Velcro learning). It is about optimizing the learning experience for the learner. Every blended learning experience should at least consider spaced-practice as a way of maximising learning outcomes and there are many ways to introduce spaced-practice into a blended experience.

Blended components can include a wide range of spaced-practice opportunities; simple repetitions, repetitions with concise cued phrases, stories, graphics, examples, analogies, metaphors. More active retrieval components include; tests, practice, exercises, simulations, case studies and role plays. Deeper retention may also involve; discussion, debate, dialogue and collaboration.

Habitual spaced-practice

Spaced-practice needs to be habitual. These habits are common in experienced and successful learners but take time to learn and in themselves require repeated practice to become habitual. John Locke and William James both emphasised the key role that ‘habit’ plays in learning, lessons we have largely ignored. Good learners develop good learning habits. They always have something to read in their pocket or bag. They tend to be obsessive note takers, often with a long series of filled notebooks. They habitually elaborate what they hear and actively try to remember. They replay and recall in their own minds, through dialogue and re-reading their notes. They also tend to kick-start new learning habits using technology, such as bookmarking and blogging.

We also have to be conscious of context and the affective, emotional and motivational side of learning. It is also about habits – sleep, time, place, having a notebook, note taking, email, social media, blogging, mobiles, wearables and so on. Practice is on the agenda not only because it works but because we now have the tools and technology to make it work. To limit the concept of spaced practice to repetition is to limit the principle and limit our imagination when it comes to solutions. As we will see, with technology we have a chance to create learning habits in ways that were never possible even a few years ago.

Top and tail

One of the simplest techniques is to top and tail identified course chunks. This could be lectures, classes, periods, breaks or modules. Take lectures, before you start, summarise succinctly what was taught in the last lecture, then summarise what you have just taught at the end of the lecture. This inserts a small dose of spaced-practice, preferably and active learning experience, to the learner, with practice being presented, three times – once during the lecture, once at the end and once again at the start of the next. There is a double dividend here, as this also plays to the known principle of primacy and recency (we remember the first and last things more than the things in between), taking advantage of a cognitive bias to consolidate learning. The same can be applied to classroom training, with summaries before and after coffee breaks. Similar techniques can be used in online learning, where modules are topped and tailed. In other words, take advantage of every break, section, module to include a summary which repeats or gives an opportunity for learners to recall, what was learnt. Across a term you can use regular quizzes, where 50% is new and 50% is old material. These overlapping tests are, in practice, spaced-practice.

Notes

As a learner, get into the habit, not only of taking notes, but rereading and rewriting those notes. When reading books, underline key points. At the end of each chapter, write a small summary. Even better write a review of the book. Marx used to write a summary of every book he read. Studies on note taking (with control groups and reversal of note takers and non-note takers to eliminate differences) show overwhelmingly that note taking increases memory/retention. Many aspects of increased memory have been studied including; increased attention, immediate recall, delayed tests, free recall, MCQs, remembering important v less important knowledge, correlations with quality of notes and deeper learning. Bligh (2000) has detailed dozens of studies in this area. Wittrick and Alesandrini (1990) found that written notes increased learning by 30% through summaries and 22% using written analogies, compared to the control group. Why does note taking increase retention? First, increased focus, attention and concentration, the necessary conditions for learning. Second, increased attention to meaning and therefore better encoding. Thirdly, rehearsal and repetition, which processes it into long term memory. All three matter.

Conclusion

Practice is arguably the most powerful, yet most overlooked benefit in learning and true performance. Implemented properly and it is possible to have huge gains in productivity, namely the retention, recall and application of whatever has been learnt. One could go further and say that without a practice strategy, there is no learning strategy.

Bibliography

Allen, G. A., Mahler, W. A., & Estes, W. K. (1969). Effects of recall tests on long-term retention of paired associates. Journal of Verbal Learning and Verbal Behavior, 8, 463- 470. 

Anderson, J R. (1994) Learning and Memory: An Integrated Approach. John Wiley& Sons, NY, NY. 
Augé, M. (1995). Non-places: Introduction to an anthropology of supermodernity. London: Verso.

Anderson, R. (1981) Interference: The Relationship Between Response Latency and Response Accuracy. Journal of Experimental Psychology: Human Learning and Memory". 7, 326-343
Barnes, C.A. (1979) Memory deficits associated with senescence: A neurophysiological and behavioral study in the rat. J. Comp. Physiol. 43, 74-104
Bligh, D. A. (2000). What's the use of lectures?. San Francisco: Jossey-Bass.

Bjork R.A. (1975) Retrieval as a memory solidifier: an interpretation of negative recency and related phenomena. In RL Solso (Ed.) Information processing and cognition (123-124) Hillsdle, NJ: Erbaum
Bjork R.A. (1988) Retrieval practice and the maintenance of knowledge. In M.M. Gruneberg, P.E. Morris, and R.N. Sykes (Eds) Practical aspects of memory: current research and issues John Wiley& Sons, NY, NY.
Carnine D. (1989) Designing practice activities. J. Learning Disabilities, 22, 603-607
Demspter F.N. (1988) The spacing effect. A case study in the failure to apply the results of psychological research. Am. Psychol. 43, 627-634
Dudai, Y. (1996) Consolidation: fragility on the road to the engram. Neuron 17, 367-370.

Ebbinghaus, H. (1913) Memory: A contribution to experimental psychology (H.A. Ruger & C.E. Bussenues, Trans.). New York: Teachers college, Columbia University. (Originally published 1885). 

Ellenbogen J. Scientific American July 2006

Gates A.I. (1917) Recitation as a factor in memorizing. Arch. Phschol. 6, 40.
Hogan, R. M., & Kintsch, W. (1971). Differential effects of study and test trials on long- term recognition and recall. Journal of Verbal Learning and Verbal Behavior, 10, 562- 567. 

Keppel G. (1964) Facilitation in short- and long-term retention of paired associates following distributed practice in learning. J. Verbal Learn. and Verbal Behav. 3, 91-111
Kuiper, T.B & Kirker, W. S. (1977). Self-reference and the encoding of personal information. Journal of Personality & Social Psychology, 35(9), 677–688.

Kuo, T. M., & Hirshman, E. (1996). Investigations of the testing effect. American Journal of Psychology, 109, 451-464. 

Landauer T.K. and Bjork R.A. (1978) Optimal rehearsal patterns and name learning. In M. Gruneberg, P.E. Morris and R.N. Sykes (Eds). Practical aspects of memory (pp 625-632). London: Academic press. 

Moray, N. (1959) Attention in dichotic listening: affective cues and the influence of instructions. Quart. J. Exp. Psychology, 11, 56-60
Morris C.D., Brandsford J.D. and Franks J.J. (1977) Levels of processing versus transfer-appropriate processing. J. Verbal Learning and Verbal Behav. 16, 519-533
Nelson T.O. and Narens L. (1990) Metamemory: a theoretical framework and new findings. In G. Bower (Ed.) The psychology of learning and motivation (Vol. 26) New York: Academic Press.
Nelson T.O., Leonesio R.J. (1988) Allocation of self-paced study time and the "labor-in-vain effect". J Exp Psychol Learn Mem Cogn. 14, 676-86.

Nungester, R. J., & Duchastel, P. C. (1982). Testing versus review: Effects on retention. Journal of Educational Psychology, 74, 18-22. 

Syed, M. (2011). Bounce: The myth of talent and the power of practice.

Tulving E. and Madigan S.A. (1970) Memory and verbal learning. In P.H. Mussen and & M.R. Rosenweig (Eds.), Annual review of psychology. Palo Alto, CA
Wickelgren W.A. (1976) Memory storage dynamics. In W.K. Estes (Ed.), Handbook of learning and cognitive process. (Vol. 4) Hillsdale, NJ: Erlbaum

Wittrock, M.C. (1974). Learning as a generative process. Educational Psychologist, 11, 87-95. 

Wittrock, M.C. (1990). Generative processes of comprehension. Educational Psychologist, 24, 345-376. 

Wittrock, M.C, & Alesandrini, K. (1990). Generation of summaries and analogies and analytic and holistic abilities. American Educational Research Journal, 27, 489-5http://www.pinterest.com/pin/184366178468962250/02. Zechmeister, E. B., & Shaughnessy, J. J. (1980). When you know that you know and when you think that you know but you don’t. Bulletin of the Psychonomic Society, 15, 41-44. 

Engagement - most used yet most misunderstood word in learning?

Engagement is an odd word in learning. It is clearly a good thing to be engaged with a learning experience but just because people are engaged, does not mean they are learning. I’ve been totally engaged for years watching stand-up comedy experiences but can barely remember a single joke from any of them. Engagement can be a positive sign in learners but is no guarantee of learning. 

Learners can be:

·      engaged but not learning

·      engaged but simply going through things they already know

·      engaged but doing harm to learning 

·      engaged and learning

In other words, engagement can be a dangerous proxy for learning. One can be engaged and give the appearance of learning, without learning taking place, even inhibiting learning. What’s more, there may be little or no long-term or far transfer into actual application in the real world.

Engaged but not learning

Engagement while watching a video may feel like you are learning but the transience effect, fact that you have a 20 second window for attention and can only manipulate 3-4 things in working memory, mean that you are forgetting much of what you have just watched, no matter how engaged you felt. Your mind is like a shooting star burning up memories it leaves behind.

Engaged but simply going through things they already know

Learners can be comfortable skating superficially across content they are familiar with. This can also create the illusion of learning. Graham Nutall, in empirical classroom research, claimed that “In most of the classrooms we have studied, each student already knows about 40–50% of what the teacher is teaching”. Engagement with the easy and familiar is different from effortful engagement with the difficult and unfamiliar, the zone in which learning takes place.

Engaged but doing harm to learning

Then there is the engagement which is over-stimulating and prevents deeper processing, reflection and generation of ideas and retrieval from your own mind. Games can be a total distraction if engagement is with the mechanics of the game and not the learning.

Delusional learning

One thing that research in cognitive psychology has gifted to us over the last decade or so, is clear evidence that learners are delusional when it comes to judgements about what they think is engagement in their own learning. The big name in the field is Bjork, along with many other high quality researchers, who say that learning is “quite misunderstood (by learners)…. we have a flawed model of how we learn and remember”. There’s often a negative correlation between people’s judgements of their learning, what they think they have learnt and how they think they learn best - and what they’ve actually learnt and the way they can actually optimise their learning. In short, our own perceptions of learning can be seriously misleading. This is why ‘fun’ and ‘engagement’ can be misleading proxies. Also, why learner surveys and happy sheets are such bad measures of what is actually learnt and not helpful when designing optimal learning strategies and experiences.

Bjork warns us about ‘illusory learning’ and shows that learners are frequently mistaken about whether and what they have learned. In fact he has described them as ‘delusional’. Despite feeling as though they have been through a strong ‘learning experience’, that experience was the feeling that they had learned but when tested that feeling was misleading. Self-reported or data showing engagement is not, therefore, a reliable method for determining whether things were actually learned in the sense of finding a place in long-term memory and behaviour, the gold-standard for actual learning. It may just signal easy engagement, shallow experiences where the learner is engaged and entertained but does no deep learning. 

Desirable difficulty

Bjork thinks that both teaching and learning can be improved and optimised using techniques that force cognitive effort. Much learning is imagined, as it is too easy. Learning designers must therefore be careful not to allow too ‘easy’ engagement and likely to lead to forgetting. Learning experiences need to be challenging . 

Real, long-term learning requires ‘Desirable’ (accomplishable) and ‘Difficult’ learning with real effort, for high-retention. It is all in the struggle, failure and overcoming of difficulty that real effortful learning takes place. This is why so much online learning fails. To simply click on faces and see speech bubbles of text, drag and drop labels, choose true or false and so on, rarely constitutes desirable difficulty. This is click-through learning.

Desirable difficulties, are potent forms of learner engagement, such as the following learning techniques (just a selection):

·      Active retrieval practice

·      Allow failure

·      Interleaving

·      Generation

Active retrieval practice

To be specific about effortful learning, by effort we mean ‘active retrieval’ as the most powerful learning strategy at your disposal. The brain, the organ that named itself, is a unique organ in that it can test itself to see what it knows or doesn’t know. At the same time this act of retrieval consolidates has been found to be even more powerful than the original learning experience. 

The first solid research on retrieval was by Gates (1917), who tested children aged 8-16 on short biographies. Some simply re-read the material several times, others were told to look up and silently recite what they had read. The latter, who actively retrieved knowledge, showed better recall. Spitzer (1939) made over 3000 11-12 year olds read 600 word articles then tested students at periods over 2 months. The greater the gap between testing (retrieval) and the original exposure or test, the greater the forgetting. The tests themselves seemed to halt forgetting. Tulving (1967) took this further with lists of 36 words, with repeated testing and retrieval. The retrieval led to as much learning as the original act of studying. This shifted the focus away from testing as just assessment to testing as retrieval, as an act of learning in itself. Roediger et al. (2011) did a study on text material covering Egypt, Mesopotamia, India and China, in the real context of real classes in a real school, a Middle School in Columbia, Illinois. Retrieval tests, only a few minutes long, produced a full grade-level increase on the material that had been subject to retrieval. McDaniel (2011) did a further study on science subjects, with 16 year olds, on genetics, evolution and anatomy. Students who used retrieval quizzes scored 92% (A-) compared to 79% for those who did not. More than this, the effect of retrieval lasted longer, when the students were tested eight months later.

So to retrieve and recall what you need to know results in much higher levels of retention. Rather than read, re-read and underline, look away and try to retrieve and recall what you need to know. Repeatedly reading and underline text is not an affective learning experience, looking away from the page to try to recall what you think you know, is better.

This deliberate retrieval requires cognitive effort and that is why recall has a strong reinforcement effect and results in higher retention and subsequent recall. The deeper processing increases retention and subsequent recall. Note that this retrieval practice is not a test or assessment but a learning experience. In fact, it is one of the most powerful learning experiences. 

Rather than click on True or False or an option in a short list (MCQ), look away, think, generate, recall and come up with the answer. The key point here is that research has shown that retrieval is a memory modifier and makes your memory more recallable. Counter-intuitively, retrieval is much more powerful than being presented with the information. In other words it is more powerful than the original ‘teaching’ event.

Roediger and Karpicke (2006) researched studying v retrieval testing (without feedback). One week later the retrieval tested group did much better. They also asked them how much you are likely to remember in one week’s time for each method – oddly, the majority of learners got it completely wrong. Learners are notoriously delusional about learning strategies which is why learning experience design that simply panders to their wishes often fails.

Allow failure

Making errors is also a critical component of successful learning. Yet how often is failure avoided in online learning, even punished? We are so keen on ‘engagement’ and relentless positivity that there is no room for failure.

According to Kornell, Hayes and Bjork (2009), generating the wrong thing, then getting it right, leads to stronger learning. The reason is that you are activating the brain’s semantic network. Retrieval testing does better than reading or watching, as it potentiates recall. Work by Kornell (2009) shows that even unsuccessful testing is better that straight presentations. 

Retrieval testing gives you better internal feedback and works even when you get few or no correct answers. Testing, even before you have access to the material, as a learning experience, also helps learning. Once again, almost bizarrely, Heustler and Metcalfe (2012) asked learners what worked best and they were largely wrong.

Interleaving 

Mixing things up, interleaving, boosts learning. Counterintuitively, blocking learning tasks and doing them in strict order “AAABBBCCC”, seems less efficient than interleaving learning tasks “ABCABCABC”. It would seem that interleaving keeps your brain on its toes with practice that increases desirable difficulty, each task being switched. You have to differentiate or discriminate, between concepts and tasks which strengthens memory associations.

A sizeable body of research, from sports to arts, medical and other subjects and skills, show its efficacy. Then idea is that you’re better off doing separate tasks either within a subject or in different subjects, rather than a series of similar tasks on one topic, subject or drill in say sport.It has also been researched in also in the real world context of the classroom in maths, Rohrer (2011), with astonishing increases in test scores (76%). It has even been shown to be useful in critical think tasks in legal training. 

Creating more diverse connections in the brain seems to help you cope with newer situations as they arise. The research suggests that some familiarity with the subject is needed before interleaving.

Generation

Generative learning is an active, effortful process where the learner generates meaning between what they already know and new knowledge, and then between new concepts and principles that are seen for the first time. This is an extension of rehearsal in the mind and retrieval practice.

Learning experiences that get the learner to generate their own learning and manifest their knowledge and skills, is better than simply reading text. It would also appear to provides the context for greater subsequent recall. Wittrock and Kelly (1984) showed that poor readers who generated relations, connections and headings had a 20% increase in comprehension, when they were asked to come up with a generative plan, comprehension jumped by 23%.

Open input by learners, is one example, which we implement in WildFire, the answers being interpreted by AI semantic analysis.

Conclusion

When learning experiences simply make things easy, in the classroom by low levels or superficial forms of participation or online with shallow clickthrough experiences, learning can be illusory. Learning professionals and learners often engage in practices that are sub-optimal, worse still, practices that may inhibit efficient learning. Bjork recommends techniques that may seem counterintuitive and difficult to implement, as they run counter to most current practice but the evidence suggest that he is right. Technology, of course, has also allowed retrieval practice, interleaving and spaced practice to be more easily implemented over time on a personalized basis.

This research shows that our methods of engagement in online learning are often sub-optimal. The problem we face is that immediate success often means long-term failure. More focus should be given to retrieval, generative learning, interleaving and spaced practice, not presentation or clicking on items and weak multiple-choice questions. We need to be presented with desirable difficulties, through partial or complete open input. This is exactly what we’ve spent the last two years building with WildFire.

There is, of course, a further step in engagement that really matters, and is often completely ignored – transfer.

Bibliography

Bjork, R. A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe & A. Shimamura (Eds.), Metacognition: Knowing about knowing (pp. 185–205). Cambridge, MA: MIT Press.

Ericsson, K. Anders, Krampe, Ralf Th. and Tesch-Romer. Clemens (1993) The Role of Deliberate Practice in the Acquisition of Expert Performance. Psychological Review, Vol. 100. No. 3, 363-406

Nuthall G (2007) The Hidden Lives of Learners. Wellington, NZ: NZCER Press.

Gates, A. I. (1917). Recitation as a factor in memorizing. Archives of Psychology, No. 40, 1-104. 

Spitzer, H. F. (1939). Studies in retention. Journal of Educational Psychology, 30, 641-656.

Tulving, E. (1967). The effects of presentation and recall of material in free-recall learning. Journal of Verbal Learning and Verbal Behavior, 6, 175􏰀184.

Roediger, H. L., Agarwal, P. K., McDaniel, M. A., & McDermott, K. B. (2011). Test-enhanced learning in the classroom: Long-term improvements from quizzing. Journal of Experimental Psychology: Applied, 17, 382-395.

McDaniel, M. A., Agarwal, P. K., Huelser, B. J., McDermott, K. B., & Roediger, H. L. (2011). Test-enhanced learning in a middle school science classroom: The effects of quiz frequency and placement. Journal of Educational Psychology, 103, 399-414

Roediger III, H.L. and Karpicke, J.D., 2006. The power of testing memory: Basic research and implications for educational practice. Perspectives on psychological science, 1(3), pp.181-210.

Kornell, N., Hays, M.J. and Bjork, R.A., 2009. Unsuccessful retrieval attempts enhance subsequent learning. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35(4), p.989.

Kornell, N. and Son, L.K., 2009. Learners’ choices and beliefs about self-testing. Memory, 17(5), pp.493-501.

Huelser, B.J. and Metcalfe, J., 2012. Making related errors facilitates learning, but learners do not know it. Memory & cognition, 40(4), pp.514-527.

Pan, S.C., 2015. The interleaving effect: mixing it up boosts learning. Sci Am, 313(2).

Goode, S. and Magill, R.A., 1986. Contextual interference effects in learning three badminton serves. Research quarterly for exercise and sport, 57(4), pp.308-314.

Hatala, R.M., Brooks, L.R. and Norman, G.R., 2003. Practice makes perfect: the critical role of mixed practice in the acquisition of ECG interpretation skills. Advances in Health Sciences Education, 8(1), pp.17-26.

Kornell, N. and Bjork, R.A., 2008. Learning concepts and categories: Is spacing the “enemy of induction”?. Psychological science, 19(6), pp.585-592.

Rohrer, D., Dedrick, R. F., & Stershic, S. (2015). Interleaved practice improves mathematics learning. Journal of Educational Psychology, 107(3), 900–908. https://doi.org/10.1037/edu0000001

Wittrock, M.C., 1992. Generative learning processes of the brain. Educational Psychologist, 27(4), pp.531-541.

Emotion in Learning Experience Design - Norman's 3 facets; Visceral, Behavioural and Reflective...

Our lives would be impoverished without positive emotions such as fun, pleasure, joy, excitement; but also middling emotions of satisfaction, calmness, boredom; even negative emotions such as anger, sadness, melancholy and fear. They are ever-present and part of what it is to be human.

One facet of Learning Experience Design is to make the effort to engage the learner by injecting emotion into the experience. This does not mean blind emotion. Over-stimulation can be a bad thing in learning. The right kind of emotion is what helps learning, as this affective dimension can motivate the learner and aid retention. It is a matter of designing for both head and heart. It would be fair to say that most of what is written about learning design has a focus on cognition and understanding, whereas much of what drives us is feeling and emotion.

Some, like Jack Quatrell, at Learning Pool, literally say that Experience Design differs from more traditional design in being Emotional Design. He invokes Donald Norman, which is a good starting point, who in Emotional Design, broke emotional design down into three components:

Visceral (appearance)

Behavioural (performance)

Reflective (memories and experience)

Visceral (appearance)

This is the automatic, unconscious reaction we have to experiences. It is what Kahneman refers to as System 1 thinking in Thinking Fast and Slow. These reactions are fast, immediate without reflection. 

Branding and general art direction speak directly to these feelings. One practical shortcut is to copy and mimic the organisations branding guidelines, in terms of palette, font, general art direction and practices. Some organisations are very keen to get their branding and values reinforced in training. For Virgin, we implemented very strong branding across course from Values to Aircraft Maintenance, with their exact tones of red and fonts. This gives the training a visual and emotional organisational context.

An alternative is to match the branding and art direction to the topic being taught. It may be that a course on interpersonal skills will have to feel warm and friendly, whereas a course on process and procedures may need to be cool, crisp and procedural. If it is a serious scientific, financial or healthcare organisation, dealing with serious issue such as laboratory procedures, money laundering or cancer therapy, then casual cartoons may not be appropriate. Real world imagery and photography may create the right first impressions for such learning. 

Gestalt theorists have also identified this as the instant reaction to an interface or experience, rather than its components or mechanics. Gestalt principles are similar to many of the findings of researchers like Mayer in online learning; proximity, similarity, figure-ground, continuity, closure, and connection. The Gestalt Law of Proximity is often quoted in interface design and states that items close to each other are perceived as groups. This matters when you want to group navigation items (forward, back etc) separately from functional items (print, zoom etc). The Gestalt Law of Similarity states that items similar to each other will be grouped by the user. This can be used generally in interface and visual design. The Law of Figure-Ground is also important, where we see figure-ground effects

Much of this can be tested with real users in getting voiced reactions to specific questions on designs. So, keep in mind the visceral reaction of your target audience, not you the designer.

Behavioural (performance)

This is about emotion and feelings around actual use or usability. How easy is it to use in terms of speed of recognition, understanding and not making errors. Are navigational items in the optimal order? Are the icons clear? Are they big enough for smartphone touchscreens? Do they react when pressed to give feedback? Are they placed consistently in the same place on the screen? There is a massive amount of good practice in interface design around usability. It is vital that the interface is a simple, consistent, predictable and easy to use as possible, as time and cognitive effort spent on the interface detracts from the cognitive effort needed to learn.

There is also a large amount of established good practice around visual design of text, graphics, animation, audio and video. We know a lot about the different affordances of different media, and how to mix them, without inducing cognitive overload. The trick is to get the best out of each medium and media mix. 

Just as importantly, is the good practice around learning, which is often very different from other forms of screen presentation, such as the need for chunking, cues, repetition, summaries and so on. Cognitive overload is common in badly designed learning content, so a knowledge of good learning theory informs the behavioural side of design.

What you get learners to DO is also really important, as that is likely to be more powerful than what they see or hear. This is where experience design needs to include interaction with the mind, beyond just clicking on items to navigate or see pop-ups. Cognitive effort matters - a lot!  We must be very careful here. It is all too easy to make learning too easy. Without challenge, difficulty and cognitive effort, you will not have the deep processing necessary for learnt knowledge, skills and behaviour to stick. The learner will skate over the surface, thinking they have been learning, when, in fact, those experiences have been transitory. This illusory sense of learning is common and is reinforced when things feel easy. It is easy to watch a video and not realise that much of what you have just watched has left your memory before you have finished watching the entire video. It stays in working memory but never gets processed into long-term memory, so disappears. Interestingly what makes you feel as though you have learnt things is just those ‘feelings and emotions’. In this case emotion is our enemy, our greatest danger. This transitory effect is well-researched, real, common and measurable. Learning needs to be effortful.

Inducing emotion may be ideal when you want attitudinal shift in diversity, equality and other belief shift or self-awareness training but can be dangerous in non-affective training, where it can induce the illusion of learning. So, keep in mind the behavioural reaction of your target audience, not just what you the designer likes or may be familiar with.

Reflective (memories and experience)

This is an important set of feelings in learning, that you can rationalise, reflect, reinforce, recall and apply your learning. This is what Kahneman called System 2 thinking, the rational, reasoning side of the brain. We have feelings of achievement, success, confidence or having overcome difficulties in a learning experience that really do matter. It can be those feelings around having got there, not because you found it easy but realising that it was hard.

This is complex and involves much more than just getting a score on the assessment, although that can be an important feeling of success. Hence the frequent request to provide printed certificates for learners, even though they have no serious accreditation body behind them. It is similar to the status people attach to watches, handbags and branded clothes. You can be made to feel better by going through learning experiences that give you feelings of success and status.

Challenging cognitive effort can propel the learner forward and make them feel as though they really are making progress. Feedback is also a powerful accelerator of learning, so personalising learning and feedback can move things forward making the learner feel good about themselves.

One important facet of reflective feeling comes through the follow-up, actually doing something. This can be triggered by nudge learning, so that the learner gets their kicks through going back to their job and actually implementing a challenge, such as mentoring a younger employee or using those features in a spreadsheet. Satisfaction with real-world application of learning can bring high levels of satisfaction and accomplishment.

It is easy to forget that one learns for a reason, ultimately to apply that knowledge, so the transfer through to action really does matter. This is often quietly forgotten in online learning but as technology increasingly allows us to learn in the workflow, it is becoming more of a reality.

This can be tested through stickability of the learning experiences but also real assessment, not only of short-term accomplishment but long-term retention.

Conclusion

Interestingly, Norman thought Americans value Behavioural more than the Visceral & Reflective, whereas Europeans, tend to value the Visceral and Behavioural. This is fascinating. He claims that different people buy things with different fuel mixtures of the three types of emotions. This may well be true in learning and having delivered online learning in many different geographies and cultures, I think it is. These differences are real but the differences are getting less, as there is a global spread of online services, such as Google, social media, Wikipedia and Netflix which are now universal, but it is important to be aware of cultural differentials. 

So there are three levels of emotional or affective experience design. The visceral is that first impression, the overall and holistic feel of the experience. The behavioural comes from using the product both in navigation but also functionality and, importantly, in actual effortful, learning experiences. The reflective are those more rational and conscious feelings around achievement and success in learning. Beyond Norman, there are also other cognitive feelings around aesthetics, beauty, layout, space, colour and simplicity in design, that also count.

It is vital that we don’t just invoke emotions and feelings just for the sake of doing so. They matter in terms of engagement, stickiness but they must also be compatible with the actual acquisition of knowledge, skills and behaviour. Retention does matter, not just the feeling you have learned but the fact that you really did learn. Kahneman is right to remind us of the existence of fast and slow ways of thinking but he also warns us against the bias and mistakes that emotional and instinctive thinking brings in its wake. 

Bibliography

Norman, D.A., 2004. Emotional design: Why we love (or hate) everyday things. Basic Civitas Books.

Kahneman, D. and Patrick, E., 2011. Thinking, fast and slow. Allen Lane.

https://www.interaction-design.org/literature/book/the-glossary-of-human-computer-interaction/gestalt-principles-of-form-perception

Miller, G., 2009. Spent: Sex, evolution, and consumer behavior. Penguin.

Fun, humour... it's place in learning...

Should all learning be fun? No one is against a bit of fun on principle but as an imperative for learning, it is an odd goal. It’s a word that rarely figures in learning theory or the psychology of learning, yet some swear by it motivational effect in games and gamification of learning through play.

One purveyor of mirth, during a keynote at a major global conference I attended, started with his usual appeal to the audience… 

“Let me give you another word for fun - HA HA (that’s two words, but let’s not quibble)…. turn to your neighbour and say that without one letter”. 

Some, like my neighbour, were genuinely puzzled ‘HAH?’ he said. 

I think it’s ‘AHA’, says I.

Oh dear. This guy wanted all learning to be like Broadway. He then showed a police training video with six police officers dancing in a six-ball pool formation singing a song. Everything has to be fun? Eh… maybe not.

Icebreakers that go down like the Titanic

In practice, fun is often used to kick-start training sessions through ice-breakers. Although even here the dangers are evident. Whipping out a ukulele and singing The Jungle Book, stopping to ask for the next word is simply patronising (true story). Ice-breakers like this go down as well as the Titanic. Many dislike the touchy-feely stuff that passes for introductory fun and they can make educators and trainers seem frivolous, lightweight, even ridiculous. There’s no harm in setting the tone, and getting attention, as good trainers and facilitators do but making people squirm is not exactly the point.

Worst piece of online learning EVER?

Turning to fun in online learning, the worst example of online learning I've ever seen, was ridiculed by PewDiePie, a legend among his 100+ million YouTube followers. He lives in my home town, Brighton, and has built his reputation on videos that praise, review games. Unusually, this piece of US gamified, online learning on Cybersecurity was his target and he eviscerates it. It is a hilarious example of endless attempts at ‘fun’; condescending scenarios, special effects, awful multiple-choice questions, interspersed with screens packed with text, even a game within a game. 

It both looks and feels awful. Not sure where the art direction came from but it’s all over the place. All wacky cartoon-style learning. It tries SO hard to be engaging with 3D animated characters but they are straight out of the clichéd, clip-art, playbook. Then the animated effects that slide, whoosh and pop up like a disjointed, surreal dream. We need to sit our teams of content designers down, and scream out the simple principle – LESS IS MORE. We have decades of research showing that all of this ‘noise’ inhibits, not enhances, learning. 

It is clearly a game designed by someone who has no actual knowledge computer games, not uncommon in online learning design. So determined are they to gamify everything, they completely kill off the learning. You get a lot of this in online learning, badly executed, Pavlovian nonsense. Like one of those ‘so bad it’s good’ cult movies. It is funny but for all the wrong reasons.

How can I sum this up. It should be compulsory to show this example in ALL instructional design courses, as an example of how NOT to design learning experiences. It is so surreal that it could pass for a deliberate takedown of online learning. When online learning has come to this, you know it is time for a rethink. Enough already with the overblown cartoons, graphics and grotesque gamification. Stop. Slow down. Keep it simple. This is what used to be called edutainment but it is neither edu nor tainment, it is the ugly cul-de-sac of an industry that has abandoned learning for cheap media production. It shows the danger in over-egging ‘experiences’ forgetting that they must be ‘learning experiences’.

Humour

Humour has a useful role in life and also in learning, But again, ike the word ‘fun’ let’s not get carried away. Humour is best used sparingly, too lift things or play the role it plays in normal life. The harder, sharper term ‘humour’ is in many ways more interesting than ‘fun’. Humour plays an important role in social behaviour, cementing relationships, revealing attitudes and generally has a cohesive social effect. It can be used that way in learning. In learning it can introduce and disarm unease around sensitive concepts, grab and gain attention, lift learners when they are flagging. Jokes use incongruities, they raise arousal and depend on the fall, they provide relief. 

Yet grand theories of humour, from Plato to Freud have largely been abandoned in favour of a more detailed analysis in cognitive psychology, where it is seen as many different things in different contexts. And there’s the rub, you must be very careful with humour as its nature, how it manifests itself and value, varies from culture to culture. It is easy to come across as crass or insensitive. Then again, when the moment is right, it can have a place.

I’m a fan of stand-up comedy and attend the Edinburgh Festival every year and watch a ton a lot of comedy. But, honestly, I can barely remember a couple of jokes from the hundreds of hours I’ve ‘experienced’ at live comedy. It was fun but they were not ‘learning’ experiences. There’s a very big difference. A touch of humour certainly helps raise attention but learning is not stand-up comedy. In fact, we famously forget most jokes, as they don’t fit into existing knowledge schemas. Fun can be the occasional cherry on the cake but never the whole cake.

There is a real danger in pandering to this kindergarten view of learning, where fun has to be injected into learning experiences. Learning is not a circus and teachers are not clowns.

Games and gamification

The massive success of online games led many to suggest that games and gamification, could be used to turbo-charge online learning. Take a little magic dust from gaming, sprinkle generously and we’ll all find it more fun, be more motivated and learn to love learning. But there’s pros and cons here, as it can both help and hinder learning. If gamification is simply scoring, bonuses and badges, the 21st century version of Pavlov's dogs, that would be a disappointment. So much gamified learning looks like a bad fruit machine. The simple stimuli, scores and rewards may keep learners going forward but it can distract from the learning, distract in the sense of encouraging the winning of the game rather than achieving learning objectives, shallow forms of engagement and the disappointment of playing astounding games at home and getting micky-mouse games in learning. The dangers is that of skating across the surface of content. It may will demand more cognitive effort, as the rules of the game and its mechanics will have to be understood and acted upon, for not much gain. The very word is off-putting for many. Don’t assume that because you like and play games that others have the same view. The danger is in taking learning back to the behaviourist era, with simple, Pavlovian, conditioned responses, or S-O-R theory. 

On the other hand, many proven, evidence-based pieces of learning theory seem to be congruent with games techniques, such as chunking, levels of performance, constructive failure, practice, doing and performance. It can be personalized when adaptive and provides specific advantages when repeated practice is necessary. Above all it has motivational qualities, although we need to be careful here as phenomena like leaderboards can have short-lived benefits.

I like the light touch gamification in Duolingo - sprints, adaptive and spaced practice. When gamification is congruent with good principles in learning theory, it has the power to increase the effectiveness of the learning.  I’m fond of the chunking, repeats and mastery but less keen on superficial scores, bonuses and badges. That’s not to say they don’t have a place but they often seem like a superfluous layer of complexity.

Learning

It is important that learners are motivated, attentive, listening, making the effort to understand and reflect, doing things, not just reacting. Audience reaction is arousal, not attention, what entertainment does. Audience focus, attention and participation is what learners do. Fun can certainly be used to raise attention, at the start of a learning experience. A list of dull learning objectives at the start of every course or module is no one’s idea of fun. Learning is about focused attention.

We should not mistake fun for other concepts, like attention, being engrossed, gripped or immersed in a task, in fact flow or being in the zone requires an absence of this type opf over-stimulation and arousal. I have read literally thousands of books in my life and rarely chortled while reading them. Athletes learn intensely in their sports and barely register a titter. Learning requires attention, focus and effort, not a good giggle. Only those who think that teaching is actually about performance or those whose only form of evaluation is ‘Happy sheets’, adhere to the nonsense that learning should be ALL fun. Others make non sequiturs, claiming that those who disagree that ALL learning should be fun, think that all learning should be dull and boring. No. Just because I don’t think that all clothes should be white, doesn’t mean I believe they should all be black. 

It is this obsession that led to the excesses of gamification, with its battery of Pavlovian techniques, which mostly distract from the effort needed to learn and retain. It is what’s led to a lot of online learning being click-through cartoons, speech bubbles, largely the presentation of text, graphics and video, with little in the way of effortful learning, apart from multiple-choice options. 

It is not that some fun and the affective side of learning don't matter, just that it is pointless motivating people to embark on learning experiences if they don't actually learn. This is not a false dichotomy, between fun and learning, it is the recognition that there are optimal learning strategies. A touch of humour certainly helps raise attention but learning is not comedy. In fact, we famously forget most jokes, as they don’t fit into existing knowledge schemas. Fun can be the occasional cherry on the cake but never the whole cake.

Which is why open input, effortful learning tools like WildFire result in much higher levels of retention. When designers focus relentlessly on fun they more often than not, destroy learning. There is perhaps no greater sin in online learning than presenting adults with hundreds of screens of cartoons, speech bubbles and endless clicking, in the name of ‘fun’.

'Fun', funnily enough, can be a rather sad word – like the children’s clown walking to the gate after a kid’s party, even tragic. It has the whiff of the kindergarden and diminishes and demeans learning. There’s an emptiness at the heart of the learning game. A refusal to accept that we know a lot about learning, that research matters. The purveyors of fun, and those who think it’s a performance art, in the end tend to serve up the sort of nonsense that creates superficial, click-through, online learning. 

In truth, learning is about focus, attention, reflection (often serious introspection), retrieval, practice, application and transfer. Learning is not a circus and teachers are not clowns.