Monday, December 07, 2020

Foreword to Conducting performance-based Instructional Analysis by Guy Wallace

Foreword to Conducting performance-based Instructional Analysis by Guy Wallace

There are two things that are in danger of extinction in learning, with all the focus on learning in the workflow and learning ‘experiences’. They lie at the ‘top' and ‘tail' of a learning programme.

The first, at the top, is the abandonment of analysis before you design a learning intervention. Detailed analysis seems to be out of fashion but its absence can lead to learning experiences that are simply illusory. Guy has a lifetime of experience which he brings to bear on the management and process of delivering a learning project. His focus on stakeholders, for example, is just one of many solid pieces of advice to guide the novice.

But it is at the tail that Guy’s method really shines. His obsession with ‘performance’ is so often swept under the carpet by those who think that learning is just about knowledge. How do we get what we know turned into actual practice and performance? This is what lies behind Guy’s methodology. He is relentless in chaining together a causality that leads to actual competence and performance.

It was refreshing to see someone who has been around long enough to remember Gilbert’s book on Human Competences and Gloria Gery on Performance Support. We are entering an age where AI and smart software can now deliver, push and pull instruction, to support learners when they need it. This puts learning much closer to performance. Transfer is that much easier when you learn at the very moment you put it into practice.

Analysis, process and performance - ignore them at your peril.

Thursday, December 03, 2020

OEB debate on Higher Education... 7 mins each... here's my 7 mins worth...

 ‘This House Believes Universities, in Their Current Form, Are Unsustainable as Mass-participant Institutions’

We each had 7 minutes to put our case… here’s mine…

My 1st argument is PERSONAL

I have two sons – twins, now 27 years old – interesting experiment – one went to University and has a degree in AI, one did NOT. He hated school and University, sitting in lectures would have sucked then life out of him. The one who did not has done as well as the other, runs his own business, and employs graduates. And that’s my point. We need to create a system that works for everyone not just the academic kids. 

We have an estimated shortage of 40,000 nurses in the UK, so let me tell you another personal story, both my mother and sister were nurses. They would not have been nurses now. The degree qualification excluded working class women and men from that profession. It increases class inequalities. We have to stop seeing University, as some sort of religious imperative. Like Catholic indulgences, where you have to pay to get the piece of paper or you fall into the HELL of low status and precarious employment. We need a Reformation and this motion suggests that. 

We need to REBALANCE our education system more towards people like my son, sister and mother.

2nd  argument is POLITICAL

In 2010 Peter Turchin, a scientist at the University of Connecticut who uses maths to model historical change, made a startling prediction in Nature. He said, “overproduction of young graduates with advanced degrees” would lead to social unrest due to rising inequalities. TTurned out he was right. The evidence… Trump, Brexit, the Gilet Jaunes in France… The graduate class has not only screwed over the rest of the working population over, it actually looks down on them – calls them deplorables…  The two best books I’ve read on Brexit by Stephen Davies and David Goodhart argue that this is what is behind such social discontent – the rising inequalities between the professional, urban, middle-class graduate classes and the rest. It was the graduates in Wall Street who caused the financial crash, nor ordinary working people.

We need to REBALANCE the inequalities in society and unite our DIVIDED societies, divided now by education…and not look down on people just because they didn’t go to University.

3rd argument is ECONOMIC

We have demoted vocational skills, outsourced manufacturing skills to China and wonder why our economies are faltering. Yet the graduate barista is now the norm. The destruction of vocational education has been a moral disaster and disgrace. But not everywhere. And I tip my hat to Germany, Austria and Switzerland who have more balanced systems and, unsurprisingly the strongest economies in Europe. If you think that Universities drive economic wealth think again. Taiwan, South Korea, and China grew their economies in spite of having low University infrastructure. 

As Bryan Caplan here at Online Educa, 80% of this is SIGNALLING but the cost of that 80% is needed elsewhere for healthcare and spending on those in real need.

In the US the cost as of now is a staggering $1.67 trillion to the national debt and don’t think that forgiving student debt is the answer. Forgiveness would be a highly regressive policy. Full cancellation would put $192 bn into the pockets of the top 20% of earners and a measly sum to the bottom. It does not rebalance, it increases inequalities.

We need to REBALANCE the system economically for growth and prosperity, as it costs too much.


My son did a degree in AI at a European University. Few of his lectures were recorded (despite the students pleading for them to be so). Two hours of high end calculus is not easy and actually a bad way to teach calculus. Online resources were rare. Universities, globally, are still trapped in the 1 hour lecture (only because the Sumerians has a 60-base number system), the essay is still a dominant form of assignment and assessment. Yet look at the scale of essay mills. You can buy a freshly written Masters thesis for less than €1000. And when it came to Covid the rush to the university of Zoom – exactly what you should NOT do in online learning was bizarre. This is ONLINE EDUCA folks we should have done better…

We need to REBALANCE towards online delivery. All courses should be Blended Learning and digital by default.


Lastly a cultural argument. Universities, as Jonathon Haidt, in his excellent book The Coddling of the American Mind shows, our universities seem to be producing people with less, rather than more, resilience in the real world. We had the unedifying spectacle of junior employees of a global publisher Random House/Penguin literally crying their little eyes out when they heard that they employer was publishing a book by Jordan Peterson. I was raised in a world where crying, when you couldn’t get your own way, was abandoned with diapers - not whenever a work colleague thought differently from you. Debate is being cancelled. Diversity of though no longer respected.

We need to REBALANCE towards freedom of speech and diversity of thought.


We’re in this debate but how many of us did NOT go to University? This, I think, is a problem. We have produced a graduate class that speaks largely to itself, in bubble conferences for graduates, talking about US. But this DEBATE is not about US!  

This is ONLINE EDUCA, so let us  pedagogically REBALANCE towards vocational, towards online and blended learning, to prevent further political, economic, pedagogic and cultural damage. With my friend Brian, I recommend this motion to the House.

Monday, October 12, 2020

Practice - mostly forgotten yet where most learning takes place...

This is Mathew Syed, Britain’s No. 1 table-tennis player for ten years. Not only that, the next five best players all lived within a few streets of him. How come? Well, they had a great coach, a club that was open 24 hours a day (they all had the keys) and a group who practised, deliberately and mercilessly to get to the top. It was nothing to do with natural ‘talent’ and everything to do with effort and practice. It is all in his excellent book ‘Bounce’ (Syed 2011) where he explains not only his sporting journey but the psychology behind that journey, in particular his discovery of deliberate ‘practice’. 

In learning, learning ‘practice’ experiences (LPXs) are likely to be far more important than the original learning experience. They will play a much more powerful role in refining, reinforcing, recalling and ultimately in acquiring knowledge and skills. One simple finding that often surprises teachers is that the simple act of recall by a learner may be a stronger reinforcement event that the original teaching experiences. Practice is therefore the learning that takes place after the initial teaching has been delivered.

When I spoke to Tim O’Reilly, a name everyone who has done any coding will know, as he’s published a zillion text books in the subject. He explained how textbooks (online or offline) are only the start of a learning journey and how coders need both structured, learning experiences at the start, as well as opportunities to learn by exploring and doing. Most will use a tool such as ‘Jupyter’ a structured environment where one can practice how to code for real. It is rare for these environments to exist in other knowledge and skills domains and we have much to learn from technical training on that front.

Of course, practice environments, as simulations, have been around for decades. Pilots use them because they need to practice skills and situations that are rare in the real world. In the Mayo Clinic last year, I got a tour of their simulation facilities. It was huge and full of practice opportunities that could be supervised, without the need for endangering actual patients. Learning practice experiences matter, as it may be too expensive or dangerous to have such experiences in the real world. VR offers an expansion of such opportunities.

Beyond these formal practice environments, in most cases what is learned is reinforced and turned into an automatic skill by actual application in the real world. This takes time but has the advantage of using learning experiences within real contexts, real experiences. Few learning experiences beat real experiences for actual skills acquisition. Most of us learn this way. We may learn how to type, use PowerPoint and many other skills by doing it and, over time, getting better at it. Apprenticeships allow for practice under supervised conditions, work shadowing, internships and other placement techniques can also be useful. Even these may not be enough, so more formal programmes of on-going coaching in sport or formal practice techniques may also be necessary. In sport, you may spend far more time training than applying your skill. Whatever the methods of formal or informal practice is adopted. Practice makes perfect or at least moves you steadily in that direction.

Of course, practice may eventually make perfect but that is not the whole story as practice is a multi-faceted and complex process. One area of practice that has over 130 years of research behind it and uncovers much of this complexity, is spaced-practice.


Formal spaced-practice, at its simplest it is the spacing of practice eexperiences over time into the future to embed learning. It uses recall, rehearsal, revision, application or deliberate practice and recall of knowledge or skills spaced over time to reinforce and consolidate them in long-term memory.

Experienced learners know that spaced practice; repeated practice, really does matter. Good learners do their homework, develop revision techniques, repeat and rehearse in their heads, take notes and so on. This is often without any real guidance from the educational system, but they eventually realise that it is what leads to success. Similarly with practical skills, where the complexity of the real world provides additional learning experoinces tha move learners from being novices to experts.

So, given that millions of teachers, lecturers, trainers, coaches and instructors are employed in the learning game, it is perhaps surprising that little or no attention is paid to the idea of spaced-practice, in the actual application and professional practice of teaching and learning. One could argue that without knowledge of this principle and its causes, those who teach are missing a key set of learning experiences in the process of learning. This a little unfair, since most traditional learning has been in fixed courses in training rooms, classrooms or lecture halls and once the student has walked out the door, they have gone. All attempts at practice, revision and application is down to the learner.

But one thing has happened that changes everything – technology. We now have the ability to design, develop and deliver spaced-practice online, to our smartphones directly to the learner.

What is spaced-practice?

Ebbinghaus, in 1885, published Uber Das Gedachtis (On memory), published in English in 1913 (Ebbinghaus 1913), a groundbreaking work, which laid the foundations for the practical science of memory. Not only did he give us the application of the scientific method to the measurement of memory, this also resulted in some startling findings. First, the famous forgetting curve, that even now, has the ability to startle those who first encounter the precipitous nature of forgetting. He also explored chunking, an essential principle in memory and learning theory, a practical response to the severe limitations of working memory. Then there are his findings on primacy and recency, showing that we have a tendency to remember the first and last things in a learning experience.

Let’s focus on the forgetting curve. This applied to the recall of short strings of letters, and not all of the evidence for forgetting is as pronounced as this. Nevertheless, it is certain that most learning experiences lead to some, and usually substantial, forgetting. Although decay rates are variable this should not detract us from the task at hand, which is to increase the productivity of learning through increased retention and recall from long-term memory.

Although students often perform better immediately after ‘massed practice’ (single bout of practice), they forget quicker and perform poorly in later tests than ‘spaced-practice’ students (Keppel 1964). This is why much end-point assessment is short-term and short-sighted. 

Forgetting is initially steep and show that memories are lost very quickly then more slowly, so forgetting is not simply proportional to the passing of time (Wickelgren 1976). One solution to this predictable process of forgetting is spaced-practice which works, as Ebbinghaus discovered, through the spacing effect, the separation of learning events over time (Dempster 1988). As forgetting is a curved descent, so methods that combat forgetting (remembering) need to be spaced across a curved ascent. 

A solution to the problem of the failure to elaborate and shunt learnt knowledge and skills from working memory to long-term memory is to repeat, retrieve, review, revise, rehearse, recall and practice at spaced intervals in the future. Evidence suggests that the periodicity of these intervals matters but it is also important that it involves active recall and not just the recognition of answers. Whatever profile the forgetting curve has, and almost all learning results in a quantifiable fall, the cure is clearly to do more to consolidate the cognitive gain beyond that initial experience. If most of what we learn is forgotten it should be an imperative to slow the forgetting curve. The science suggests that this one technique has the greatest chance of substantially increasing productivity and performance in learning.

Science of spaced-practice

Memory theory is one of the most developed areas of experimental psychology and learning theory, yet the learning industry, schools, further education, higher education, corporate and adult learning have taken little practical advantage from these theoretical advances. So what does the science tell us?

All ‘learning’ comes down to encoding into memory with performance being the retrieval of that stored knowledge and skills (Anderson 1994). Yet, much learning has been shown to play to short-term memory with rapid and on-going decay. This cramming or sheep-dip learning plays to short-term memory and recall, and does not have high retention value (Dempster 1988). 

Encoding matters, so chunking of the original material to prevent cognitive overload, along with techniques to grab and sustain attention really do matter. But to consolidate memories, repeated active practice pushes knowledge and skills from working to long-term memory then consolidates these memories to make them more permanent. Spaced events, combined with repeated retrieval, consolidate memories and improve accurate recall. In practice, the repeated spaced-practice intervals get longer over time.

We have known for over a century that memories decay over time. We also know that memory is better encoded the more times it is actively learnt and that the same amount of active learning is better if distributed over time. This is important as spaced-practice gives you better performance with optimal effort. Additionally, memory is a process of reconstruction and the more we recall a learnt item, the more recallable it becomes in the future (Bjork 1988).

Memories are encoded (Anderson 1994) into long-term memory and can be consolidated (made stronger) by repeated, active spaced-practice. Spaced-practice also increases the probability, speed and performance. The promise is that the learner will be more likely to recall something quicker and better. This is why the retrieval through spaced-practice is so important. It prevents memory loss (Bjork 1975).  The more we recall, the more recallable memories become.

But the efficiency of recall can be made all the stronger by the use of ‘cues’, namely stimuli that help you recall knowledge and skills. Cues are important, as it is cues that activate the memories for recall. They’re like the handles on suitcases, which you can use to haul out stored memories. Cues can be mnemonics, contextual and there is evidence Kuiper & Kirker (1977) to suppose that they are strong when self-referenced i.e. the learner creates their own cues when encoding things to be remembered. 

Memory systems used by high performing memory champions use ‘memory palace’ techniques that are remarkably efficient in aiding cued recall. Typically you would place the items to be remembered along a well-known street you know or in the rooms of your own house, then use that known place as a cue-rich environment you are already familiar with, to recall new knowledge.

If courses are chunked, and cues deliberately provided, so that each chunk has cues or encourages the learner to create their own cues, this is useful in the construction of spaced-practice, as it is the cues and not just bits of content that can be spaced and used for recall. Later practice events can be contextual cues,, where a contextualised scenario is presented.

Active recall

Rather than simple repetition; reading, watching or listening in spaced-practice, active recall, pulling something out of memory, not just recognising something from a list or multiple choice question, improves future performance. This is something we have known for a century (Gates 1917). The act of active recall develops and strengthens memory. It also improves the process of recall in ways that passive recall – reading, listening and watching do not. In practice, it is active recall that really matters in knowledge and skills, not recognition. An additional advantage is that if we learn in a way that mimics the conditions of future recall (rarely just recognition), recall is all the more certain (Morris et al 1977).

Let’s get more precise on active recall. Professor Roediger at Washington University in St. Louis has researched this in detail. In this study students had to memorise pictures. Group 1 were simply asked to remember as many as they could, Group 2 were given a booster quiz where they were asked to actively recall as many pictures as they could, Group 3 were given spaced-practice recall exercises. No extra study was provided in any of these exercises. In a test a week later there were clear differences in performance. 

A weakness of this experiment is the charge that there was still more ‘learning time’ through the recall tests, so Roediger split students into two groups. Group 1 read a science essay and were allowed to reread that essay. Group 2 did the same but rather than re-read they were asked booster recall questions. A recall test 2 days later showed a clear difference and that difference was even more marked after 1 week,

We can take advantage in a formal, pushed system to push out cues and active recall, on knowledge that users have shown they don’t know, don’t know well or don’t feel confident about. 

Active recall and/or practice generally produces much greater performance benefits, especially repeated and spaced testing. These have been shown in many trials to be superior to the re-presentation of content. Allen, Mahler, & Estes (1969), Hogan & Kintsch (1971) Nungester & Duchastel (1982) Cuddy & Jacoby (1982) Kuo & Hirshman Izawa (1992). When this active practice is accompanied by feedback, it has even greater benefits. 

But it is the combination of spaced-practice with active recall experiences (Landauer and Bjork (1978), with repeated sessions, as well as greater gaps, that leads to optimal retention and recall. The timing of this practice is important. We know that there is a point, soon after the learning experience, where it is essential to practice but as decay slows over time, the practice sessions can be increasingly spaced out over time. This typically follows a minutes, hours, days, weeks, months pattern.


All of the above encourage student agency, where the learner plays a role in determining what they need to practice more. Agency, as a feature of memory, is important in terms of improving performance. This self-awareness of one’s own learning and processes, such as memory, is called ‘metacognition’. This metacognition (Tulving & Madigan 1970) can be used to control and improve learning (Nelson & Narens 1990). We also know that, if learners’ metacognition skills are low, they often fail to plan their study so that optimal learning and retention takes place (Nelson & Leonesio 1988). This can work to the student’s disadvantage. Zechmeister and Shaughnessy (1980) showed that metacognition of massed repetitions give learners a false view of their ability to recall knowledge. 

An additional feature of spaced-practice is the self-awareness of the learner in relation to their confidence that they know something.  However, simplistic metacognition, the students’ knowledge of their own learning is a double-edged sword. Self-perception of ability can both help and hinder learning. The bottom line is that one can study too little or too much (Nelson and Leonesio 1988). This has led to spaced-practice systems that allow the learner to express a rating on their feeling of confidence about their ability.

Spaced-practice can therefore be selective in that items that are clearly known can have less weight than items which the user is not confident about, had difficulty in learning or clearly doesn’t know. Here spaced-practice can involve algorithmic inferences that use performance data about each individual learner, then route that learner through a series of items, or network of knowledge, based on optimising their learning and spaced-practice.

It should also be noted that the use of spaced-practice methods improves a learners metacognitive skills and makes them better learners. Applying a formal method can lead to more informal methods and habits being adopted.

Blended learning

To give spaced-practice a real, practical, performance context, let us consider its place in blended learning. Blended learning has become an acceptable shorthand for learning experiences that are sophisticated in that they are designed around the real needs of the learners, types of learning and resources you have at your disposal, along with costs. Note that blended learning is not blended teaching, where you simply slam together a bit of offline and online (sometimes known as Velcro learning). It is about optimizing the learning experience for the learner. Every blended learning experience should at least consider spaced-practice as a way of maximising learning outcomes and there are many ways to introduce spaced-practice into a blended experience.

Blended components can include a wide range of spaced-practice opportunities; simple repetitions, repetitions with concise cued phrases, stories, graphics, examples, analogies, metaphors. More active retrieval components include; tests, practice, exercises, simulations, case studies and role plays. Deeper retention may also involve; discussion, debate, dialogue and collaboration.

Habitual spaced-practice

Spaced-practice needs to be habitual. These habits are common in experienced and successful learners but take time to learn and in themselves require repeated practice to become habitual. John Locke and William James both emphasised the key role that ‘habit’ plays in learning, lessons we have largely ignored. Good learners develop good learning habits. They always have something to read in their pocket or bag. They tend to be obsessive note takers, often with a long series of filled notebooks. They habitually elaborate what they hear and actively try to remember. They replay and recall in their own minds, through dialogue and re-reading their notes. They also tend to kick-start new learning habits using technology, such as bookmarking and blogging.

We also have to be conscious of context and the affective, emotional and motivational side of learning. It is also about habits – sleep, time, place, having a notebook, note taking, email, social media, blogging, mobiles, wearables and so on. Practice is on the agenda not only because it works but because we now have the tools and technology to make it work. To limit the concept of spaced practice to repetition is to limit the principle and limit our imagination when it comes to solutions. As we will see, with technology we have a chance to create learning habits in ways that were never possible even a few years ago.

Top and tail

One of the simplest techniques is to top and tail identified course chunks. This could be lectures, classes, periods, breaks or modules. Take lectures, before you start, summarise succinctly what was taught in the last lecture, then summarise what you have just taught at the end of the lecture. This inserts a small dose of spaced-practice, preferably and active learning experience, to the learner, with practice being presented, three times – once during the lecture, once at the end and once again at the start of the next. There is a double dividend here, as this also plays to the known principle of primacy and recency (we remember the first and last things more than the things in between), taking advantage of a cognitive bias to consolidate learning. The same can be applied to classroom training, with summaries before and after coffee breaks. Similar techniques can be used in online learning, where modules are topped and tailed. In other words, take advantage of every break, section, module to include a summary which repeats or gives an opportunity for learners to recall, what was learnt. Across a term you can use regular quizzes, where 50% is new and 50% is old material. These overlapping tests are, in practice, spaced-practice.


As a learner, get into the habit, not only of taking notes, but rereading and rewriting those notes. When reading books, underline key points. At the end of each chapter, write a small summary. Even better write a review of the book. Marx used to write a summary of every book he read. Studies on note taking (with control groups and reversal of note takers and non-note takers to eliminate differences) show overwhelmingly that note taking increases memory/retention. Many aspects of increased memory have been studied including; increased attention, immediate recall, delayed tests, free recall, MCQs, remembering important v less important knowledge, correlations with quality of notes and deeper learning. Bligh (2000) has detailed dozens of studies in this area. Wittrick and Alesandrini (1990) found that written notes increased learning by 30% through summaries and 22% using written analogies, compared to the control group. Why does note taking increase retention? First, increased focus, attention and concentration, the necessary conditions for learning. Second, increased attention to meaning and therefore better encoding. Thirdly, rehearsal and repetition, which processes it into long term memory. All three matter.


Practice is arguably the most powerful, yet most overlooked benefit in learning and true performance. Implemented properly and it is possible to have huge gains in productivity, namely the retention, recall and application of whatever has been learnt. One could go further and say that without a practice strategy, there is no learning strategy.


Allen, G. A., Mahler, W. A., & Estes, W. K. (1969). Effects of recall tests on long-term retention of paired associates. Journal of Verbal Learning and Verbal Behavior, 8, 463- 470. 

Anderson, J R. (1994) Learning and Memory: An Integrated Approach. John Wiley& Sons, NY, NY. 
Augé, M. (1995). Non-places: Introduction to an anthropology of supermodernity. London: Verso.

Anderson, R. (1981) Interference: The Relationship Between Response Latency and Response Accuracy. Journal of Experimental Psychology: Human Learning and Memory". 7, 326-343
Barnes, C.A. (1979) Memory deficits associated with senescence: A neurophysiological and behavioral study in the rat. J. Comp. Physiol. 43, 74-104
Bligh, D. A. (2000). What's the use of lectures?. San Francisco: Jossey-Bass.

Bjork R.A. (1975) Retrieval as a memory solidifier: an interpretation of negative recency and related phenomena. In RL Solso (Ed.) Information processing and cognition (123-124) Hillsdle, NJ: Erbaum
Bjork R.A. (1988) Retrieval practice and the maintenance of knowledge. In M.M. Gruneberg, P.E. Morris, and R.N. Sykes (Eds) Practical aspects of memory: current research and issues John Wiley& Sons, NY, NY.
Carnine D. (1989) Designing practice activities. J. Learning Disabilities, 22, 603-607
Demspter F.N. (1988) The spacing effect. A case study in the failure to apply the results of psychological research. Am. Psychol. 43, 627-634
Dudai, Y. (1996) Consolidation: fragility on the road to the engram. Neuron 17, 367-370.

Ebbinghaus, H. (1913) Memory: A contribution to experimental psychology (H.A. Ruger & C.E. Bussenues, Trans.). New York: Teachers college, Columbia University. (Originally published 1885). 

Ellenbogen J. Scientific American July 2006

Gates A.I. (1917) Recitation as a factor in memorizing. Arch. Phschol. 6, 40.
Hogan, R. M., & Kintsch, W. (1971). Differential effects of study and test trials on long- term recognition and recall. Journal of Verbal Learning and Verbal Behavior, 10, 562- 567. 

Keppel G. (1964) Facilitation in short- and long-term retention of paired associates following distributed practice in learning. J. Verbal Learn. and Verbal Behav. 3, 91-111
Kuiper, T.B & Kirker, W. S. (1977). Self-reference and the encoding of personal information. Journal of Personality & Social Psychology, 35(9), 677–688.

Kuo, T. M., & Hirshman, E. (1996). Investigations of the testing effect. American Journal of Psychology, 109, 451-464. 

Landauer T.K. and Bjork R.A. (1978) Optimal rehearsal patterns and name learning. In M. Gruneberg, P.E. Morris and R.N. Sykes (Eds). Practical aspects of memory (pp 625-632). London: Academic press. 

Moray, N. (1959) Attention in dichotic listening: affective cues and the influence of instructions. Quart. J. Exp. Psychology, 11, 56-60
Morris C.D., Brandsford J.D. and Franks J.J. (1977) Levels of processing versus transfer-appropriate processing. J. Verbal Learning and Verbal Behav. 16, 519-533
Nelson T.O. and Narens L. (1990) Metamemory: a theoretical framework and new findings. In G. Bower (Ed.) The psychology of learning and motivation (Vol. 26) New York: Academic Press.
Nelson T.O., Leonesio R.J. (1988) Allocation of self-paced study time and the "labor-in-vain effect". J Exp Psychol Learn Mem Cogn. 14, 676-86.

Nungester, R. J., & Duchastel, P. C. (1982). Testing versus review: Effects on retention. Journal of Educational Psychology, 74, 18-22. 

Syed, M. (2011). Bounce: The myth of talent and the power of practice.

Tulving E. and Madigan S.A. (1970) Memory and verbal learning. In P.H. Mussen and & M.R. Rosenweig (Eds.), Annual review of psychology. Palo Alto, CA
Wickelgren W.A. (1976) Memory storage dynamics. In W.K. Estes (Ed.), Handbook of learning and cognitive process. (Vol. 4) Hillsdale, NJ: Erlbaum

Wittrock, M.C. (1974). Learning as a generative process. Educational Psychologist, 11, 87-95. 

Wittrock, M.C. (1990). Generative processes of comprehension. Educational Psychologist, 24, 345-376. 

Wittrock, M.C, & Alesandrini, K. (1990). Generation of summaries and analogies and analytic and holistic abilities. American Educational Research Journal, 27, 489-5 Zechmeister, E. B., & Shaughnessy, J. J. (1980). When you know that you know and when you think that you know but you don’t. Bulletin of the Psychonomic Society, 15, 41-44. 

Friday, October 09, 2020

Engagement - most used yet most misunderstood word in learning?

Engagement is an odd word in learning. It is clearly a good thing to be engaged with a learning experience but just because people are engaged, does not mean they are learning. I’ve been totally engaged for years watching stand-up comedy experiences but can barely remember a single joke from any of them. Engagement can be a positive sign in learners but is no guarantee of learning. 

Learners can be:

·      engaged but not learning

·      engaged but simply going through things they already know

·      engaged but doing harm to learning 

·      engaged and learning

In other words, engagement can be a dangerous proxy for learning. One can be engaged and give the appearance of learning, without learning taking place, even inhibiting learning. What’s more, there may be little or no long-term or far transfer into actual application in the real world.

Engaged but not learning

Engagement while watching a video may feel like you are learning but the transience effect, fact that you have a 20 second window for attention and can only manipulate 3-4 things in working memory, mean that you are forgetting much of what you have just watched, no matter how engaged you felt. Your mind is like a shooting star burning up memories it leaves behind.

Engaged but simply going through things they already know

Learners can be comfortable skating superficially across content they are familiar with. This can also create the illusion of learning. Graham Nutall, in empirical classroom research, claimed that “In most of the classrooms we have studied, each student already knows about 40–50% of what the teacher is teaching”. Engagement with the easy and familiar is different from effortful engagement with the difficult and unfamiliar, the zone in which learning takes place.

Engaged but doing harm to learning

Then there is the engagement which is over-stimulating and prevents deeper processing, reflection and generation of ideas and retrieval from your own mind. Games can be a total distraction if engagement is with the mechanics of the game and not the learning.

Delusional learning

One thing that research in cognitive psychology has gifted to us over the last decade or so, is clear evidence that learners are delusional when it comes to judgements about what they think is engagement in their own learning. The big name in the field is Bjork, along with many other high quality researchers, who say that learning is “quite misunderstood (by learners)…. we have a flawed model of how we learn and remember”. There’s often a negative correlation between people’s judgements of their learning, what they think they have learnt and how they think they learn best - and what they’ve actually learnt and the way they can actually optimise their learning. In short, our own perceptions of learning can be seriously misleading. This is why ‘fun’ and ‘engagement’ can be misleading proxies. Also, why learner surveys and happy sheets are such bad measures of what is actually learnt and not helpful when designing optimal learning strategies and experiences.

Bjork warns us about ‘illusory learning’ and shows that learners are frequently mistaken about whether and what they have learned. In fact he has described them as ‘delusional’. Despite feeling as though they have been through a strong ‘learning experience’, that experience was the feeling that they had learned but when tested that feeling was misleading. Self-reported or data showing engagement is not, therefore, a reliable method for determining whether things were actually learned in the sense of finding a place in long-term memory and behaviour, the gold-standard for actual learning. It may just signal easy engagement, shallow experiences where the learner is engaged and entertained but does no deep learning. 

Desirable difficulty

Bjork thinks that both teaching and learning can be improved and optimised using techniques that force cognitive effort. Much learning is imagined, as it is too easy. Learning designers must therefore be careful not to allow too ‘easy’ engagement and likely to lead to forgetting. Learning experiences need to be challenging . 

Real, long-term learning requires ‘Desirable’ (accomplishable) and ‘Difficult’ learning with real effort, for high-retention. It is all in the struggle, failure and overcoming of difficulty that real effortful learning takes place. This is why so much online learning fails. To simply click on faces and see speech bubbles of text, drag and drop labels, choose true or false and so on, rarely constitutes desirable difficulty. This is click-through learning.

Desirable difficulties, are potent forms of learner engagement, such as the following learning techniques (just a selection):

·      Active retrieval practice

·      Allow failure

·      Interleaving

·      Generation

Active retrieval practice

To be specific about effortful learning, by effort we mean ‘active retrieval’ as the most powerful learning strategy at your disposal. The brain, the organ that named itself, is a unique organ in that it can test itself to see what it knows or doesn’t know. At the same time this act of retrieval consolidates has been found to be even more powerful than the original learning experience. 

The first solid research on retrieval was by Gates (1917), who tested children aged 8-16 on short biographies. Some simply re-read the material several times, others were told to look up and silently recite what they had read. The latter, who actively retrieved knowledge, showed better recall. Spitzer (1939) made over 3000 11-12 year olds read 600 word articles then tested students at periods over 2 months. The greater the gap between testing (retrieval) and the original exposure or test, the greater the forgetting. The tests themselves seemed to halt forgetting. Tulving (1967) took this further with lists of 36 words, with repeated testing and retrieval. The retrieval led to as much learning as the original act of studying. This shifted the focus away from testing as just assessment to testing as retrieval, as an act of learning in itself. Roediger et al. (2011) did a study on text material covering Egypt, Mesopotamia, India and China, in the real context of real classes in a real school, a Middle School in Columbia, Illinois. Retrieval tests, only a few minutes long, produced a full grade-level increase on the material that had been subject to retrieval. McDaniel (2011) did a further study on science subjects, with 16 year olds, on genetics, evolution and anatomy. Students who used retrieval quizzes scored 92% (A-) compared to 79% for those who did not. More than this, the effect of retrieval lasted longer, when the students were tested eight months later.

So to retrieve and recall what you need to know results in much higher levels of retention. Rather than read, re-read and underline, look away and try to retrieve and recall what you need to know. Repeatedly reading and underline text is not an affective learning experience, looking away from the page to try to recall what you think you know, is better.

This deliberate retrieval requires cognitive effort and that is why recall has a strong reinforcement effect and results in higher retention and subsequent recall. The deeper processing increases retention and subsequent recall. Note that this retrieval practice is not a test or assessment but a learning experience. In fact, it is one of the most powerful learning experiences. 

Rather than click on True or False or an option in a short list (MCQ), look away, think, generate, recall and come up with the answer. The key point here is that research has shown that retrieval is a memory modifier and makes your memory more recallable. Counter-intuitively, retrieval is much more powerful than being presented with the information. In other words it is more powerful than the original ‘teaching’ event.

Roediger and Karpicke (2006) researched studying v retrieval testing (without feedback). One week later the retrieval tested group did much better. They also asked them how much you are likely to remember in one week’s time for each method – oddly, the majority of learners got it completely wrong. Learners are notoriously delusional about learning strategies which is why learning experience design that simply panders to their wishes often fails.

Allow failure

Making errors is also a critical component of successful learning. Yet how often is failure avoided in online learning, even punished? We are so keen on ‘engagement’ and relentless positivity that there is no room for failure.

According to Kornell, Hayes and Bjork (2009), generating the wrong thing, then getting it right, leads to stronger learning. The reason is that you are activating the brain’s semantic network. Retrieval testing does better than reading or watching, as it potentiates recall. Work by Kornell (2009) shows that even unsuccessful testing is better that straight presentations. 

Retrieval testing gives you better internal feedback and works even when you get few or no correct answers. Testing, even before you have access to the material, as a learning experience, also helps learning. Once again, almost bizarrely, Heustler and Metcalfe (2012) asked learners what worked best and they were largely wrong.


Mixing things up, interleaving, boosts learning. Counterintuitively, blocking learning tasks and doing them in strict order “AAABBBCCC”, seems less efficient than interleaving learning tasks “ABCABCABC”. It would seem that interleaving keeps your brain on its toes with practice that increases desirable difficulty, each task being switched. You have to differentiate or discriminate, between concepts and tasks which strengthens memory associations.

A sizeable body of research, from sports to arts, medical and other subjects and skills, show its efficacy. Then idea is that you’re better off doing separate tasks either within a subject or in different subjects, rather than a series of similar tasks on one topic, subject or drill in say sport.It has also been researched in also in the real world context of the classroom in maths, Rohrer (2011), with astonishing increases in test scores (76%). It has even been shown to be useful in critical think tasks in legal training. 

Creating more diverse connections in the brain seems to help you cope with newer situations as they arise. The research suggests that some familiarity with the subject is needed before interleaving.


Generative learning is an active, effortful process where the learner generates meaning between what they already know and new knowledge, and then between new concepts and principles that are seen for the first time. This is an extension of rehearsal in the mind and retrieval practice.

Learning experiences that get the learner to generate their own learning and manifest their knowledge and skills, is better than simply reading text. It would also appear to provides the context for greater subsequent recall. Wittrock and Kelly (1984) showed that poor readers who generated relations, connections and headings had a 20% increase in comprehension, when they were asked to come up with a generative plan, comprehension jumped by 23%.

Open input by learners, is one example, which we implement in WildFire, the answers being interpreted by AI semantic analysis.


When learning experiences simply make things easy, in the classroom by low levels or superficial forms of participation or online with shallow clickthrough experiences, learning can be illusory. Learning professionals and learners often engage in practices that are sub-optimal, worse still, practices that may inhibit efficient learning. Bjork recommends techniques that may seem counterintuitive and difficult to implement, as they run counter to most current practice but the evidence suggest that he is right. Technology, of course, has also allowed retrieval practice, interleaving and spaced practice to be more easily implemented over time on a personalized basis.

This research shows that our methods of engagement in online learning are often sub-optimal. The problem we face is that immediate success often means long-term failure. More focus should be given to retrieval, generative learning, interleaving and spaced practice, not presentation or clicking on items and weak multiple-choice questions. We need to be presented with desirable difficulties, through partial or complete open input. This is exactly what we’ve spent the last two years building with WildFire.

There is, of course, a further step in engagement that really matters, and is often completely ignored – transfer.


Bjork, R. A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe & A. Shimamura (Eds.), Metacognition: Knowing about knowing (pp. 185–205). Cambridge, MA: MIT Press.

Ericsson, K. Anders, Krampe, Ralf Th. and Tesch-Romer. Clemens (1993) The Role of Deliberate Practice in the Acquisition of Expert Performance. Psychological Review, Vol. 100. No. 3, 363-406

Nuthall G (2007) The Hidden Lives of Learners. Wellington, NZ: NZCER Press.

Gates, A. I. (1917). Recitation as a factor in memorizing. Archives of Psychology, No. 40, 1-104. 

Spitzer, H. F. (1939). Studies in retention. Journal of Educational Psychology30, 641-656.

Tulving, E. (1967). The effects of presentation and recall of material in free-recall learning. Journal of Verbal Learning and Verbal Behavior6, 175􏰀184.

Roediger, H. L., Agarwal, P. K., McDaniel, M. A., & McDermott, K. B. (2011). Test-enhanced learning in the classroom: Long-term improvements from quizzing. Journal of Experimental Psychology: Applied, 17, 382-395.

McDaniel, M. A., Agarwal, P. K., Huelser, B. J., McDermott, K. B., & Roediger, H. L. (2011). Test-enhanced learning in a middle school science classroom: The effects of quiz frequency and placement. Journal of Educational Psychology, 103, 399-414

Roediger III, H.L. and Karpicke, J.D., 2006. The power of testing memory: Basic research and implications for educational practice. Perspectives on psychological science1(3), pp.181-210.

Kornell, N., Hays, M.J. and Bjork, R.A., 2009. Unsuccessful retrieval attempts enhance subsequent learning. Journal of Experimental Psychology: Learning, Memory, and Cognition35(4), p.989.

Kornell, N. and Son, L.K., 2009. Learners’ choices and beliefs about self-testing. Memory17(5), pp.493-501.

Huelser, B.J. and Metcalfe, J., 2012. Making related errors facilitates learning, but learners do not know it. Memory & cognition40(4), pp.514-527.

Pan, S.C., 2015. The interleaving effect: mixing it up boosts learning. Sci Am313(2).

Goode, S. and Magill, R.A., 1986. Contextual interference effects in learning three badminton serves. Research quarterly for exercise and sport57(4), pp.308-314.

Hatala, R.M., Brooks, L.R. and Norman, G.R., 2003. Practice makes perfect: the critical role of mixed practice in the acquisition of ECG interpretation skills. Advances in Health Sciences Education8(1), pp.17-26.

Kornell, N. and Bjork, R.A., 2008. Learning concepts and categories: Is spacing the “enemy of induction”?. Psychological science19(6), pp.585-592.

Rohrer, D., Dedrick, R. F., & Stershic, S. (2015). Interleaved practice improves mathematics learning. Journal of Educational Psychology, 107(3), 900–908.

Wittrock, M.C., 1992. Generative learning processes of the brain. Educational Psychologist27(4), pp.531-541.

Thursday, October 08, 2020

Emotion in Learning Experience Design - Norman's 3 facets; Visceral, Behavioural and Reflective...

Our lives would be impoverished without positive emotions such as fun, pleasure, joy, excitement; but also middling emotions of satisfaction, calmness, boredom; even negative emotions such as anger, sadness, melancholy and fear. They are ever-present and part of what it is to be human.

One facet of Learning Experience Design is to make the effort to engage the learner by injecting emotion into the experience. This does not mean blind emotion. Over-stimulation can be a bad thing in learning. The right kind of emotion is what helps learning, as this affective dimension can motivate the learner and aid retention. It is a matter of designing for both head and heart. It would be fair to say that most of what is written about learning design has a focus on cognition and understanding, whereas much of what drives us is feeling and emotion.

Some, like Jack Quatrell, at Learning Pool, literally say that Experience Design differs from more traditional design in being Emotional Design. He invokes Donald Norman, which is a good starting point, who in Emotional Design, broke emotional design down into three components:

Visceral (appearance)

Behavioural (performance)

Reflective (memories and experience)

Visceral (appearance)

This is the automatic, unconscious reaction we have to experiences. It is what Kahneman refers to as System 1 thinking in Thinking Fast and Slow. These reactions are fast, immediate without reflection. 

Branding and general art direction speak directly to these feelings. One practical shortcut is to copy and mimic the organisations branding guidelines, in terms of palette, font, general art direction and practices. Some organisations are very keen to get their branding and values reinforced in training. For Virgin, we implemented very strong branding across course from Values to Aircraft Maintenance, with their exact tones of red and fonts. This gives the training a visual and emotional organisational context.

An alternative is to match the branding and art direction to the topic being taught. It may be that a course on interpersonal skills will have to feel warm and friendly, whereas a course on process and procedures may need to be cool, crisp and procedural. If it is a serious scientific, financial or healthcare organisation, dealing with serious issue such as laboratory procedures, money laundering or cancer therapy, then casual cartoons may not be appropriate. Real world imagery and photography may create the right first impressions for such learning. 

Gestalt theorists have also identified this as the instant reaction to an interface or experience, rather than its components or mechanics. Gestalt principles are similar to many of the findings of researchers like Mayer in online learning; proximity, similarity, figure-ground, continuity, closure, and connection. The Gestalt Law of Proximity is often quoted in interface design and states that items close to each other are perceived as groups. This matters when you want to group navigation items (forward, back etc) separately from functional items (print, zoom etc). The Gestalt Law of Similarity states that items similar to each other will be grouped by the user. This can be used generally in interface and visual design. The Law of Figure-Ground is also important, where we see figure-ground effects

Much of this can be tested with real users in getting voiced reactions to specific questions on designs. So, keep in mind the visceral reaction of your target audience, not you the designer.

Behavioural (performance)

This is about emotion and feelings around actual use or usability. How easy is it to use in terms of speed of recognition, understanding and not making errors. Are navigational items in the optimal order? Are the icons clear? Are they big enough for smartphone touchscreens? Do they react when pressed to give feedback? Are they placed consistently in the same place on the screen? There is a massive amount of good practice in interface design around usability. It is vital that the interface is a simple, consistent, predictable and easy to use as possible, as time and cognitive effort spent on the interface detracts from the cognitive effort needed to learn.

There is also a large amount of established good practice around visual design of text, graphics, animation, audio and video. We know a lot about the different affordances of different media, and how to mix them, without inducing cognitive overload. The trick is to get the best out of each medium and media mix. 

Just as importantly, is the good practice around learning, which is often very different from other forms of screen presentation, such as the need for chunking, cues, repetition, summaries and so on. Cognitive overload is common in badly designed learning content, so a knowledge of good learning theory informs the behavioural side of design.

What you get learners to DO is also really important, as that is likely to be more powerful than what they see or hear. This is where experience design needs to include interaction with the mind, beyond just clicking on items to navigate or see pop-ups. Cognitive effort matters - a lot!  We must be very careful here. It is all too easy to make learning too easy. Without challenge, difficulty and cognitive effort, you will not have the deep processing necessary for learnt knowledge, skills and behaviour to stick. The learner will skate over the surface, thinking they have been learning, when, in fact, those experiences have been transitory. This illusory sense of learning is common and is reinforced when things feel easy. It is easy to watch a video and not realise that much of what you have just watched has left your memory before you have finished watching the entire video. It stays in working memory but never gets processed into long-term memory, so disappears. Interestingly what makes you feel as though you have learnt things is just those ‘feelings and emotions’. In this case emotion is our enemy, our greatest danger. This transitory effect is well-researched, real, common and measurable. Learning needs to be effortful.

Inducing emotion may be ideal when you want attitudinal shift in diversity, equality and other belief shift or self-awareness training but can be dangerous in non-affective training, where it can induce the illusion of learning. So, keep in mind the behavioural reaction of your target audience, not just what you the designer likes or may be familiar with.

Reflective (memories and experience)

This is an important set of feelings in learning, that you can rationalise, reflect, reinforce, recall and apply your learning. This is what Kahneman called System 2 thinking, the rational, reasoning side of the brain. We have feelings of achievement, success, confidence or having overcome difficulties in a learning experience that really do matter. It can be those feelings around having got there, not because you found it easy but realising that it was hard.

This is complex and involves much more than just getting a score on the assessment, although that can be an important feeling of success. Hence the frequent request to provide printed certificates for learners, even though they have no serious accreditation body behind them. It is similar to the status people attach to watches, handbags and branded clothes. You can be made to feel better by going through learning experiences that give you feelings of success and status.

Challenging cognitive effort can propel the learner forward and make them feel as though they really are making progress. Feedback is also a powerful accelerator of learning, so personalising learning and feedback can move things forward making the learner feel good about themselves.

One important facet of reflective feeling comes through the follow-up, actually doing something. This can be triggered by nudge learning, so that the learner gets their kicks through going back to their job and actually implementing a challenge, such as mentoring a younger employee or using those features in a spreadsheet. Satisfaction with real-world application of learning can bring high levels of satisfaction and accomplishment.

It is easy to forget that one learns for a reason, ultimately to apply that knowledge, so the transfer through to action really does matter. This is often quietly forgotten in online learning but as technology increasingly allows us to learn in the workflow, it is becoming more of a reality.

This can be tested through stickability of the learning experiences but also real assessment, not only of short-term accomplishment but long-term retention.


Interestingly, Norman thought Americans value Behavioural more than the Visceral & Reflective, whereas Europeans, tend to value the Visceral and Behavioural. This is fascinating. He claims that different people buy things with different fuel mixtures of the three types of emotions. This may well be true in learning and having delivered online learning in many different geographies and cultures, I think it is. These differences are real but the differences are getting less, as there is a global spread of online services, such as Google, social media, Wikipedia and Netflix which are now universal, but it is important to be aware of cultural differentials. 

So there are three levels of emotional or affective experience design. The visceral is that first impression, the overall and holistic feel of the experience. The behavioural comes from using the product both in navigation but also functionality and, importantly, in actual effortful, learning experiences. The reflective are those more rational and conscious feelings around achievement and success in learning. Beyond Norman, there are also other cognitive feelings around aesthetics, beauty, layout, space, colour and simplicity in design, that also count.

It is vital that we don’t just invoke emotions and feelings just for the sake of doing so. They matter in terms of engagement, stickiness but they must also be compatible with the actual acquisition of knowledge, skills and behaviour. Retention does matter, not just the feeling you have learned but the fact that you really did learn. Kahneman is right to remind us of the existence of fast and slow ways of thinking but he also warns us against the bias and mistakes that emotional and instinctive thinking brings in its wake. 


Norman, D.A., 2004. Emotional design: Why we love (or hate) everyday things. Basic Civitas Books.

Kahneman, D. and Patrick, E., 2011. Thinking, fast and slow. Allen Lane.

Miller, G., 2009. Spent: Sex, evolution, and consumer behavior. Penguin.