Breakthrough that literally opens up online learning? Using AI for free text input

When teachers ask learners whether they know something they rarely ask them multiple choice questions. Yet the MCQ remains the staple in online learning, even at the level of scenario based learning. Open input remains rare. Yet there is ample evidence that it is superior in terms of retention and recall. Imagine allowing learners to type, in their own words, what they think they know about something, and AI does the job of interpreting that input?

Open input

We’ve developed different levels of more natural open input that takes online learning forward. The first involves using AI to identify the main concepts and getting learners to enter the text/numbers, rather than choosing from a list. The cognitive advantage is that the learner focuses on recalling the idea into their own minds, an act that has been shown to increases retention and recall. There is even evidence that this type of retrieval has a stronger learning effect than the original act of being taught the concept. These concepts then act a ’cues’ which learners hang their learning upon, for recall. We know this works well.

Free text input

But let’s take this a stage further and try more general open input. The learners reads a number of related concepts in text, with graphics, even watching video, and has to type in a longer piece of text, in their own words. This we have also done. This longer form of open-input allows the learner to rephrase and generate their thoughts and the AI software does analysis on this text.

Ideally, one takes the learner through three levels:

1. Read text/interpret graphics/watch video

2. AI generated open-input with cues

3. AI generated open-input of fuller freeform text in your own words

This gives us a learning gradient in terms and of increasing levels of difficulty and retrieval. You move from exposure and reflection, to guided effortful retrieval and full, unaided retrieval. Our approach increases the efficacy of learning in terms of speed of learning, better retrieval and better recall, all generated and executed by AI.

The process of interpretation on the generated text, in your own words, copes with synonyms, words close in meaning and different sentence constructions, as it uses the very latest form of AI. It also uses the more formal data from the structured learning. We have also got this working by voice only input, another breakthrough in learning, as it is a more natural form of expression in practice. 

The opportunities for chatbots is also immense. 

If you work in corporate learning and want to know more, please contact us at WildFire and we can show you this in action
.

Evidence for this approach

Much advice and most practice from educational institutions – re-reading, highlighting and underlining – is wasteful. In fact, these traditional techniques can be dangerous, as they give the illusion of mastery. Indeed, learners who use reading and re-reading show overconfidence in their mastery, compared to learners who take advantage of effortful learning.

Yet significant progress has been made in cognitive science research to identify more potent strategies for learning. The first strategy, mentioned as far back as Aristotle, Francis Bacon then William James, is ‘effortful’ learning. It is what the learner does that matters. 

Simply reading, listening or watching, even repeating these experiences, is not enough. The learning is in the doing. The learner must be pushed to make the effort to retrieve their learning to make it stick in long-term memory.

Active retrieval

 ‘Active retrieval’ is the most powerful learning strategy, even more powerful than the original learning experience.  The first solid research on retrieval was by Gates (1917), who tested children aged 8-16 on short biographies. Some simply re-read the material several times, others were told to look up and silently recite what they had read. The latter, who actively retrieved knowledge, showed better recall. Spitzer (1939) made over 3000 11-12 year olds read 600 word articles then tested students at periods over 2 months. The greater the gap between testing (retrieval) and the original exposure or test, the greater the forgetting. The tests themselves seemed to halt forgetting. Tulving (1967) took this further with lists of 36 words, with repeated testing and retrieval. The retrieval led to as much learning as the original act of studying. This shifted the focus away from testing as just assessment to testing as retrieval, as an act of learning in itself. Roediger et al. (2011) did a study on text material covering Egypt, Mesopotamia, India and China, in the real context of real classes in a real school, a Middle School in Columbia, Illinois. Retrieval tests, only a few minutes long, produced a full grade-level increase on the material that had been subject to retrieval. McDaniel (2011) did a further study on science subjects, with 16 year olds, on genetics, evolution and anatomy. Students who used retrieval quizzes scored 92% (A-) compared to 79% for those who did not. More than this, the effect of retrieval lasted longer, when the students were tested eight months later. So designing learning as a retrieval learning experience, largely using open-input, where you have to pull things from your memory and make a real effort to type in the missing words, given their context in a sentence.

Open input

Most online learning relies heavily on multiple-choice questions, which have become the staple of much e-learning content. These have been shown to be effective, as almost any type of test item is effective to a degree, but they have been shown to be less effective than open-response, as they test recognition from a list, not whether it is actually known.

Duchastel and Nungester (1982) found that multiple-choice tests improve your performance on recognition in subsequent multiple-choice tests and open input improves performance on recall from memory. This is called the ‘test practice effect’. Kang et al. (2007) showed that, with 48 undergraduates, reading academic Journal quality material, open input is superior to multiple-choice (recognition) tasks. Multiple choice testing had an affect similar to that of re-reading whereas open-input resulted in more effective student learning. McDaniel et al. (2007) repeated this experiment in a real course with 35 students enrolled in a web-based Brain and Behavior course at the University of New Mexico. The open-input quizzes produced more robust benefits than multiple-choice quizzes. ‘Desirable difficulties’ is a concept coined by Elizabeth and Robert Bjork, to describe the desirability of creating learning experiences that trigger effort, deeper processing, encoding and retrieval, to enhance learning. The Bjorks have researched this phenomenon in detail to show that effortful retrieval and recall is desirable in learning, as it is the effort taken in retrieval that reinforces and consolidates that learning. A multiple-choice question is a test of recognition from a list. They do not elicit full recall from memory. Studies comparing multiple-choice with open retrieval show that when more effort is demanded of students, they have better retention.As open-response takes cognitive effort, the very act of recalling knowledge also reinforces that knowledge in memory. The act of active recall develops and strengthens memory. It improves the process of recall in ways that passive recall – reading, listening and watching do not.

Design implications

Meaning matters and so we rely primarily on reading and open response, where meaningful recall is stimulated. This act alone, even when you don’t know the answer, is a strong reinforcer, stronger indeed, than the original exposure. Interestingly, even when the answer is not known, the act of trying to answer is also a powerful form of learning. 

So, the deliberate choice of open-response questions, where the user types in the words, then more  substantial open input, is a deliberate, design strategy to take advantage of known AI and learning techniques to increase recall and retention. Note that no learner is subjected to the undesirable difficulty of getting stuck, as letters are revealed one by one, and the answer given after three attempts. Hints are also possible in the system.

Bibliography

Bjork, R. A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe & A. Shimamura (Eds.), Metacognition: Knowing about knowing (pp. 185–205). Cambridge, MA: MIT Press.

Bower G. H. (1972) Mental imagery in associative learning in Gregg L,W. Cognition in learning and memory New York, Wiley

Gardener (1988) Generation and priming effects in word fragment completion Journal of Experimental Psychology: Learning, Memory and Cognition 14, 495-501

Butler, A. C., & Roediger, H. L. (2008). Feedback enhances the positive effects and reduces the negative effects of multiple-choice testing. Memory & Cognition, 36, 604-616.

Duchastel, P. C., & Nungester, R. J. (1982). Testing effects measured with alternate test forms. Journal of Educational Research, 75, 309-313.

Gardener (1988) Generation and priming effects in word fragment completion Journal of Experimental Psychology: Learning, Memory and Cognition 14, 495-501

Gates, A. I. (1917). Recitation as a factor in memorizing. Archives of Psychology, No. 40, 1-104. 

Hirshman, E. L., & Bjork, R. A. (1988). The generation effect: Support for a two-factor theory. Journal of Experimental Psychology: Learning, Memory, & Cognition, 14, 484–494. 

Jacoby, L. L. (1978). On interpreting the effects of repetition: Solving a problem versus remembering a solution. Journal of Verbal Learning and Verbal Behavior, 17, 649-667.

Kang, S. H. K., McDermott, K. B., & Roediger, H. L., III. (2007). Test format and corrective feedback modulate the effect of testing on long-term retention. European Journal of Cognitive Psychology, 19, 528-558. 

McDaniel, M. A., Einstein, G. O., Dunay, P. K., & Cobb, R.  (1986).  Encoding difficulty and memory:  Toward a unifying theory.  Journal of Memory and Language, 25, 645-656.

McDaniel, M. A., Agarwal, P. K., Huelser, B. J., McDermott, K. B., & Roediger, H. L. (2011). Test-enhanced learning in a middle school science classroom: The effects of quiz frequency and placement. Journal of Educational Psychology, 103, 399-414

Miller, G.A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63, 81-97. 

Richland, L. E., Bjork, R. A., Finley, J. R., & Linn, M. C. (2005). Linking cognitive science to education: Generation and interleaving effects. In B. G. Bara, L. Barsalou, & M. Bucciarelli (Eds.), Proceedings of the twenty-seventh annual conference of the cognitive science society. Mahwah, NJ: Erlbaum. 

Roediger, H. L., Agarwal, P. K., McDaniel, M. A., & McDermott, K. B. (2011). Test-enhanced learning in the classroom: Long-term improvements from quizzing. Journal of Experimental Psychology: Applied, 17, 382-395.

Spitzer, H. F. (1939). Studies in retention. Journal of Educational Psychology, 30, 641-656. 

Tulving, E. (1967). The effects of presentation and recall of material in free-recall learning. Journal of Verbal Learning and Verbal Behavior, 6, 175􏰀184.

AI-driven speech may revolutionise online learning

Over the last year or so we built the world’s first AI content creation service, WildFire, which creates online learning in minutes not months, at a much lower cost and with high retention. The reason for claiming it is high retention, is that we largely abandoned multiple-choice-questions for open-input, making the learner think, recall and actively input their thoughts. It is this ‘effortful’ learning that really matters in learning. This worked well and we have delivered online learning on factual knowledge, high-end academic content, processes, procedures and management content to a range of audiences in large organisations, from apprentices to high-end clinicians, in finance, healthcare, travel and manufacturing. Having seen how well open-input worked, we turned our attention to the use of AI to go several steps further and improve the interface. What if the learner could simply speak the answers? ... it was a revelation.

Speech

We learn to speak almost effortlessly, whereas, writing takes many years. So why not exploit what we do everyday in our lives - use speech input. It was thought that women spoke much more than men, a myth started in The Female Brain, by Louann Brizendine, who claimed that, whereas women spoke on average 20,000 words a day, it was only 6,000 for men. This proved to be nonsense. An actual study, at the University of Arizona by Mehl (2007), using an electronic recording device to sample everyday speech from 396 people found that we speak, on average, around 16,000 words a day, with no significant difference between men and women.

This is much greater than the average for writing. So it makes sense to use speech in learning. Consider also that if spelling is not part of the learning, you eliminate problems around misspelling, especially for those who are nervous on that score or who may have dyslexia. On top of this you do not have to make the physical effort to move a cursor around the screen into a field, then physically type.

Retention

However, it is interesting to compare different forms of input in terms of retention. So, you think of an answer, then:

   CHOOSE your answer from a list (multiple choice)

   TYPE your answer

   SPEAK your answer

CHOOSE (MCQs)

It is clear that simply clicking on an already provided answer from a list is the least effective of the three. The answer is there in front of your eyes, you have a 25% chance of getting it right without knowing anything, questions are often designed so that you can guess and the distractors are often remembered rather then the correct answers. It makes you wonder why the online learning industry is so wedded to MCQs.

TYPE

This has the advantage of making you recall the answer into your brain first (a powerful reinforcement event), then actively type in the answer, another reinforcement event, without having been given the answer or suffering from the drawbacks of simply choosing from a pre-written list. We have found this to be a much more powerful way of learning.

SPEAK

Things get interesting here, as you are communicating directly, without any of the artificiality of choosing from a list or typing. My initial impression (not based on any studies) is that this may be even better. Being hands free, your attention and cognitive focus is entirely on thinking and expressing your thoughts. None of your cognitive bandwidth is taken up by moving the cursor, typing and letter-by-letter spelling. You get a focus on meaning but there’s an additional advantage, as you get more of a flow and the learning is faster.

Podcasts

Using another form of AI, text to speech, we can also, automatically, create podcasts. This is built into the service. Simply tick a box and your online learning will create a podcast of the module or page by page speech. This is a useful supplement to the active learning.

Context

In addition, as we have audio only learning, including navigation, using the words, NEXT, BACK, GO and SCROLL, so we can place the learning experience within VR, which we have done, instantly and cheaply. We know that context helps retention, so speech input allows a further level of retention to be achieved. This is getting interesting in say, training fror healthcare professionals in a hospital or cabin crew inside an aircraft.

Conclusion

Simultaneously, using different forms of AI, we hope to have increased the efficacy of online learning by the:

1. Superfast creation of content

2. Higher retention open-input

3. Higher retention speech input

4. Automatically created podcasts

5. Full 3D VR delivery

All at lower costs and far greater speed than traditional and expensive methods. If you are interested we can show you all of this by Skype. Contact us here.

Bibliography

Mehl M (2007). Are Women Really More Talkative Than Men? Science ,Vol. 317, Issue 5834, pp. 82

Does ‘Design thinking’ lead to bad learning design?

Fads come and go, and ‘Design Thinking’ seems to be one on the rise at the moment. It’s a process with lots of variants but, in the talks I’ve seen on the subject, and the results I’ve seen emerge from the process, I’m not wholly convinced. The problem is that we may well need less ‘design’ and more ‘thinking’.  The combination is likely to dumb down the learning in favour of superficial design. Imagine applying this theory to medicine. You wouldn’t get far by simply asking patients what they need to cure their problems, you need a growing body of good research tried and tested methods, and expertise. So let’s break the Design Thinking process down to see how it works in practice and examine the steps one by one.

Empathise

Donald Norman, in Design Thinking though that empathy in design was wrong-headed, a waste of time and energy and that the designers time would be better spent understanding the task. He is right. It is a conceit.

Donald Norman says, of this call for empathy in design, that “the concept is impossible, and even if possible, wrong”. I was seeing empathy used in pieces that actually mentioned Norman as one of their heroes! Yet here he was saying it was wrong-headed. He is absolutely right. There is no way you can put yourself into the heads of the hundreds, thousands, even tens and hundreds of thousands of learners. As Norman says “It sounds wonderful but the search for empathy is simply misled.” Not only is it not possible to understand individuals in this way, it is just not that useful.

It is not empathy but data you need. Who are these people, what do they need to actually do and how can we help them. As people they will be hugely variable but what they need to know and do, in order to achieve a goal, is relatively stable. This has little to do with empathy and a lot to do with understanding and reason.
All too often we latch on to a noun in the learning world without thinking much about what it actually means, what experts in the field say about it and bandy it about as though it were a certain truth. This is the opposite of showing empathy. It is the rather empty use of language.

Empathy for the learner is an obvious virtue but what exactly does that mean? For years, in practice, this meant Learning Styles. For many it still is Learning Styles, being sensitive to learner’s differences, diversity and needs in terms of preferences. This, of course has been a disastrous waste of time, as research has shown. Other faddish outcomes over-sensitive to supposed learner needs have been Myers-Briggs, NLP and no end of faddish ideas about what we ‘think’ learners need, rather than what research tells us they actually benefit from.

Research in cognitive psychology has given us clear evidence that learners are often mistaken when it comes to judgements about their own learning. Bjork, along with many other high quality researchers, have shown that learning is “quite misunderstood (by learners)…. we have a flawed model of how we learn and remember”. There’s often a negative correlation between people’s judgements of their learning, what they think they have learnt, how they think they learn best - and what they’ve ‘actually’ learnt and the way they can ‘actually’ optimise their learning. In short, our own perceptions of learning are seriously delusional. This is why engagement, fun, learner surveys and happy sheets are such bad measures of what is actually learnt and the enemy of optimal learning strategies. In short, empathy and asking learners what they want can seriously damage design.
In truth replacing a good needs analysis, including a thorough understanding of your target audience is not bettered by calling it empathy. That is simply replacing analysis with an abstract word to make it sound more in tune with the times.

Define

Identifying learner needs and problems has led to a ton of wasteful energy spent on slicing them up into digital natives/immigrants and personas that often average out differentiation and personalisation. The solution is not to identify ideal learners as personas but provide sophisticated pedagogic approaches that are adaptive and provide personal feedback. Design thinking makes the mistake of thinking there is such a thing as ideal learners without realising that you need analysis of the target audience, not ‘averaged out’ personas.

Design Thinking seems to push people towards thinking that learning problems are ‘design’ problems. Many are not. You need top understand the nature of the cognitive problems and researched solutions to those problems. By all means define the problems but those problems but know what a learning problem is.

One area, however, where I think design thinking could be useful is in identifying the context, workflow and moments of need. So, understanding the learner’s world, their business environment. That’s fine. On this I agree, But I rarely hear this from practising ‘Design Thinking’ practitioners, who tend to focus on the screen design itself, rather than design of a blended learning experience, based on the types of learning to be delivered in real environments, in the workflow with performance support. You need a deep understanding of the technology and its limitations.
There is also an argument for having a compete set of skills on the team but this has nothing to do with design thinking. The delivery of online learning is a complex mix of learning, design, technical, business and fiscal challenges. What's needed is balance in the team not a process that values an abstract method with a focus on 'design' alone.

Ideate

This is the key step, where design thinkers are supposed to provide challenge and creative solutions. It is the step where it can all go wrong. Creative solutions tend to be based on media delivery, not effortful learning, chunking, interleaving, open input, spaced practice and many other deeper pedagogic issues that need to be understood before you design anything. There’s often a dearth of knowledge about the decades of research in learning and cognitive science that should inform design. It is replaced by rather superficial ideas around media production and presentation, hence the edutainment we get, all ‘tainment’ and no ‘edu’. It focuses on presentation not effortful learning.

Few design thinkers I’ve heard show much knowledge of designing for cognitive load, avoiding redundancy and have scant knowledge of the piles of brilliant work done by Nass, Reeves, Mayer, Clark, Roediger, MacDaniel and many other researchers who have worked for decades uncovering what good online learning design requires. This is also why co-design is so dangerous. It leads to easy learning, all front and no depth.

What I’ve seen is lots of ‘ideation’ around gamification (but the trivial, Pavlovian aspects of games - scoring, badges and leaderboards). Even worse is the over-designed, media rich, click-through learning, loosely punctuated by multiple-choice questions. Remember that media rich does not mean mind-rich. Even then, designers rarely know the basic research, for example, on the optimal number of options in MCQs or that open input is superior.

Protoype

It is easy to prototype surface designs and get voiced feedback on what people like but this is a tiny part of the story. It is pointless prototyping learning solutions in the hope that you’ll uncover real learning efficacy (as opposed to look and feel) without evaluating those different solutions. This means the tricky and inconvenient business of real research, with controls, reasonable sample sizes, randomly selected learners and clear measurement of retention in long-term memory, even transfer. Few with just ‘Design Thinking’ skills have the skills, time and budget to do this. This is why we must rely on past research and build on this body of knowledge, just as clinicians do in medicine. We need to be aware of the work of Bjork, Roediger, Karpicke, Heustler and Metcalfe, who show that asking learners what they think is counterproductive. And build on the research that shows what techniques work for high retention.

A problem is that prototyping is often defined by the primitive tools used by learning designers, that can only produce presentation-like, souped-up Powerpoint and MCQs, whereas real learning requires much deeper structures. Few have made the effort to explore tools that allow open input and free text input, which really does increase retention and recall. Low fidelity prototyping won’t hack it if you want open input and sophisticated adaptive and personalised learning through AI – and that’s where things are heading.

One area that Design Thinking can help is with the ‘user interface’ but this is only one part of the deliverable and often not that important. It is important to make it as frictionless as possible but this comes as much through technical advances, touchscreen, voice, open input, than design.

Test

Testing is a complex business. I used to run a large online learning test lab – the largest in the UK. We tested for usability, accessibility, quality assurance and technical conformance and, believe me, to focus just on ‘design’ is a big mistake. You need to focus not on surface design but, more importantly, on all sorts of things, such as learning efficacy. Once again, learner testimony can help but it can also hinder. Learners often report illusory learning when they are presented with high quality media – this means absolutely nothing. Testing is pointless if you’re not testing the real goal – actual retained learning. Asking people for qualitative opinions does not do that.
In truth testing is quite tricky. You have to be clear about what you are testing, cover everything and have good reporting. There are tried and tested methods, that few have ever studied, so this is a really weak link. Just shoving something under the nose of a learner is not enough. We found early on that it is a short number of iterations with an expert that really works with interface design, along with A/B testing. Not some simple suck it and see trial.

Conclusion

I've heard several presentations on this and done the reading but my reaction is still the same. Is that it? It seems like a short-circuited version of a poor, project management course. I honestly think that the danger of ‘Design Thinking’ is that it holds us back.  We’ve had this for several years now, where design trumps deep thinking, knowledge of how we learn, knowledge of cognitive overload and knowledge of optimal learning strategies. It gives us the illusion of creativity but at the expense of sound learning. Walk around any large online learning exhibition and observe the output – over-engineered design that lacks depth. Design thinking lures us into thinking that we have solved learning problems when all we have done is polish presentation. The real innovations I’ve seen come from a deep understanding of the research, technology and innovative solutions based on that research, like nudge learning and WildFire. Delivery, I think, is better rooted in strong practices, such as ISO standards and practices guided by evidence, which have evolved over time and not simplistic processes that are often simplified further and sold as bromides. As one commentator, who tried Design Thinking, said "we ended up doing nothing more than polishing turds!".

Has the old 'graphics-text-graphics-text-MCs had its day? Evidence that effortful, open-response learning much better

To what degree does contemporary online learning reflect contemporary learning theory? The old paradigm of graphic-text-MCQ is way out of line with recent (and past) learning theory, so that, no matter how much glitz, animation and graphics you produce, the fundamentals of retention and recall are ignored. On a quick walk round the Learning technologies show this year I saw much the same as I've seen for the last 30 years, some worse. The same old, over-engineered content that takes months to make at high cost but with low retention value.

So, over the last year or so I’ve been working on online learning (WildFire) that largely abandons the multiple choice question, with lots of graphics production and glitz, for a more stripped-down approach that focuses on effortful learning. I wanted to produce content quickly, in minutes not months, cheaply (at least 80% cheaper) and to a higher quality than existing online learning (based on retention and recall).

Science of learning

There are good and bad ways to learn. Unfortunately much of what feels intuitive, is in fact, wrong. The science of learning has shown that researched, counterintuitive strategies, often ignored in practice, produce optimal learning.

For example, much advice and most practice from educational institutions – re-reading, highlighting and underlining – is wasteful. In fact, these traditional techniques can be dangerous, as they give the illusion of mastery. Indeed, learners who use reading and re-reading show overconfidence in their mastery, compared to learners who take advantage of effortful learning.

Yet significant progress has been made in cognitive science research to identify more potent strategies for learning. The first strategy, mentioned as far back as Aristotle, Francis Bacon then William James, is ‘effortful’ learning. It is what the learner does that matters.

Simply reading, listening or watching, even repeating these experiences, is not enough. The learning is in the doing. The learner must be pushed to make the effort to retrieve their learning to make it stick in long-term memory. This one act is the best defence against the brain’s other propensity, identified by Ebbinghaus in 1885 – forgetting.

With this in mind, I wanted to design learning experiences to deliver good learning using some fundamental principles in researched learning theory:

1. Active retrieval

2. Multiple-choice v open-response

3. Typing in words

1. Active retrieval

To be specific about effortful learning, by effort I mean ‘active retrieval’ as the most powerful learning strategy at your disposal. The brain, the organ that named itself, is a unique organ in that it can test itself to see what it knows or doesn’t know. At the same time this act of retrieval consolidates has been found to be even more powerful than the original learning experience.

Study 1 - Gates

The first solid research on retrieval was by Gates (1917), who tested children aged 8-16 on short biographies. Some simply re-read the material several times, others were told to look up and silently recite what they had read. The latter, who actively retrieved knowledge, showed better recall.

Study 2 - Spitzer

Spitzer (1939) made over 3000 11-12 year olds read 600 word articles then tested students at periods over 2 months. The greater the gap between testing (retrieval) and the original exposure or test, the greater the forgetting. The tests themselves seemed to halt forgetting.

Study 3 - Tulving

Tulving (1967) took this further with lists of 36 words, with repeated testing and retrieval. The retrieval led to as much learning as the original act of studying. This shifted the focus away from testing as just assessment to testing as retrieval, as an act of learning in itself.

Study 4 – Roediger

Roediger et al. (2011) did a study on text material covering Egypt, Mesopotamia, India and China, in the real context of real classes in a real school, a Middle School in Columbia, Illinois. Retrieval tests, only a few minutes long, produced a full grade-level increase on the material that had been subject to retrieval.

Study 5 – McDaniel

McDaniel (2011) did a further study on science subjects, with 16 year olds, on genetics, evolution and anatomy. Students who used retrieval quizzes scored 92% (A-) compared to 79% for those who did not. More than this, the effect of retrieval lasted longer, when the students were tested eight months later.

Design implications

So I’ve been designing learning as a retrieval learning experience, largely using open-input, where you have to pull things from your memory and make a real effort to type in the missing words, given their context in a sentence. First, as the research shows, this tells you what you know, half-know or don’t know. Second, it consolidates what you know in long-term memory. Third, it encourages you to improve your performance.

2. Open-response v multiple-choice

Most online learning relies heavily on multiple-choice questions, which have become the staple of much e-learning content. These have been shown to be effective, as almost any type of test item is effective to a degree, but they have been shown to be less effective than open-response, as they test recognition from a list, not whether it is actually known.

Study 1 - Duchastel and Nungester

Duchastel and Nungester (1982) found that multiple-choice tests improve your performance on recognition in subsequent multiple-choice tests and open input improves performance on recall from memory. This is called the ‘test practice effect’.

Study 2 – Kang

Kang et al. (2007) showed that, with 48 undergraduates, reading academic Journal quality material, open input is superior to multiple-choice (recognition) tasks. Multiple choice testing had an affect similar to that of re-reading whereas open-input resulted in more effective student learning.

Study 3 – McDaniel

McDaniel et al. (2007) repeated this experiment in a real course with 35 students enrolled in a web-based Brain and Behavior course at the University of New Mexico. The open-input quizzes produced more robust benefits than multiple-choice quizzes.

Study 4 - Bjork

‘Desirable difficulties’ is a concept coined by Elizabeth and Robert Bjork, to describe the desirability of creating learning experiences that trigger effort, deeper processing, encoding and retrieval, to enhance learning. The Bjorks have researched this phenomenon in detail to show that effortful retrieval and recall is desirable in learning, as it is the effort taken in retrieval that reinforces and consolidates that learning.

Design implications

A multiple-choice question is a test of recognition from a list. They do not elicit full recall from memory. Studies comparing multiple-choice with open retrieval show that when more effort is demanded of students, they have better retention.. As open-response takes cognitive effort, the very act of recalling knowledge also reinforces that knowledge in memory. The act of active recall develops and strengthens memory. It improves the process of recall in ways that passive recall – reading, listening and watching do not. Active recall, pulling something out of memory, is therefore more effective in terms of future performance.

Note that multiple-choice questions are useful for situations where they are deemed necessary, for example, in common misconceptions or lists, which is why they are also used but not the dominant form of learning.

3. Typing in words

A fascinating finding by Jacoby was the precise identification of filling in the missing letters of a word as a powerful act of consolidation of memory. This is called the ‘generation effect’. In other words, only a relatively small amount of retrieval effort is needed to have a powerful effect on memory.

Study 1 – Jacoby

Jacoby (1978) uncovered the fact that cramming led to short-term gains but long-term forgetting. Learners achieved high scores on the first, immediate test but forgot 50% in subsequent tests, compared to those who retrieved material, and forgot only 13%. He also showed that, in presenting word pairs, where some learners got the entire word pair, others got one word with two or more letters deleted from the interior of that word, a bit like an unfinished crossword, the simple act of filling-in-the-blanks resulted in higher retention and recall.

Study 2 – McDaniel

McDaniel et al (1986) also pinpointed leaving out letters as a way to stimulate retrieval. Learners were asked to fill-in-the-blank missing letters in Fairy Tales and showed significant gains in recall. The cognitive effort required to complete the terms strengthened retention and recall.

Study 3 – Hirshman & Bjork

Hirshman & Bjork (1988) got learners to type in the missing letters in words (salt-p_pp_r), which resulted in higher retention rates for conceptual pairs than the words being read on their own.

Study 4 – Richland

Richland et al. (2005) took this research into a real world environment and showed similarly positive effects. They concluded that it is the effortful engagement in the process of retrieval that leads to better recall.

Design implications

Meaning matters and so I rely primarily on reading and open response, where meaningful recall is stimulated. This act alone, even when you don’t know the answer, is a strong reinforcer, stronger indeed, than the original exposure. Interestingly, even when the answer is not known, the act of trying to answer is also a powerful form of learning.

So, the deliberate choice of open-response questions, where the user types in the words until they are correct, is a deliberate, design strategy to take advantage of known techniques to increase recall and retention. Note that no learner is subjected to the undesirable difficulty of getting stuck, as letters are revealed one by one, and the answer given after three attempts. Hints are also possible in the system.

For more information on WildFire click here.

Bibliography

Bjork, R. A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe & A. Shimamura (Eds.), Metacognition: Knowing about knowing (pp. 185–205). Cambridge, MA: MIT Press.

Bower G. H. (1972) Mental imagery in associative learning in Gregg L,W. Cognition in learning and memory New York, Wiley

Gardener (1988) Generation and priming effects in word fragment completion Journal of Experimental Psychology: Learning, Memory and Cognition 14, 495-501

Butler, A. C., & Roediger, H. L. (2008). Feedback enhances the positive effects and reduces the negative effects of multiple-choice testing. Memory & Cognition, 36, 604-616.

Duchastel, P. C., & Nungester, R. J. (1982). Testing effects measured with alternate test forms. Journal of Educational Research, 75, 309-313.

Gardener (1988) Generation and priming effects in word fragment completion Journal of Experimental Psychology: Learning, Memory and Cognition 14, 495-501

Gates, A. I. (1917). Recitation as a factor in memorizing. Archives of Psychology, No. 40, 1-104.

Hirshman, E. L., & Bjork, R. A. (1988). The generation effect: Support for a two-factor theory. Journal of Experimental Psychology: Learning, Memory, & Cognition, 14, 484–494.

Jacoby, L. L. (1978). On interpreting the effects of repetition: Solving a problem versus remembering a solution. Journal of Verbal Learning and Verbal Behavior, 17, 649-667.

Kang, S. H. K., McDermott, K. B., & Roediger, H. L., III. (2007). Test format and corrective feedback modulate the effect of testing on long-term retention. European Journal of Cognitive Psychology, 19, 528-558.

McDaniel, M. A., Einstein, G. O., Dunay, P. K., & Cobb, R.  (1986).  Encoding difficulty and memory:  Toward a unifying theory.  Journal of Memory and Language, 25, 645-656.

McDaniel, M. A., Agarwal, P. K., Huelser, B. J., McDermott, K. B., & Roediger, H. L. (2011). Test-enhanced learning in a middle school science classroom: The effects of quiz frequency and placement. Journal of Educational Psychology, 103, 399-414

Miller, G.A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63, 81-97.

Richland, L. E., Bjork, R. A., Finley, J. R., & Linn, M. C. (2005). Linking cognitive science to education: Generation and interleaving effects. In B. G. Bara, L. Barsalou, & M. Bucciarelli (Eds.), Proceedings of the twenty-seventh annual conference of the cognitive science society. Mahwah, NJ: Erlbaum.

Roediger, H. L., Agarwal, P. K., McDaniel, M. A., & McDermott, K. B. (2011). Test-enhanced learning in the classroom: Long-term improvements from quizzing. Journal of Experimental Psychology: Applied, 17, 382-395.

Spitzer, H. F. (1939). Studies in retention. Journal of Educational Psychology, 30, 641-656.

Tulving, E. (1967). The effects of presentation and recall of material in free-recall learning. Journal of Verbal Learning and Verbal Behavior, 6, 175􏰀184.

Curious story of the first Teaching Machines... did Skinner really rear his daughter in a box?

Mention Skinner and people will tell you of how he put his daughter in a box, reared her with stimuli for food and that she then sued him, suffered from mental illness and eventually committed suicide. None of this is true and when Lauren Slater wrote a book, Opening Skinner's Box, claiming it was, Deborah Skinner, his actual daughter, an artist who lives in England, wrote a scathing article in The Guardian saying it was all hogwash. Every few years a book of this sort pops out, from people who want to make exaggerated claims about the malign influence of technology in learning.

In fact, Skinner's Box was actually the device in which he trained rats and pigeons. Some critics mistake this for the air-conditioned crib Skinner designed, designed to keep a child warm. It had nothing to do with teaching or learning. Quite another of Skinner's activities were his 'Teaching Machines'. These also arouse strong reactions. Yet they were a product of their time, simply mechanical and actually quite ingenious, as they did what few instructional designers do today and that's accept open input by the learner. They were not the first, that was Pressey decades earlier, with his multiple choice questions.

Teaching machines didn’t appear in a vacuum. These were serious psychologists who based their designs on deeply held beliefs about learning theory. They were created, unsurprisingly, during the behaviourist era, by quirky academics with strong views. It was also a period of technological and mass manufacturing. Yet, oddly, the educational system and manufacturers remained stubbornly immune to their charm. So, despite all the fuss, nothing really happened on scale and few who work in technology for learning see these machines as having had real influence on their work. Nevertheless, it is a fascinating period and one from which we can learn.

19th Century Precursors

Automona had been written about since the Greeks and then actually produced, for centuries, by the Byzantines, Arabs and Europeans, creating highly imaginative, essentially mechanical, clockwork devices that performed fixed choreographed movements and tasks. 

But it wasn’t until the 19th century that mechanical devices were patented for teaching and learning. Mellan (1936) uncovered hundreds of these patents, although most were hand-cranked devices that offered little in the way of feedback. The first that was automated, was in 1866 by Halycom Skinner (no relation to BJ Skinner), for an automated spelling teaching machine. It fell short of giving feedback but was credible in terms of teaching. George Altman was another who had a scrolling device for teaching arithmetic patented in 1897, then Aikins in 1911 obtained a patent for a spelling machine that made you match letters to a shown object, which actually mentions the psychologist Thorndike as a justification for its efficacy. This late Victorian era was looking for industrial solutions to make mass schooling more efficient. None were actually manufactured and sold on scale.

Pressy

The true origin of teaching machines was the relatively unknown figure of Sidney Pressey, who came up with his idea for a teaching machine in 1915. He had to shelve the idea, as the First World War intervened, until he finally filed a patent in 1926. This was the first known machine to deliver content, accept input and deliver feedback. He is therefore the true originator of the first teaching machine.

Pressey was a cognitive psychologist long before it was seen as a school of psychology. He refused to accept learning theory based on the reductionist behaviourism of animal psychologists such as Pavlov, the behaviourist evangelist Watson or Skinner, who he knew personally, and had little time for learning theory that excluded consciousness, language and mental phenomena. The claim, therefore, that Teaching Machines were based on crude behaviourism, is simply false. His teaching machines reflected his cognitive-based learning theory.

His first machine used old typewriter parts to present multiple-choice questions with four options. The learner pressed a key for the right answer and the results were stored on a counter. It had the three necessary conditions for a teaching machine, the presentation of content, input by users and feedback. 

His second machine had the innovation of not moving on until you got the right answer and he continued to innovate with teaching machines into the late 1950s. Pressey understood that such machines could be used for both teaching and testing. You could set the machine, using a simple lever, to only move on if the learner got the right answer or alternatively assess by recording all of their answers, right and wrong. 

Using the second machine was easy, the learner simply pressed one of five keys (1-5), it had a small window that showed the numbers of questions asked and a window on the side showing the number of questions they got correct. In teaching mode the learners had to continue until the correct answer was chosen and the next question appeared. The questions number did not change until it was answered correctly and the window on the side showed the number of tries. He argued that this was quick, gave immediate results so that the learner didn’t have to wait days for results and saved the teacher time from the drudgery of marking, also eliminating marking errors. He also argued that this could free teachers to teach in a more inspirational manner. The learner could also repeat the experience until they got full mastery. You could quickly reset for the next student in seconds or the next test and could cope with up to 100 questions. These arguments are sound. An interesting attachment to the main machine delivered a candy if you passed a threshold number of correct answers (the threshold could be changed on the machine via a dial). All for under $15. Unfortunately, his timing was bad and the Great Depression put an end to his dream of manufacturing and popularising individualised learning.

Skinner’s teaching Machine

Skinner came nearly 40 years later, as a well known cultural figure, and grabbed all the attention with his own Teaching Machine, the GLIDER in 1954.

 

His yellow, wooden box contained a spindle for various rotating, paper discs. The questions were written along the radii of the discs and shown one by one in a window. The student had to write they answer on a roll of paper to the right of the questions in another aperture. When the student advanced the question, a model answer was seen, so the student could compare what they had written with the correct answer, without being able to change their answer. The learning was structured in a series of small steps. Hints and prompts maximise success and being right, so there is progress towards more complex knowledge. Skinner saw the machine as giving quick feedback, free from error, providing active learning and the fact that the student moves at their own pace was seen by him as a real benefit, whether faster or slower, at the rate most appropriate for that student. He claimed that this machine-based learning doubled the rate of learning, compared to the traditional classroom.

The content was carefully programmed to build, step by step towards synthesis and complex ideas. The machines then began to include more complex branching, with audio and screen presentations. Industrial and military applications focused on vocational learning. 

Learning theories

Pressey has very specific views on learning theory, more towards cognitive psychology than pure behaviourism. Errors or the correction of misconceptions were, for him, fundamental to learning, hence his fondness for multiple choice questions, which had 4/5 wrong answers. He saw learning as a complex process where relatively stable, cognitive structures had to be created. This had to be achieved through the analysis of errors, along with individualisation, diagnosis and feedback. Learning, for Pressey, was not a form of reinforcement, as with animals but involved uniquely human mediation through language, speaking, listening, reading and writing. It was a deeply cognitive process. He even formulated an early theory of Blended Learning, which he called, rather clumsily, ‘Adjunct Autoinstruction’. This involved the combination of programmed learning through technology and human teaching.

Plessy was the antithesis of Skinner, whose teaching machine was designed around positive reinforcement, hence his avoiding multiple choice questions, where the wrong answers (negative stimuli) outnumbered the right answer, that were actually given to the student, in advance of them having to think. Skinner saw this as weak learning and didn’t buy the idea that the study of wrong answers was anything but a distraction and, more seriously, seeding confusion in terms of what was learned.

Contemporary relevance

There are several lessons we can learn from this episode in the development of learning technology. First, that the cultural inertia in education is as strong today as it was then. Second, that learning technology, if it is to teach, must provide the presentation of material, cognitive interaction and feedback. Third, that marking is an area ripe for automation as it frees teachers to do more and better teaching. 

However the most important lessons lie around pedagogy. There is a serious debate around the nature of interaction and feedback, with one side still sticking to Pressey's multiple-choice questions and their variants versus Skinner's open input. AI is also being used to automatically create Skinner type content, along with AI identified links to the outside world. Open input (a feature of Skinner’s machine), and rarely applied even now in online learning, can now be interpreted using semantic analysis of open text answers. It requires more cognitive efforts with multiple choice questions, the answer is already given and you are selecting, rather than having to think deeply and recall. this was not coercion but a structured apron h to learning that puts the responsibility in the hand of the learner.

The advantages which automated, computer-based learning offer are much the same as they were 100 years ago, when Pressy built his first Teaching Machine. In fact, there is renewed interest in spaced, deliberate and retrieval practice, which have all shown significant learning gains, as well as learning in small steps (chunking). At its most advanced, adaptive learning systems resequence learning experiences to match the individual student’s progress. This optimises the path the student takes, based on individualised and aggregated data. Every learner learns uniquely. This keeps the student, on their learning journey, at the right level, neither pushing too far ahead nor making it too easy, both of which can destroy motivation and actual learning.

It turns out that both were correct. Pressey was right in seeing error correction with feedback a as powerful force in learning Metcalfe (2017). Skinner was right in seeing effortful learning, Brown (2014), as an important driver in learning. These were the germs of truth in their view that technology could significantly revolutionise learning, which has happened.

Influence

Pressey was convinced that education had to be reformed and called for an ‘industrial revolution’ in learning, based on the use of technology. He suffered a breakdown when his devices failed to sell and felt that the education system was closed to innovation. Skinner has similar frustrations. So by the 1960s mechanical teaching machines had had their day. As mechanical devices they were clunky and relied on discs, barrels, levers and buttons, all hardware and no software. They had little real effect on learning technology in the long-term, other than objects of obscure interest by commentators who perpetuate myths about Skinner’s Box being a Teaching Machine (it was not) and him hot housing his daughter (which he did not).

It is also a mistake to see these machines as being some sort of consequence of strict behaviourism. Pressey was more of a cognitive psychologist than behaviourist and Skinner's design had open input by the students, something few systems have to this day, hardly a primitive stimulus-response. Behaviourism was more than Skinner. In the long-term, Tolman's latent learning, along with Thorndike's work on transfer have stood the test of time. They were all much more sophisticated that their recent critics suggest.

In truth it was Babbage’s calculating machine in 1882 that was the real breakthrough, paving the way for computers and computer based learning. It was computers that were to provide the hardware and more importantly, the flexibility of software, logic and media presentation abilities that form the real evolutionary path for technology based learning. The 1970s saw the real rise of the personal computer and real teaching machines, with real software. The other great technological developments were in media; mass produced radios and television happened at the same time in the 50s. It was this confluence of hardware and software that created teaching machines that really were manufactured, sold and eventually became the desktops, laptops, tablets and smartphones, bought by billions of consumers on a global scale, using that global network - the internet.

Having worked for many years in an adaptive learning company, that specialised in personalised learning, I can scotch the idea that these machines ever influenced the vision or design of personalised learning. No one ever made reference to Pressey or Skinner. I doubt that anyone, apart from myself, even knew of the existence of these mechanical machines.

Bibliography

Slater, L., 2005. Opening Skinner's box: Great psychological experiments of the twentieth century. WW Norton & Company.

Benjamin, L. T. (1988). A history of teaching machines. American Psychologist, 43(9), 703–712. https://doi.org/10.1037/0003-066X.43.9.703

Mellan, I., 1936. Teaching and educational inventions. The Journal of Experimental Education, 4(3), pp.291-300.

Petrina, S., 2004. Sidney Pressey and the automation of education, 1924-1934. Technology and Culture, 45(2), pp.305-330.
Ferster, B., 2014. Teaching machines: Learning from the intersection of education and technology. JHU Press.

Metcalfe, J., 2017. Learning from errors. Annual review of psychology, 68, pp.465-489.

Brown, P.C., 2014. Make it stick. Harvard University Press.

https://www.youtube.com/watch?v=CFYruzWeFwQ General summary of Teaching Machines
https://www.youtube.com/watch?v=GXHmFZyKEVY Skinner on his Teach Machine