Has the old 'graphics-text-graphics-text-MCs had its day? Evidence that effortful, open-response learning much better

To what degree does contemporary online learning reflect contemporary learning theory? The old paradigm of graphic-text-MCQ is way out of line with recent (and past) learning theory, so that, no matter how much glitz, animation and graphics you produce, the fundamentals of retention and recall are ignored. On a quick walk round the Learning technologies show this year I saw much the same as I've seen for the last 30 years, some worse. The same old, over-engineered content that takes months to make at high cost but with low retention value.

So, over the last year or so I’ve been working on online learning (WildFire) that largely abandons the multiple choice question, with lots of graphics production and glitz, for a more stripped-down approach that focuses on effortful learning. I wanted to produce content quickly, in minutes not months, cheaply (at least 80% cheaper) and to a higher quality than existing online learning (based on retention and recall).

Science of learning

There are good and bad ways to learn. Unfortunately much of what feels intuitive, is in fact, wrong. The science of learning has shown that researched, counterintuitive strategies, often ignored in practice, produce optimal learning.

For example, much advice and most practice from educational institutions – re-reading, highlighting and underlining – is wasteful. In fact, these traditional techniques can be dangerous, as they give the illusion of mastery. Indeed, learners who use reading and re-reading show overconfidence in their mastery, compared to learners who take advantage of effortful learning.

Yet significant progress has been made in cognitive science research to identify more potent strategies for learning. The first strategy, mentioned as far back as Aristotle, Francis Bacon then William James, is ‘effortful’ learning. It is what the learner does that matters.

Simply reading, listening or watching, even repeating these experiences, is not enough. The learning is in the doing. The learner must be pushed to make the effort to retrieve their learning to make it stick in long-term memory. This one act is the best defence against the brain’s other propensity, identified by Ebbinghaus in 1885 – forgetting.

With this in mind, I wanted to design learning experiences to deliver good learning using some fundamental principles in researched learning theory:

1. Active retrieval

2. Multiple-choice v open-response

3. Typing in words

1. Active retrieval

To be specific about effortful learning, by effort I mean ‘active retrieval’ as the most powerful learning strategy at your disposal. The brain, the organ that named itself, is a unique organ in that it can test itself to see what it knows or doesn’t know. At the same time this act of retrieval consolidates has been found to be even more powerful than the original learning experience.

Study 1 - Gates

The first solid research on retrieval was by Gates (1917), who tested children aged 8-16 on short biographies. Some simply re-read the material several times, others were told to look up and silently recite what they had read. The latter, who actively retrieved knowledge, showed better recall.

Study 2 - Spitzer

Spitzer (1939) made over 3000 11-12 year olds read 600 word articles then tested students at periods over 2 months. The greater the gap between testing (retrieval) and the original exposure or test, the greater the forgetting. The tests themselves seemed to halt forgetting.

Study 3 - Tulving

Tulving (1967) took this further with lists of 36 words, with repeated testing and retrieval. The retrieval led to as much learning as the original act of studying. This shifted the focus away from testing as just assessment to testing as retrieval, as an act of learning in itself.

Study 4 – Roediger

Roediger et al. (2011) did a study on text material covering Egypt, Mesopotamia, India and China, in the real context of real classes in a real school, a Middle School in Columbia, Illinois. Retrieval tests, only a few minutes long, produced a full grade-level increase on the material that had been subject to retrieval.

Study 5 – McDaniel

McDaniel (2011) did a further study on science subjects, with 16 year olds, on genetics, evolution and anatomy. Students who used retrieval quizzes scored 92% (A-) compared to 79% for those who did not. More than this, the effect of retrieval lasted longer, when the students were tested eight months later.

Design implications

So I’ve been designing learning as a retrieval learning experience, largely using open-input, where you have to pull things from your memory and make a real effort to type in the missing words, given their context in a sentence. First, as the research shows, this tells you what you know, half-know or don’t know. Second, it consolidates what you know in long-term memory. Third, it encourages you to improve your performance.

2. Open-response v multiple-choice

Most online learning relies heavily on multiple-choice questions, which have become the staple of much e-learning content. These have been shown to be effective, as almost any type of test item is effective to a degree, but they have been shown to be less effective than open-response, as they test recognition from a list, not whether it is actually known.

Study 1 - Duchastel and Nungester

Duchastel and Nungester (1982) found that multiple-choice tests improve your performance on recognition in subsequent multiple-choice tests and open input improves performance on recall from memory. This is called the ‘test practice effect’.

Study 2 – Kang

Kang et al. (2007) showed that, with 48 undergraduates, reading academic Journal quality material, open input is superior to multiple-choice (recognition) tasks. Multiple choice testing had an affect similar to that of re-reading whereas open-input resulted in more effective student learning.

Study 3 – McDaniel

McDaniel et al. (2007) repeated this experiment in a real course with 35 students enrolled in a web-based Brain and Behavior course at the University of New Mexico. The open-input quizzes produced more robust benefits than multiple-choice quizzes.

Study 4 - Bjork

‘Desirable difficulties’ is a concept coined by Elizabeth and Robert Bjork, to describe the desirability of creating learning experiences that trigger effort, deeper processing, encoding and retrieval, to enhance learning. The Bjorks have researched this phenomenon in detail to show that effortful retrieval and recall is desirable in learning, as it is the effort taken in retrieval that reinforces and consolidates that learning.

Design implications

A multiple-choice question is a test of recognition from a list. They do not elicit full recall from memory. Studies comparing multiple-choice with open retrieval show that when more effort is demanded of students, they have better retention.. As open-response takes cognitive effort, the very act of recalling knowledge also reinforces that knowledge in memory. The act of active recall develops and strengthens memory. It improves the process of recall in ways that passive recall – reading, listening and watching do not. Active recall, pulling something out of memory, is therefore more effective in terms of future performance.

Note that multiple-choice questions are useful for situations where they are deemed necessary, for example, in common misconceptions or lists, which is why they are also used but not the dominant form of learning.

3. Typing in words

A fascinating finding by Jacoby was the precise identification of filling in the missing letters of a word as a powerful act of consolidation of memory. This is called the ‘generation effect’. In other words, only a relatively small amount of retrieval effort is needed to have a powerful effect on memory.

Study 1 – Jacoby

Jacoby (1978) uncovered the fact that cramming led to short-term gains but long-term forgetting. Learners achieved high scores on the first, immediate test but forgot 50% in subsequent tests, compared to those who retrieved material, and forgot only 13%. He also showed that, in presenting word pairs, where some learners got the entire word pair, others got one word with two or more letters deleted from the interior of that word, a bit like an unfinished crossword, the simple act of filling-in-the-blanks resulted in higher retention and recall.

Study 2 – McDaniel

McDaniel et al (1986) also pinpointed leaving out letters as a way to stimulate retrieval. Learners were asked to fill-in-the-blank missing letters in Fairy Tales and showed significant gains in recall. The cognitive effort required to complete the terms strengthened retention and recall.

Study 3 – Hirshman & Bjork

Hirshman & Bjork (1988) got learners to type in the missing letters in words (salt-p_pp_r), which resulted in higher retention rates for conceptual pairs than the words being read on their own.

Study 4 – Richland

Richland et al. (2005) took this research into a real world environment and showed similarly positive effects. They concluded that it is the effortful engagement in the process of retrieval that leads to better recall.

Design implications

Meaning matters and so I rely primarily on reading and open response, where meaningful recall is stimulated. This act alone, even when you don’t know the answer, is a strong reinforcer, stronger indeed, than the original exposure. Interestingly, even when the answer is not known, the act of trying to answer is also a powerful form of learning.

So, the deliberate choice of open-response questions, where the user types in the words until they are correct, is a deliberate, design strategy to take advantage of known techniques to increase recall and retention. Note that no learner is subjected to the undesirable difficulty of getting stuck, as letters are revealed one by one, and the answer given after three attempts. Hints are also possible in the system.

For more information on WildFire click here.

Bibliography

Bjork, R. A. (1994). Memory and metamemory considerations in the training of human beings. In J. Metcalfe & A. Shimamura (Eds.), Metacognition: Knowing about knowing (pp. 185–205). Cambridge, MA: MIT Press.

Bower G. H. (1972) Mental imagery in associative learning in Gregg L,W. Cognition in learning and memory New York, Wiley

Gardener (1988) Generation and priming effects in word fragment completion Journal of Experimental Psychology: Learning, Memory and Cognition 14, 495-501

Butler, A. C., & Roediger, H. L. (2008). Feedback enhances the positive effects and reduces the negative effects of multiple-choice testing. Memory & Cognition, 36, 604-616.

Duchastel, P. C., & Nungester, R. J. (1982). Testing effects measured with alternate test forms. Journal of Educational Research, 75, 309-313.

Gardener (1988) Generation and priming effects in word fragment completion Journal of Experimental Psychology: Learning, Memory and Cognition 14, 495-501

Gates, A. I. (1917). Recitation as a factor in memorizing. Archives of Psychology, No. 40, 1-104.

Hirshman, E. L., & Bjork, R. A. (1988). The generation effect: Support for a two-factor theory. Journal of Experimental Psychology: Learning, Memory, & Cognition, 14, 484–494.

Jacoby, L. L. (1978). On interpreting the effects of repetition: Solving a problem versus remembering a solution. Journal of Verbal Learning and Verbal Behavior, 17, 649-667.

Kang, S. H. K., McDermott, K. B., & Roediger, H. L., III. (2007). Test format and corrective feedback modulate the effect of testing on long-term retention. European Journal of Cognitive Psychology, 19, 528-558.

McDaniel, M. A., Einstein, G. O., Dunay, P. K., & Cobb, R.  (1986).  Encoding difficulty and memory:  Toward a unifying theory.  Journal of Memory and Language, 25, 645-656.

McDaniel, M. A., Agarwal, P. K., Huelser, B. J., McDermott, K. B., & Roediger, H. L. (2011). Test-enhanced learning in a middle school science classroom: The effects of quiz frequency and placement. Journal of Educational Psychology, 103, 399-414

Miller, G.A. (1956). The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological Review, 63, 81-97.

Richland, L. E., Bjork, R. A., Finley, J. R., & Linn, M. C. (2005). Linking cognitive science to education: Generation and interleaving effects. In B. G. Bara, L. Barsalou, & M. Bucciarelli (Eds.), Proceedings of the twenty-seventh annual conference of the cognitive science society. Mahwah, NJ: Erlbaum.

Roediger, H. L., Agarwal, P. K., McDaniel, M. A., & McDermott, K. B. (2011). Test-enhanced learning in the classroom: Long-term improvements from quizzing. Journal of Experimental Psychology: Applied, 17, 382-395.

Spitzer, H. F. (1939). Studies in retention. Journal of Educational Psychology, 30, 641-656.

Tulving, E. (1967). The effects of presentation and recall of material in free-recall learning. Journal of Verbal Learning and Verbal Behavior, 6, 175􏰀184.