Friday, September 04, 2015

10 essential points on use of (recall not recognition) open-response questions

Multiple choice questions are, essentially, a test of recognition. They do not elicit full recall from memory. That’s not to say they are not useful but it does say that they are of limited use.. In practice, it is active recall that really matters in knowledge and skills, not recognition. So why not move up the assessment ladder and consider open response?
1. Tests recall not recognition
Open-response test items ask for recall of the actual text. This is very different from the recognition that multiple choice questions demand. It’s a step up in terms of competence.  We can distinguish between four levels of learning:
Familiar – knew but can’t now remember
Recognised – correctly recognise answer in a multiple choice question
Recalled – recall with effort but without help and takes time
Automatic – immediate, effortless, high-performance recall
It is quick and ‘Automatic’ recall that is the goal of high-performance learning and expert ability.
2. Reinforces learning
Open-response takes cognitive effort and the very act of recalling knowledge reinforces that knowledge in memory. Active recall, pulling something out of memory, not just recognising something from a list, improves future performance, something we have known for a century (Gates 1917). The act of active recall develops and strengthens memory. It also improves the process of recall in ways that passive recall – reading, listening and watching do not. So retrieval in itself, prevents memory loss (Bjork 1975) and the more we recall, the more recallable memories become.
3. Accept synonyms
You may accept the word ‘synonym’ as the correct answer but should you not also accept ‘substitute’ or ‘replacement”? Always consider correct synonyms as correct answers, unless you’re asking for that one, specific word.
4. Accept common misspellings
Unless you are also testing for spelling, accept common misspellings, especially double letters, silent letters, capitalisation and common mistakes. You may also want to accept both British and American spellings. Consider also typos, especially transposition errors i.e. when someone either accidentally types in two letter the wrong way round (a common typing error). Ntoe htat trnaspoesd lettres aer usaully tpying errosr adn hte maennig remians intcat.
5. Feedback on common misconceptions
For example, if you are doing a medical test and you’ve typed in virus, as opposed to bacterium/bacteria, it may be useful to point out that this is a common error in diagnosis, and that it seriously affects the recommended treatment.
6. Provide letter count
Some like to leave the filed blank or have a continuous line where the answer is to be typed. Others like to provide small dashes, so that the learner knows what’s expected in terms of letter length. It depends on how tough you want to be. In formative assessment, I prefer dashes.
A _____ is a small infectious agent that replicates only inside the living cells of other organisms.
A ----- is a small infectious agent that replicates only inside the living cells of other organisms.
7. First letter hint
First letter hints avoid the wild guessing but you must be aware that you’re not testing for full automatic recall.
A v---- is a small infectious agent that replicates only inside the living cells of other organisms.
8. Meaningful hints
You may want to provide meaningful semantic hints, as open input can be a challenge. Moving the learner meaningfully towards typing in the right answer can reinforce learning. If the answer is virus and the learner has typed in ‘germs’ or ‘disease’ or ‘infection’, you may want to clarify their mistake. Open-response is often useful for this tight form of assessment around conceptual knowledge. For example. Knowing the difference between a molecule, mixture and compound in chemistry.
9. Timed responses
To up the stakes, and distinguish between recall and automatic recall, add a timer. Automatic recall can be tested by putting a time limit on the answer.
10. Provide a get out
Whatever check and feedback technique(s) you use, make sure the learner has a chance, eventually, to get out of an endless loop of guessing. Given the mathematical possibility that there are millions of options for even relatively short words, don’t trap them into guessing and trying forever. You do have to either provide letter-by-letter reveals or eventually provide enough information for them to get the right answer or the answer itself Note that it may still be important for the learner to type that correct answer in, as this is an important act of reinforcement.
I have spent some time building an open response tool that creates e-learning on the fly. You simply put in your text (sentence, paragraph, document, paper, company policy, textbook - whatever) and it automatically creates e-learning with formative, open-input assessment. It also links out to Wikipedia for additional information and does the same, creating e-learning on the fly. If you want to know more about this automatic creation of e-learning tool, contact me here.
Bjork R.A. (1975) Retrieval as a memory solidifier: an interpretation of negative recency and related phenomena. In RL Solso (Ed.) Information processing and cognition (123-124) Hillsdle, NJ: Erbaum

Gates A.I. (1917) Recitation as a factor in memorizing. Arch. Phschol. 6, 40.

No comments: