All of the Above – 20 ways to cheat MULTIPLE CHOICE questions
Forget objectives, start with questions
I never liked the 'start with the learning objectives' advice on designing learning content, and preferred to start with writing and polishing up these objectives as test items. This seemed that much more real, practical and relevant than forced 'Mager-like' objectives. Get the assessment right and the rest seemed to follow. This, I suspect, is why students much prefer to get past test papers to establish the real contents of a course.
Of course, writing good test items is far more difficult than many imagine, which is why many tests are not really tests of understanding, merely tests of recall. An interesting way of coming at this problem is to do some reverse engineering.
If you think this doesn’t work, think again. Poundstone number crunched 100 tests with a total of 2456 questions to get some of these statistical biases. I have 30 years experience in writing the damn things.
Second-guessing the test designer
Many multiple choice questions are poorly written. What better way to expose these errors than write a crib sheet for learners? So here goes with my 20 ways to cheat Multiple Choice tests:
1. Skip the hard questions, mark them with a cross, and go back to them. This means you’ll not lose marks for unanswered easy questions.
2. Cover the options and try to answer. Prevents being misled by clever wrong options.
3. If in doubt choose ‘B’, poor questions designers do not truly randomise the right options and have a bias towards ‘B’. Next best is ‘C’.
4. If in doubt choose the ‘longest option’. Question designers often cannot make a right option any shorter, but have complete freedom with wrong options.
5. If in doubt choose TRUE, in true/false questions, as they come easier to mind for designers.
6. Reverse answers. There is more T/F alteration in tests than in truly randomized sequences. So, if you’re sure you’ve got one right, reverse the next answer.
7. Eliminate the outlier. Look for similarities in options and eliminate outliers (in bold) e.g. 4p-q, 2p+q, 4p+q, 3p+q. Look for these internal patterns.
8. If two options are opposites, one is likely to be correct. Designers first made up option is likely to be the correct option’s opposite.
9. Favour options with careful qualifiers, such as ‘sometimes, occasionally etc.’ as tested knowledge usually has more finite than absolute qualities.
10. Be wary of options with absolute qualifiers, such as ‘always, never etc’. As these are often too definite to be reasonably correct.
11. Choose a middle order option i.e. out of 100, 150. 200, 250, choose 150 or 200. Designers tend to have a bias, where right answers tend to be lower than the highest and higher than the lowest option.
12. For questions that demand an ‘except’ or ‘not’, mark each option with a T for true and F for false against each option. And underline the word ‘not’ as it’s sometimes missed.
13. ‘All of the above’ and ‘None of the above’ are both significantly likely to be correct. is likely to be correct. For it to be correct the writer has to design options that were all correct, so, if you can’t spot any wrong answers, or see that two or more are correct, it increases the probability of ‘All of the above’ being correct. Similarly with ‘None of the above’.
14. Typo or punctuation error, the option is likely to be wrong. Writers tend to proofread correct answers only.
15. Look for grammatical agreement between the question and its options; ‘An.....’ and words starting with vowels or agreement between subject, object or verb.
16. If you’re stuck, go with the ‘Least bad rule’. Eliminate least likely answers first.
17. Look for clues about answers from other questions. Designers often, unintentionally, put clues, even answers, to questions in other questions.
18. Ignore never heard of answers. If you’ve never heard of the answer, it’s likely to be made up and incorrect.
19. Go with your first impression. The more you read, the more you tend to read into the wrong options.
20. Always guess, unless there is a penalty. It’s a 1 in 4 chance, so don’t give it up.
This crib sheet can be used by students or question designers to improve their tests. Good students put themselves in the shoes of the test designer to improve their chance, so the more you know about their techniques, the better designer you’ll be.
Next post - how to write a great MC question...