Monday, June 15, 2015

All of the Above – 20 ways to cheat MULTIPLE CHOICE questions

Many multiple choice questions are poorly written. What better way to expose these errors than write a cheat-sheet for learners? I never liked the 'start with the learning objectives' advice on designing learning content, and prefer to start with writing and polishing up these objectives as test items. This seemsmuch more real, practical and relevant than forced 'Mager-like' objectives. Get the assessment right and the rest seemed to follow. This, I suspect, is why students much prefer to get past test papers to establish the real contents of a course.
Of course, writing good test items is far more difficult than many imagine, which is why many tests are not really tests of understanding, merely tests of recall. An interesting way of coming at this problem is to do some reverse engineering. If you think this doesn’t work, think again. Poundstone number crunched 100 tests with a total of 2456 questions to get some of these statistical biases. I have 30 years experience in writing the damn things.
Second-guessing the test designer
So here goes with my 20 ways to cheat Multiple Choice tests:
1.  Skip the hard questions, mark them with a cross, and go back to them. This means you’ll not lose marks for unanswered easy questions.
2.  Cover the options and try to answer. Prevents being misled by clever wrong options.
3.  If in doubt choose ‘B’, poor questions designers do not truly randomise the right options and have a bias towards ‘B’. Next best is ‘C’.
4.  If in doubt choose the ‘longest option’. Question designers often cannot make a right option any shorter, but have complete freedom with wrong options. This is very common.
5.  If in doubt choose TRUE, in true/false questions, as they come easier to mind for designers.
6.  Reverse answers. Statistically, there is more T/F alteration in tests than in truly randomized sequences. So, if you’re sure you’ve got one right, reverse the next answer.
7.  Eliminate the outlier. Look for similarities in options and eliminate outliers (in bold) e.g. 4p-q, 2p+q, 4p+q, 3p+q. Look for these internal patterns.
8.  If two options are opposites, one is likely to be correct. Designers first made up option is likely to be the correct option’s opposite.
9.  Favour options with careful qualifiers, such as ‘sometimes, occasionally etc.’ as tested knowledge usually has more finite than absolute qualities.
10.  Be wary of options with absolute qualifiers, such as ‘always, never etc’. As these are often too definite to be reasonably correct.
11.  Choose a middle order option i.e. out of 100, 150. 200, 250, choose 150 or 200. Designers tend to have a bias, where right answers tend to be lower than the highest and higher than the lowest option.
12.  For questions that demand an ‘except’ or ‘not’, mark each option with a T for true and F for false against each option. And underline the word ‘not’ as it’s sometimes missed.
13.  ‘All of the above’ and ‘None of the above’ are both significantly likely to be correct. For it to be correct the writer has to design options that were all correct, so, if you can’t spot any wrong answers, or see that two or more are correct, it increases the probability of ‘All of the above’ being correct. Similarly with ‘None of the above’.
14.  Typo or punctuation error, the option is likely to be wrong. Writers tend to proofread correct answers only.
15.  Look for grammatical agreement between the question and its options; ‘An.....’ and words starting with vowels or agreement between subject, object or verb.
16.  If you’re stuck, go with the ‘Least bad rule’. Eliminate least likely answers first.
17.  Look for clues about answers from other questions. Designers often, unintentionally, put clues, even answers, to questions in other questions.
18.  Ignore never heard of answers. If you’ve never heard of the answer, it’s likely to be made up and incorrect.
19.  Go with your first impression. The more you read, the more you tend to read into the wrong options.
20.  Always guess, unless there is a penalty. It’s a 1 in 4 chance, so don’t give it up.

This crib sheet can be used by students or question designers to improve their tests. Good students put themselves in the shoes of the test designer to improve their chance, so the more you know about their techniques, the better designer you’ll be.
Next post - how to write a great MC question...
Andthere's 500 more tips like these, all categorised, here.

 Subscribe to RSS


Blogger Jussi Tuominen said...

Hi Donald!

Great story - starting from questions should be made mandatory in every learning cycle!-)

Starting from questions provides many advantages. Here are some:
- everybody in the team gets active already in planning phase
- you can see what are the real issues before investing your resources
- you can assess the course validity and effectiveness on advance
- sometimes you notice that answering questions is enough - funner and more effective way of learning!

We introduced our adLearning concept in the OEB2014. Since then we have tested our method to create questions at the start of courses in collaboration between teachers and students and we have very interesting results from Finland. OFTEN THE STUDENTS CAN CREATE BETTER QUESTIONS THAN TEACHERS! Apparently, it is true that when you learn something you start to forget what was difficult.

I'm looking forward to see your next post.

Take a look at our service and get a free account.

Jussi Tuominen

6:05 AM  

Post a Comment

<< Home