Sunday, December 14, 2014

How to outguess Multiple Choice Tests

In his book ‘How to predict the unpredictable’ William Poundstone exposes the weakness of human bias and competence in setting MC tests. Over six years ago I wrote this piece All of the Above - how to cheatMultiple Choice questions, so it was heartening (or disheartening) to see some real evidence that backed up those claims. Poundstone chewed up stats on 100 tests from schools, colleges and other professional sources. He found the following weaknesses in serious professional certification exams, SATs and in many educational assessments that matter in terms of selection.
True-False test strategy
1. Go through the whole test getting those you know right first.
2. If the known answers before and after the one you don’t know are the same, choose the opposite, as there’s more alternation of TRUE and FALSE than in a truly random sequence, so to choose a different answer from the last one increases your chance of getting it right from 50% to 63%.
3. If the known answers before and after are different, choose TRUE as there’s more right TRUE (56%) answers than FALSE (44%).
Multiple Choice test strategy
1. Pick ‘none of the above’ or ‘all of the above’ as this gives you up to a 90% increase on random guessing.
2. Pick B on a four choice test, as this gives you an 11% increase, as human testers can’t randomise.
3. Avoid the previous choice gives you an 8% increase, again as human testers can’t randomise.
4. Choose the longest item. Even on the supposedly unbeatable SAT exams.
5. Eliminate outliers.
6. Always guess, as instinct can sometimes kick in to produce the right answer.

We live in an age of constant assessment, that is a shame. However, to live in an age of inefficient assessment is even worse, as smart, coached and tutored students can ‘game’ the system. For assessment designers it may be worth looking at these and other ‘game’ strategies to eliminate the weaknesses of your tests. First randomise the right answers. Humans just cannot randomise. Then look for the common biases. One final point. Randomisation is an algorithm. This is a case where algorithmic power of computers can eliminate human cognitive bias and weak competence. 


Rob Alton said...

Yep, usually B or C or the longest distractor in my experience . People who write tests (especially in elearning) often don't really fully understand the subject they are 'teaching' (or instructionally designing if you believe the BS job title hype).

It's a great shame that elearners are often being taught by people who have done the equivalent of cramming the night before an exam.

Still, it's all about the money and the graphics, isn't it?

Nic Price said...

Like you say, the problem is often in the design of the question and this requires more awareness and skill on the part of setters. Randomisation is useful but not universally good. Where there is a logical order to options (e.g. numbers) it is fairer to provide them in the logical order.

There are measures that can make MCQs less gameable. An underused strategy is negative marking to discourage guessing. I particularly like the idea of confidence-based marking (confident answers are required for top marks but there is punitive downgrading for confident+wrong selections). I think there is resistance because it openly acknowledges that there is gaming and answers are often guesswork.

Donald Clark said...

Rob - certain truth in this but some of the worst tests in his sample were from Universities by professional lecturers and subject matter experts. They were the easiest to 'game'. It's a skill, whatever your background.

Rob Alton said...

Indeed. The MCQ Is often the first port of call for most people when writing tests for the first time. I've got over 20 yrs experience in learning design and I must say that it can sometimes still take a lot of time to write an effective MC item. Throw confidence testing into the mix and there goes a whole heap of time.

The Guardian daily quizzes have some really skilled item writers on board.

Martin Owen said...

How many educators have had any professional development in assessment design - certainly not most University teachers. However, it is worth looking at the practice in problem based medical education courses -a lot of MCQ in medicine that is really well designed and it certainly changed my opinion on MCQs. Try putting "design medicine mcqs for medical professionals" into Scholar.

Donald Clark said...

Agree. The bottom line is that this is a skill and needs some training. It simply reinforces my opinion that teaching is not an art but a set of practices and competences.

Rob Alton said...

It is getting better in Higher Ed. Many Academics are now more willing to learn how to teach/assess and their institutions are offering resources that help them to do this.


Norman Lamont said...

I had some very good training in the 80s on testing from a guy who worked in assessment for international aviation. He charged 1.5 days per question. It's all in the planning and user testing.

Brian Mulligan said...

"teaching is not an art but a set of practices and competences." Donald, this reflects my own opinion that you don't need to know about cognitive psychology or have a grad diploma in Teaching and Learning to be a good teacher - just a good bag of tricks. (This is true for the design of good objectives tests also). Have you written elsewhere on this?

Donald Clark said...

I have but not wholly convinced by that definition Bryan. Doesn't it beg the question 'What practices?'. I've seen widespread bad practices in education - whole word teaching, for example which lasted for 2 decades, learning styles etc. Some knowledge of psychology helps, especially knowledge of how memory works. This can stop the endless lecturing, cognitive overload etc.