Wednesday, November 12, 2008

All of the Above - how to cheat Multiple Choice questions

Forget objectives, start with questions

I never liked the 'start with the learning objectives' advice on designing content, and preferred to start with writing and polishing these objectives as test items. This seemed that much more real, practical and relevant than forced 'Mager-like' objectives. Get the assessment right and the rest seemed to follow. This, I suspect, is why students much prefer to get past test papers to establish the real contents of a course.

Of course, writing good test items is far more difficult than many imagine, which is why many tests and not really tests of understanding, merely tests of recall. An interesting way of coming at this problem is to do some reverse engineering. Ask how students can cheat their way through a test.

There's the usual crib notes. The best I heard was, remove the lable from a bottle of orange juice, writing the crib notes on the back of the label.paste it back on, then drink orange to reveal the notes. But let's assume that pure cheating is out. What are you left with?

Second-guessing the test designer

Many multiple choice questions are poorly written. What better way to expose these errors than write a crib sheet for learners? So here goes with my 20 ways to cheat Multiple Choice tests:

1.    Skip the hard questions, mark them with a cross, and go back to them. This means you’ll not lose marks for unanswered easy questions.

2.    If in doubt choose ‘C’, poor questions designers do not truly randomise the right options and have a bias towards ‘C’. Next best is ‘B’.

3.    If in doubt choose the ‘longest option’. Question designers often cannot make a right option any shorter, but have complete freedom with wrong options.

4.    Look for similarities in options and eliminate outliers (in bold) e.g. 4p-q, 2p+q, 4p+q, 3p+q.

5.    Now note that there’s only one ‘-‘, which makes 4p+q more likely. Look for these internal patterns.

6.    All of the above’ is likely to be correct. For it to be correct the writer has to design options that were all correct, so, if you can’t spot any wrong answers, or see that two or more are correct, it increases the probability of ‘All of the above’ being correct. Similarly with ‘None of the above’.

7.    Choose a middle order option i.e. out of 100, 150. 200, 250, choose 150 or 200. Designers tend to have a bias, where right answers tend to be lower than the highest and higher than the lowest option.

8.    For questions that demand an ‘except’ or ‘not’, mark each option with a T for true and F for false against each option. And underline the word ‘not’ as it’s sometimes missed.

9.    If there’s a typo or punctuation error, the option is likely to be wrong. Writers tend to proofread correct answers only.

10.  Look for grammatical agreement between the question and its options; ‘An.....’ and words starting with vowels or agreement between subject, object or verb.

11.  Go with your first impression. The more you read, the more you tend to read into the wrong options.

12.  If you’re stuck, go with the ‘Least bad rule’. Eliminate least likely answers first.

13.  Look for clues about answers from other questions. Designers often, unintentionally, put clues, even answers, to questions in other questions.

14.  If you’ve never heard of the answer, it’s likely to be made up and incorrect.

15.  First cover the options and try to answer. Prevents being misled by clever wrong options.

16.  If two options are opposites, one is likely to be correct. Designers first made up option is likely to be the correct option’s opposite.

17.  Favour options with careful qualifiers, such as ‘sometimes, occasionally etc.’ as tested knowledge usually has more finite than absolute qualities.

18.  Conversely, be wary of options with absolute qualifiers, such as ‘always, never etc’. As these are often too definite to be reasonably correct.

19.  Always guess, unless there is a penalty. It’s a 1 in 4 chance, so don’t give it up.

20.  Eliminate obvious answer on 4 options then guess, don’t fail to answer. This reduces the odds from ‘1 in 4’ to ‘1 in 3’. Far better than just guessing or not answering, depending on any penalty scores for wrong answers.

None of the above

This crib sheet can be used by question designers to improve their tests. Good students put themselves in the shoes of the test designer to improve their chance, so the more you know about their techniques, the better designer you’ll be.

I still see binary option questions with ‘Try again’ logic, grammatical disagreement and stupid options. Some time back in this blog, after they refused to respond when I emailed the mistakes through, I had a go at BBC Bitesize’s science tests, as they were riddled with these errors. 140 comments later, it still pops up on the home page when you go to BBC Bitesize through Google.

Writing good multiple choice questions is not easy. What’s easy is simply extracting all the nouns, objects and quantities, then testing for recall. The trick is to push beyond this to test understanding. It’s not the ‘what’ but the ‘why’ that often matters but remains untested.

Why questions matter

Professor Dylan Wiliam, Deputy Director of the Institute of Education, Professor of Educational Assessment gave a brilliant ALT talk (Seb Schmoller put me on to this) on what calls ‘hinge questions’, questions that literally diagnose poor understanding. He explains how one can use these questions as powerful verbal test items in a classroom, where it is difficult to diagnose 30 kids quickly. This is a technique every teacher should learn.

The ball sitting on a table is not moving. It’s not moving because:

A. No forces are pushing or pulling on the ball.

B. Gravity is pulling down, but the table is in the way.

C. The table pushes up with the same force that gravity pulls down.

D. Gravity is holding it on to the table.

E. There’s a force inside the ball keeping it from rolling off the table.

This question not only catches common misconceptions, it diagnoses between those who have understood the ‘physics’. C is correct.

What can we do to preserve the ozone layer?

A. Reduce the amount of carbon dioxide in the air

B. Reduce the greenhouse effect

C. Stop cutting down the rainforest

D. Properly dispose of air conditioners and fridges

Looks like the designer ran out of options and put the last item in to make up the numbers, but it’s actually the right answer. Interestingly, this was one of dozens of mistakes in BBC Bitesize,

Which of the following is NOT an effect of burning fossil fuels?

A. global warning

B. ozone depletion

C. acid rain

D. smog

E. fog (right according to BBC)

Ozone depletion is also correct as the result of CFCs which are completely artificial (they did not exist in nature prior to synthesis by humans). They were used in air conditioning/cooling units, as aerosol spray propellants prior to the 1980s, and in the cleaning processes of delicate electronic equipment. They are not the result of burning fossil fuels.

Wiliam’s point about the imortance of classroom questioning, is that; 

“The variability at teacher level is about four times the variability at school level. If you get one of the best teachers, you will learn in six months what an average teacher will take a year to teach you. If you get one of the worst teachers, that same learning will take you two years. There’s a four-fold difference in the speed of learning created by the most and the least effective teachers. And it’s not class size, it’s not between class grouping, it’s not within class grouping – it’s the quality of the teacher.”

This led him to determine what separates good from bad teachers.

“And actually, new teachers are actually pretty bad. You don’t really learn to teach at all well until you’re six or seven years into the profession. And some recent data from Australia shows that the amount of value added by teachers actually carries on increasing for about twenty years.”

And here’s a brilliant paragraph.

“The key concept here—the big trap—is that teachers do not create learning. That’s true teachers do not create learning, and yet most teachers behave as if they do. Learners create learning. Teachers create the conditions under which learning can take place. Our schools don’t function like that, which is why somebody once joked that schools are places where kids go to watch teachers work.”

The solution, given the fact that reducing clss sizes is incredibly expensive, is to use diagnistic 'hinge' questions. This accelerates the teacher's knowledge of the state of learning of the learners and accelerates the learning.

Dylan William also co-authored, with professor Paul Blackgave, the brilliant Inside the Black Box: Raising Standards through Classroom Assessment, another ‘should be compulsory’ text for educators. This makes the obvious point that far too much teaching is simply 'chalk and talk'. If you don't believe this walk into any school and you'll see it in practice. It's a manifesto for more formative assessment.

Why questions matter

What matters? All of the above and more. Questions really do matter in learning. 

Questions and curiosity

First, they stimulate curiosity. Almost all of my learning as an adult has this dynamic. Something intigues me and I follow it up as I'm curious to find the answer. This is the great joy of having the internet as a resource. It has made this type of inquiry and research possible.

Questions and diagnosis

Good qestions diagnose your strengths and weaknesses. You don't know what you don't know and questions uncover the often uncomfortable truth that you know less than you thought you know.

Questions and improvement

Questions and searching for answers are fundamental to the process of learning. Roger Schank has been using this apporach in all sorts of contexts, and this truly structured Socratic approach, works well when used by a skilled practitioner. 

Questions and motivation

To create the conditions for learning, as opposed to just delivering content, questions are the true stimulus. 

Yet, despite these advantages, few have the real skills to either construct or deliver formative or summative feedback at the level necessary for true learning. It's a real skill.


 Subscribe to RSS


Blogger Rob Alton said...

Indeed. MCs are seen as being the easiest to write, especially for people new to the game. They're not as the examples you show point out.

Have you heard about the new system from Rocket? You type in your stem and it does the rest for you.

Seriously, you should do feedback next - another area of mediocrity. Correct or Incorrect?

4:30 PM  
Blogger John Wootton said...

Is Dylan the great man when it comes to AfL?

a Yes
b No
c Depends how long you've been teaching
d Depends whether the Senior Leadership Team themselves, have remembered how to teach

John Wootton
Deputy Headteacher

8:55 PM  
Anonymous Seb Schmoller said...

Donald, for ease of your readers' reference, Dylan Wiliam's "Assessment, learning and technology: prospects at the periphery of control", can be accessed in various formats as follows Text transcript [75 kB PDF]; Slides [400 kB PDF]; MP3 recording [12 MB].

9:45 PM  
Blogger claire said...

Hi seb schmoller,

Thanks for your sharing! Why not integrate Dylan Wiliam's PPT and Recording together for online presentation with PPT2Flash.

Here is a sample. I like this way better. Hope it helps!


3:06 AM  
Blogger claire said...

Hi seb schmoller,

Thanks for your sharing! Why not integrate Dylan Wiliam's PPT and Recording together for online presentation with PPT2Flash.

Here is a sample. I like this way better. Hope it helps!


3:08 AM  
Blogger Blogger In Middle-earth said...

Kia ora Donald

Thanks for this.

One thing I think is missing in this post (I might be wrong) is the usefulness and power of the mistake in learning.

Getting back a test paper, for me (as a secondary student way back), was a learning experience. I was aware that my peers would look gloomily at thier results. I didn't.

I used to squirrel the test paper away and refer to it often, examining all my errors and things I got marked down for.

The mistake is a much maligned, yet under-rated tool in learning.

Ka kite
from Middle-earth

9:19 AM  
Blogger Donald Clark said...

Couldn't agree more Ken. Roger Schank built uop a whole view of learning, based on the idea that we learn primarily from our mistakes. The correction of errors and mistakes is the powerhouse in learning, yet the 'red pen' approach and lack of opportunity for safe failure is the norm in learning.

Learners get nowhere near adequate opportunties to learn by doing and make mistakes in practice, and teachers often do not like the toil of marking and correcting failure. This double hit stops lots of kids in their tracks. In maths, it can be disastrous.

This is why I am such a fan of good e-learning. If we can allow safe failure and give adequate diagnostic feedback, along with remedial help - we'll get dramatic improvements in knowledge and performance.

9:44 AM  
Anonymous Anonymous said...


In your view is a candidate cheating/misconduct if they answer 'True' or 'False' to ALL answers in a non-negatively marked MCQ paper and did not break examination rules or regulations?

Is there any evidence, case law, regarding this?


10:58 AM  
Blogger Nic Price said...

Hi Donald,
I'm currently building a platform for certainty-marked questions/quizzing for use by schools. As you have noted, negative marking can change behaviour on MCQs. Certainty-marked assessments are a sophisticated form of negative marking where the participant effective adjusts the stakes based on their certainty. This brings many benefits including:
- questions are much easier to devise (and can be much simpler to read/interpret)
- there is better insight (secure knowledge / uncertainty / misconception)
- it creates more accurate and reliable data
There is really good research and evidence in their use by Professor Tony Gardner-Medwin, who used it extensively (including for formal assessment) whilst at UCL. It's been used by various e-learning companies (but not as much as it could/should have been).
You can read more about my project at - there's a prototype and I'm almost ready to start piloting with a group of volunteers.

4:07 PM  
Blogger Donald Clark said...

Quite a lots of assessment systems use this approach including spaced-practice systems. Good approach.

4:10 PM  
Blogger Nic Price said...

I've struggled to find many examples, hence why I decided to build a platform for it. I can count on one hand the number of times a corporate client has been adventurous enough to use it (properly) over conventional scoring.

4:21 PM  

Post a Comment

<< Home