All of the Above - how to cheat Multiple Choice questions

Forget objectives, start with questions

I never liked the 'start with the learning objectives' advice on designing content, and preferred to start with writing and polishing these objectives as test items. This seemed that much more real, practical and relevant than forced 'Mager-like' objectives. Get the assessment right and the rest seemed to follow. This, I suspect, is why students much prefer to get past test papers to establish the real contents of a course.

Of course, writing good test items is far more difficult than many imagine, which is why many tests and not really tests of understanding, merely tests of recall. An interesting way of coming at this problem is to do some reverse engineering. Ask how students can cheat their way through a test.

There's the usual crib notes. The best I heard was, remove the lable from a bottle of orange juice, writing the crib notes on the back of the label.paste it back on, then drink orange to reveal the notes. But let's assume that pure cheating is out. What are you left with?

Second-guessing the test designer

Many multiple choice questions are poorly written. What better way to expose these errors than write a crib sheet for learners? So here goes with my 20 ways to cheat Multiple Choice tests:

1.    Skip the hard questions, mark them with a cross, and go back to them. This means you’ll not lose marks for unanswered easy questions.

2.    If in doubt choose ‘C’, poor questions designers do not truly randomise the right options and have a bias towards ‘C’. Next best is ‘B’.

3.    If in doubt choose the ‘longest option’. Question designers often cannot make a right option any shorter, but have complete freedom with wrong options.

4.    Look for similarities in options and eliminate outliers (in bold) e.g. 4p-q, 2p+q, 4p+q, 3p+q.

5.    Now note that there’s only one ‘-‘, which makes 4p+q more likely. Look for these internal patterns.

6.    ‘All of the above’ is likely to be correct. For it to be correct the writer has to design options that were all correct, so, if you can’t spot any wrong answers, or see that two or more are correct, it increases the probability of ‘All of the above’ being correct. Similarly with ‘None of the above’.

7.    Choose a middle order option i.e. out of 100, 150. 200, 250, choose 150 or 200. Designers tend to have a bias, where right answers tend to be lower than the highest and higher than the lowest option.

8.    For questions that demand an ‘except’ or ‘not’, mark each option with a T for true and F for false against each option. And underline the word ‘not’ as it’s sometimes missed.

9.    If there’s a typo or punctuation error, the option is likely to be wrong. Writers tend to proofread correct answers only.

10.  Look for grammatical agreement between the question and its options; ‘An.....’ and words starting with vowels or agreement between subject, object or verb.

11.  Go with your first impression. The more you read, the more you tend to read into the wrong options.

12.  If you’re stuck, go with the ‘Least bad rule’. Eliminate least likely answers first.

13.  Look for clues about answers from other questions. Designers often, unintentionally, put clues, even answers, to questions in other questions.

14.  If you’ve never heard of the answer, it’s likely to be made up and incorrect.

15.  First cover the options and try to answer. Prevents being misled by clever wrong options.

16.  If two options are opposites, one is likely to be correct. Designers first made up option is likely to be the correct option’s opposite.

17.  Favour options with careful qualifiers, such as ‘sometimes, occasionally etc.’ as tested knowledge usually has more finite than absolute qualities.

18.  Conversely, be wary of options with absolute qualifiers, such as ‘always, never etc’. As these are often too definite to be reasonably correct.

19.  Always guess, unless there is a penalty. It’s a 1 in 4 chance, so don’t give it up.

20.  Eliminate obvious answer on 4 options then guess, don’t fail to answer. This reduces the odds from ‘1 in 4’ to ‘1 in 3’. Far better than just guessing or not answering, depending on any penalty scores for wrong answers.

None of the above

This crib sheet can be used by question designers to improve their tests. Good students put themselves in the shoes of the test designer to improve their chance, so the more you know about their techniques, the better designer you’ll be.

I still see binary option questions with ‘Try again’ logic, grammatical disagreement and stupid options. Some time back in this blog, after they refused to respond when I emailed the mistakes through, I had a go at BBC Bitesize’s science tests, as they were riddled with these errors. 140 comments later, it still pops up on the home page when you go to BBC Bitesize through Google.

Writing good multiple choice questions is not easy. What’s easy is simply extracting all the nouns, objects and quantities, then testing for recall. The trick is to push beyond this to test understanding. It’s not the ‘what’ but the ‘why’ that often matters but remains untested.

Why questions matter

Professor Dylan Wiliam, Deputy Director of the Institute of Education, Professor of Educational Assessment gave a brilliant ALT talk (Seb Schmoller put me on to this) on what calls ‘hinge questions’, questions that literally diagnose poor understanding. He explains how one can use these questions as powerful verbal test items in a classroom, where it is difficult to diagnose 30 kids quickly. This is a technique every teacher should learn.

The ball sitting on a table is not moving. It’s not moving because:

A. No forces are pushing or pulling on the ball.

B. Gravity is pulling down, but the table is in the way.

C. The table pushes up with the same force that gravity pulls down.

D. Gravity is holding it on to the table.

E. There’s a force inside the ball keeping it from rolling off the table.

This question not only catches common misconceptions, it diagnoses between those who have understood the ‘physics’. C is correct.

What can we do to preserve the ozone layer?

A. Reduce the amount of carbon dioxide in the air

B. Reduce the greenhouse effect

C. Stop cutting down the rainforest

D. Properly dispose of air conditioners and fridges

Looks like the designer ran out of options and put the last item in to make up the numbers, but it’s actually the right answer. Interestingly, this was one of dozens of mistakes in BBC Bitesize,

Which of the following is NOT an effect of burning fossil fuels?

A. global warning

B. ozone depletion

C. acid rain

D. smog

E. fog (right according to BBC)

Ozone depletion is also correct as the result of CFCs which are completely artificial (they did not exist in nature prior to synthesis by humans). They were used in air conditioning/cooling units, as aerosol spray propellants prior to the 1980s, and in the cleaning processes of delicate electronic equipment. They are not the result of burning fossil fuels.

Wiliam’s point about the imortance of classroom questioning, is that; 

“The variability at teacher level is about four times the variability at school level. If you get one of the best teachers, you will learn in six months what an average teacher will take a year to teach you. If you get one of the worst teachers, that same learning will take you two years. There’s a four-fold difference in the speed of learning created by the most and the least effective teachers. And it’s not class size, it’s not between class grouping, it’s not within class grouping – it’s the quality of the teacher.”

This led him to determine what separates good from bad teachers.

“And actually, new teachers are actually pretty bad. You don’t really learn to teach at all well until you’re six or seven years into the profession. And some recent data from Australia shows that the amount of value added by teachers actually carries on increasing for about twenty years.”

And here’s a brilliant paragraph.

“The key concept here—the big trap—is that teachers do not create learning. That’s true teachers do not create learning, and yet most teachers behave as if they do. Learners create learning. Teachers create the conditions under which learning can take place. Our schools don’t function like that, which is why somebody once joked that schools are places where kids go to watch teachers work.”

The solution, given the fact that reducing clss sizes is incredibly expensive, is to use diagnistic 'hinge' questions. This accelerates the teacher's knowledge of the state of learning of the learners and accelerates the learning.

Dylan William also co-authored, with professor Paul Blackgave, the brilliant Inside the Black Box: Raising Standards through Classroom Assessment, another ‘should be compulsory’ text for educators. This makes the obvious point that far too much teaching is simply 'chalk and talk'. If you don't believe this walk into any school and you'll see it in practice. It's a manifesto for more formative assessment.

Why questions matter

What matters? All of the above and more. Questions really do matter in learning. 

Questions and curiosity

First, they stimulate curiosity. Almost all of my learning as an adult has this dynamic. Something intigues me and I follow it up as I'm curious to find the answer. This is the great joy of having the internet as a resource. It has made this type of inquiry and research possible.

Questions and diagnosis

Good qestions diagnose your strengths and weaknesses. You don't know what you don't know and questions uncover the often uncomfortable truth that you know less than you thought you know.

Questions and improvement

Questions and searching for answers are fundamental to the process of learning. Roger Schank has been using this apporach in all sorts of contexts, and this truly structured Socratic approach, works well when used by a skilled practitioner. 

Questions and motivation

To create the conditions for learning, as opposed to just delivering content, questions are the true stimulus. 

Yet, despite these advantages, few have the real skills to either construct or deliver formative or summative feedback at the level necessary for true learning. It's a real skill.