All of the Above – 20 ways to cheat MULTIPLE CHOICE questions

Many multiple choice questions are poorly written. What better way to expose these errors than write a cheat-sheet for learners?

Of course, writing good test items is far more difficult than many imagine. Many make obvious mistakes. An interesting way to coming at this problem is to do some reverse engineering. If you think this doesn’t work, think again. Poundstone number crunched 100 tests with a total of 2456 questions to get some of these statistical biases.

Second-guessing the test designer

So here goes with my 20 ways to cheat Multiple Choice tests:

1.  Skip the hard questions, mark them with a cross, and go back to them. This means you’ll not lose marks for unanswered easy questions.

2.  Cover the options and try to answer. Prevents being misled by clever wrong options.

3.  If in doubt choose ‘B’, poor questions designers do not truly randomise the right options and have a bias towards ‘B’. Next best is ‘C’.

4.  If in doubt choose the ‘longest option’. Question designers often cannot make a right option any shorter, but have complete freedom with wrong options. This is quite common.

5.  If in doubt choose TRUE, in true/false questions, as they come easier to mind for designers.

6.  Reverse answers. Statistically, there is more T/F alteration in tests than in truly randomized sequences, as the brain struggles to randomise properly. So, if you’re sure you’ve got one right, reverse the next answer.

7.  Eliminate the outlier. Look for similarities in options and eliminate outliers e.g. 4p-q, 2p+q, 4p+q, 3p+q. Look for these internal patterns.

8.  If two options are opposites, one is likely to be correct. Designers tend to create the opposite of the right answer as their first distractor, so one of them is likely to be correct.

9.  Favour options with careful qualifiers, such as ‘sometimes, occasionally etc.’ as tested knowledge usually has more finite than absolute qualities.

10.  Be wary of options with absolute qualifiers, such as ‘always, never etc’. As these are often too definite to be correct.

11.  Choose a middle order option i.e. out of 100, 150. 200, 250, choose 150 or 200. Designers tend to have a bias, where right answers tend to be lower than the highest and higher than the lowest option.

12.  For questions that demand an ‘except’ or ‘not’, mark each option with a T for true and F for false against each option. And underline the word ‘not’ as it’s sometimes missed.

13.  ‘All of the above’ and ‘None of the above’ are both significantly likely to be correct. For it to be correct, the writer has to design options that were all correct, so, if you can’t spot any wrong answers, or see that two or more are correct, it increases the probability of ‘All of the above’ being correct. Similarly with ‘None of the above’.

14.  Typo or punctuation error, the option is likely to be wrong. Writers tend to proofread correct answers only.

15.  Look for grammatical agreement between the question and its options; ‘An.....’ and words starting with vowels or agreement between subject, object or verb.

16.  If you’re stuck, go with the ‘Least bad rule’. Eliminate least likely answers first.

17.  Look for clues about answers from other questions. Designers often, unintentionally, put clues, even answers, to questions in other questions.

18.  Ignore never heard of answers. If you’ve never heard of the answer, it’s likely to be made up and incorrect.

19.  Go with your first impression. The more you read, the more you tend to read into the wrong options.

20.  Always guess, unless there is a penalty. It’s a 1 in 4 chance, so don’t give it up.

Conclusion

This crib sheet can be used by students or question designers to improve their tests. 

Next post - how to write a great MC question...
And there's 500 more tips like these, all categorised, here.

School of Athens: explains a lot about modern schooling?

If one artwork captures the roots of our Western intellectual tradition it is The School of Athens (Scuola di Atene) by Raphael. Note the title. The figures are set within a ‘school’ both the place, and metaphorically, the golden thread of a tradition that still has he influence on education today. The school is actually Roman architecture, not Greek, but is meant to echo the schools of the two central, principle figures; Plato (Academy) and Aristotle (Lyceum).

Plato and Aristotle

Plato steps forward and points to the sky (heavens), while Aristotle stands still with his hand level, palm down to the ground (real world). This represents two different philosophical traditions that were to shape, not only western philosophy but also religion and learning, both theory and practice. In their hands, Plato holds his Timaeus, Aristotle, his Ethics. This shows a divergence between the theoretical, cosmological and metaphysical concerns of Plato and the grounded, earthly and practical approach of Aristotle. They represent two schools of thought but also two approaches to schooling. This is a simplification but Plato, the rationalist is contrasted with Aristotle, the empiricist. This persists today in the arts/academic versus science/vocational debate around curricula and educational policy.

(see Plato and Aristotle as learning theorists)

Socrates

Another figure, stands off to the left, dressed simply in green, a secular colour in the Renaissance, in deep dialogue with a young man, with his back to Plato and Aristotle. Although the figure behind looks across to Plato, as it is through the Platonic dialogues that we know most about this man - Socrates. He had a profound influence on the western approach to learning that is still alive today. The sceptic, whose educational approach was to deconstruct through dialogue, strip away pre-conceptions and expose ignorance. He doesn’t conform to any of the traditions around him and survives today, in the Socratic method, as someone who believes in an approach that eschews lectures for dialogue, feedback and reflection. (see Socrates as learning theorist)

Mathematics

There is in this image, another theme, related to both Plato and Aristotle, but also other figures, such as Euclid and Pythagoras. Pythagoras is the figure writing in a book in the foreground on the left, surrounded by acolytes. He represents abstract mathematics and the idea that learning is about the master transmitting immutable knowledge to their students. His parallel figure in the foreground on the right is Euclid (some say Archimedes), leaning down to demonstrate his proofs, on what looks like a slate, with callipers, where the students are in discussion, working through the proofs in their heads. Again, this contrast exists between the didactic teaching of a canon and the more learner-centric view of the learner as someone who has to learn by doing and reflection.

Other figures

Diogenes sits as a sceptic, alone, looking at no one, in front of Plato and Aristotle. He’s a check on these systematic thinkers, representing another learning thread that was by this time coming alive in the University system and certainly came from the Greeks – scepticism, and its close relative, cynicism. There’s a host of other characters, such a Zoroaster and Averroes, showing non Greek threads but the main pantheon of teachers are mostly Greek.

Artists

That an intellectual tradition is represented as a great work of art is one thing, but Raphael also injected another theme into the fresco. He represents some of the figures from known representations of busts, others, it is speculated, have the faces of famous artists, Plato (Leonardo da Vinci), Aristotle (Giuliano da Sangallo), Heraclitus (Michelangelo), Plotinus (Donatello). Raphael is thought to have included himself, as the figure at the elbow of Epicurus (on left lifting the bowl from the plinth). The sculptures behind the figures are Apollo (left), God of music and light, and Athena (right) Goddess of wisdom, again reflecting rhetorically the arts and knowledge as underlying themes in learning. Again, we have a lasting theme in education, the role of the arts.

From philosophy to theology

It may seem odd that this painting was commissioned by a Pope and is to be found in the Vatican. However, remember that this fresco is one of many frescos in this room, and adjoining rooms, that represent largely Christian and theological issues. Theology had, well before this point and for many centuries, held an iron grip on the educational process, that was to continue, and never really disappear, even in our supposedly secular age.

Technology

There’s no large-scale lecturing in this image, although nascent technology in the several books (3), scroll (1), pens and notebooks in which notes are being taken (3), compasses (1), globes (2) and what appear to be slates (2), are already being used to assist learning and teaching.

Conclusion

The main triumvirate of Greek philosophers define the strands for learning and educational theory that are alive today. The great schism between the academic and practical was set in motion and the Socratic tradition defined, but, so often ignored.

Eurozzzzone - a giant squeezyjet, tax-funded, city-break, gravy train

Have a look at this ELIG 2015 conference. Check out the speaker list. Notice how non-pan-European it is? Notice its almost complete focus on Germany and he fact that Microsoft & IBM are there but almost no one from the rest of Europe? Their slogan is 'We change the way Europe learns' - oh yeah!

Weird experience

A few years ago I was invited to attend a one week trip to Spain, paid for by some European grant. I was the UK rep and there were reps from each of the other 14 countries. It was a revelation, in that it was a complete waste of time. At the end of the fact-finding trip we got together to discuss our findings, which to paraphrase the discussion, showed that Spain was milking European grants like a starved piglet at the teat, but to no great effect in terms of entrepreneurial progress (the trip was about entrepreneurship). I was the only business person of the 14, and was politely told to forget the criticisms and write the report free from any negative conclusions. It turned out to be a massive fraud with ghost employees and huge sums extracted from the system. Many are now in jail.

It was a shocking introduction to the neverneverland that is the European Union. As the Euroland implodes in a sort of low-key rerun of the second world war, one wonders what all that money spent on ‘euro-learning’ actually brought us?

Research

Answer - nothing. Academics have been sqeezyjetting around Europe for years to meetings that were little more than excuses for short-breaks or a nosh-up. The collaborative projects weren’t designed around competences or goals, merely a bunch of people who were good at form-filling. Then there’s that obnoxious group of middlemen, no better than street drug runners, who promise to get you a chunk of the motherload (for a fee of course). The whole sorry tale is largely one of useless research on useless projects set up by worse than useless hustlers. In practice, the real work was being done by hard-working people in real companies and organisations doing things with real people in the real world. European projects are like the Eurovision Song Contest, countries send their least talented people to a contest that is best known as a parody of the real world and the output is woeful. It used to be something to laugh at, now it’s a politicised, block vote idiotfest. I’m positive about Europe as a single market but sad that so much money has been spent with so little meaningful output. The real action is in hte commercial conferences, such as Online Educa, that genuinely try to attract a pan-European audience and do so with little or no government support. 

One market myth

Europe is not a single market in education and training because people learn best in their first language, and in the UK we don’t have a second. Almost all education and training in Europe is delivered by local and national suppliers. There has huge effort to create pan-European companies by supporting pan-European research, but it hasn’t worked. Giunti Labs seem to have been on some sort of permanent financial drip from Brussels, along with several other companies that would never have survived in the commercial world.

Let’s take the ‘E’ out of e-learning

ELEARNINGEUROPA, EU4ALL, ERGO, ECLO, ELIG, EDEN, EFQUEL, EIFEL, EMDEL and on and on it goes…..Dozens of crap acronyms all staring with ‘E’ and hundreds of administrators, unread reports, AGMs, meetings and conferences. When I ran an e-learning company I had absolutely nothing to do with any of these or any other European quangos. Have they delivered? I think not. We're far more likely to look to the US and the E-learning Guild, that attempts to dig deep into real practice, than any European organisation, wieghed down by beurocrats. Would the world miss them, I think not. Let’s take the ‘E’ out of e-learning – Europe that is!

10 Oculus Touch ideas that bring VR within reach of learners. Let’s do it!

I’ve put an Oculus headset on hundreds of people from 5 year to 85 year olds, all over the world. I’ve even put one on a dog! The reaction is the same ‘Awesome!... Amazing!... I want one now!” Well, apart from the dog, who barked.

There are many reasons why I think this new medium is relevant to learning. And remember, VR is a medium, not a gadget.

What is Oculus Touch?

When you open your Oculus box, you’ll get a lightweight headset with built-in headphones (for 360 degree audio) and a wireless Xbox controller. Within months you’ll also be able to buy two hand controllers that will provide hand-presence, the feeling that your hands are there in that 3D world. This will unlock lots of interesting learning applications. Each has a small wireless pad with a ring that gives you three things:

·      motion-based input

·      physical controls

·      haptic feedback

As well as the ability to move your hands, tracked by the same IR LED system that tracks your head, you’ll be able to input actions and receive haptic feedback (vibrations).

1. Why? Head and hands

This homunculus shows exactly why VR needs handsets. Our brains are huge but lots of that brainpower is related to our hands. We think with our brains but do most of what we need to do in life with our hands. So the VR headset alone is not enough, we want to look, then we want to touch, then handle, manipulate. Arms and hands are what we use for most practical tasks, even gestures in communications. Even an Xbox controller, that will come with the Oculus Rift in Q1 next year, is OK for moving through environments, making choices from menus and shooting things, isn’t enough. As you believe you’re in another world, the natural thing to do is lift your arms, make gestures, reach out, touch things, pick them up, place them and so on. This ‘near-field VR’ is an important advance and opens up a whole new raft of learning applications

2. Relevant learning theory

First up, some basic lessons from proven learning theory. VR gives you immediate and involuntary attention (necessary condition for learning), intense emotion, low-cost simulated worlds, ability to learn from failure, context and real increases in retention. But the big win is in giving learners the ability to learn by doing. This is rare in institutional learning, yet we know that it is, in practice, how most us of learn. For more on this see this article.

http://donaldclarkplanb.blogspot.co.uk/search?q=VR

3. Tool use

For vocational tasks where one needs to use real objects, such as tools and meters, this will be possible, for example, in the training gas inspectors and any skills that involves the manipulation of real objects. I can see this progressing to the training of everyone from hairdressers with scissors to surgeons with scalpels.

4. Build

When one has to dismantle or build an engine, boiler or any other piece of machinery, these handsets will allow you to do maintenance and perform any task that involves the placing, insertion, pushing, pulling, screwing, unscrewing and joining of objects. The possibilities in porn have not gone unnoticed but there’s no end of possibilities in learning.

5. Vehicle sims

For all of those driving skills or any other vehicle skills on boats, trains and planes, one can see how manual manipulation matters. More than this you’ll be tested on defensive driving, on why mobile use is dangerous, even driving while under the influence of drugs.

6. Lab work

Lab work largely involves doing things within arms length. Techniques can be practised and perfected without using or breaking expensive kit. MRI machines and a host of other expensive healthcare kit can also be used without wasting valuable investigative time on real equipment.

7. Sports

In sports where one uses one arms and hands – that’s almost everything - I can see this being used in coaching, especially in technique – in everything from tennis, golf and fly fishing to throwing a javelin. Dangerous pastimes, such as parachuting or scuba-diving could benefit from preparatory training.

8. Gestures

For soft skills, the ability to use body language and gesture may be important. But it is the use of gestures as controls that also matter, so that you’re unencumbered by navigational menu issues or Xbox controller issues. You will be able to shake hands, thumbs up, give the middle finger, wave goodbye.

9. Training, assessment & certification

I have seen a 3D gas inspection simulator that trains, assesses and gives partial certification. This could provide an opportunity for huge reductions in the cost of all three. At present vocational learning is plagued by high cost assessment.

10. Social media

Make no mistake, Zuckerberg wants to take social media into VR. With one and a half billion regular Facebook users, this is the big audience, the big win, not just the gamers. Want to hold a virtual meeting, seminar, tutorial, class, collaborative learning session? That is mind-blowing.

Renaissance of vocational learning

These handsets give you more freedom and movement, more importantly they allow you to DO more. This will allow us to implement what the likes of James, Dewey, Kolb and Schank have been saying for decades. Many countries are now turning to re-examine vocational learning. In the UK, the winning party fought the election on the promise of 3 million apprenticeships, Finland, S Korea, Singapore and other countries are reassessing the excesses of overly academic approaches to education. This approach could make vocational cool. Imagine a world where young people want to do things, rather than spend years inside lecture halls. We learn by doing – so lets do it!

‘The Strange Case of Rachel Doleza’: why diversity training does more harm than good

NAACP Chapter President Rachel Doleza fakes her blackness, and rejects her parents, to black-up, play a role and get paid to be outraged on behalf of others. Granted, it’s an extreme case, but it’s not unusual. In the world I know, education and training, there’s a vast army of people playing the same game. You wouldn’t spot them, as they’re normally plain, old, white, middle-class folks who deliver compliance training on diversity. Hundreds of millions are spent yearly on fees to these people, yet there is no evidence that it works. In fact, there is very strong evidence to show that it doesn’t work.

As a Trustee of several public bodies, I’ve been subjected to several of these sessions – here’s but one example. But let’s look at the substantive evidence.

Diversity training damned by research

Has diversity training become and end in itself rather than a means to an end? The vast amount of time and money spent on diversity training, when evaluated, is found wanting, mostly ineffective, even counter-productive. With evidence from large scale studies, from Dobbin, Kalev and Kochan, as well as many other focused pieces of research, you'd have thought that the message would have got through. The sad truth is that few on either the supply or demand side, give a damn about whether it works or not. It's become an article of faith.

Dobbins: Virtually no effect… generates a backlash

Harvard’s Frank Dobbin conducted the first major, systematic study of diversity programmes across 708 private sector companies, using employment data and surveys on employment practices. His research concluded that, “Practices that target managerial bias through…diversity training, show virtually no effect.” In fact, “Research to date suggests that… training often generates a backlash.”

Many other studies show that diversity training has activated, rather than reduced diversity (Kidder et al 2004, Rynes and Rosen 1995, Sidanias et al 2001, Naff and Kellough 2003, Benedict et al 1998, Nelson et al 1996). These are all referenced in the report. The research is a very thorough piece of work, and well worth reading, which is why it was completely ignored.

Kalev: Diversity training harmful

Most diversity training is not evaluated at all or languishes in the Level 1, lala land of ‘happy sheets’. So check out Alexandra Kalev’s study from the University of Arizona. 31 years of data from 830 companies – how’s that for a Level 4 evaluative study! Her latest study found, after the delivery of diversity training, a 7.5% DROP in women managers, 10% DROP in black women managers and a 12% DROP in black men in senior management positions. There were similar DROPS among Latinos and Asians.

The strength of this study comes from the quantity and integrity of the data. It relies on compulsory federal EEOC (Equal Employment Opportunity Commission) filings on the number of women and people of colour in management, along with details of diversity training programmes.

The bottom line is that the vast majority of diversity courses are useless, especially when driven by HRs perception of avoiding prosecution. The problem centres around courses run in response to legislative and external pressures. Kalev found that , "Most employers….force their managers and workers to go through training, and this is the least effective option in terms of increasing diversity. . . . Forcing people to go through training creates a backlash against diversity."

Diversity courses are “more symbolic than substantive" says University of California LAW Professor, Lauren Edelman, She independently reviewed Kalev's study and concluded that the problem was training in "response to the general legal environment and the fact that organizations copy one another."

Kochan: built on sand

Thomas Kochan, Professor of management at MIT’s Sloan School of Management’s five year study had previously come to the same conclusions, "The diversity industry is built on sand," he concluded. "The business case rhetoric for diversity is simply naive and overdone. There are no strong positive or negative effects of gender or racial diversity on business performance." The problem, according to Kochan, is the bogus claim that diversity leads to increased productivity. This is simply unproven as there is little or no hard data on the subject. Kochan found that none of the companies he contacted for his study had carried out any systematic evaluation of diversity training. Evidence around productivity is mostly anecdotal and repeated as a mantra by interested parties.

Pendry & Stewart: no evaluative evidence

Louise Pendry of Exeter University claims that there’s no evaluative evidence showing that these programmes work. Even worse, many may do more harm than good. Tracie Stewart, a professor at Georgia University, has identified "backlash" or "victim blame", after some courses, where the learners harbour resentment against other minority groups for the way they are made to feel. Rather than bringing people together, it may be reinforcing differences.

Munira Mirza: damning testimonies

Munira Mirza investigated diversity training for the BBC and uncovered some awful training, including the popular Jane Elliot’s ‘blue eyes/brown eyes’ classroom courses. What was interesting were the comments posted after the broadcast:

When I was about 12 we had a policeman come in to school to talk about racism. He showed us a photo of a white man in police uniform running after a black man in jeans. He asked us what we thought was going on. Everyone- including a black child that he pointedly asked -said that it was a criminal being chased by a policeman. We were then told that we had made a "racist assumption" as actually the black bloke was a plain-clothes police officer. No-one raised the point that we would have probably said the exact same thing if the plain clothes officer had been white and a load of 12 year olds were told that they were racist. How helpful was that?

Hannah, Peterborough

You cannot over-estimate the damage to race relations that "diversity awareness" training is causing in this country. It's having the opposite effect to that intended, causing divisions, resentment, and an increase in judgements based on race, where previously such things were actually quite rare. How do I know this? I was involved in putting together a diversity "toolkit" for a government department, and saw first-hand the effect it had as it was rammed down the throats of the staff.

Michael, Brighton UK

This is an example of companies trying to see if two wrongs really do make a right. I don't doubt that some people are racist in the workplace, but punishing many because of the actions of a few is ludicrous.

Andy Thorley, Crewe, Cheshire

Groupthink

Companies, worldwide spend many hundreds of millions of dollars each year on diversity training. The tragic truth is that most of this is wasted. Groupthink seems to be at the heart of the matter. Groupthink among people who employ and promote people like themselves creates the problem. Groupthink among compliance training companies ,who simply do what they do without supporting evidence and tout ineffective ‘courses’. Groupthink in HR, who find it easier to just run ‘courses’ rather than tackle real business problems. The whole edifice is a house of cards.

Conclusion

One of the problems, that Dobbin, Kalev and Kochan found, was the focus on ‘sensitivity training’ where people are often forced to focus on interpersonal conflict. These were the training courses that produced a backlash, as they were intrinsically accusatory. One bright spot was the finding that some diversity initiatives, namely those that were voluntary and aligned with business goals, were successful. This is similar to Professor Frank Dobbin’s study at Harvard, who showed, in his massive study that ‘training’ was not the answer, and that other management interventions were much better, such as mentoring. Rachel Doleza was sucked into a fake world that does real harm to diversity, equal opportunities and diversity in the workplace. We don't solve these problems by 'pretending' to be what we are not.