True-False test
strategy
1. Go through the whole test getting those you know right
first.
2. If the known answers before and after the one you don’t
know are the same, choose the opposite, as there’s more alternation of TRUE and
FALSE than in a truly random sequence, so to choose a different answer from
the last one increases your chance of getting it right from 50% to 63%.
3. If the known answers before and after are different,
choose TRUE as there’s more right TRUE (56%) answers than FALSE (44%).
Multiple Choice test
strategy
1. Pick ‘none of the above’ or ‘all of the above’ as this
gives you up to a 90% increase on random guessing.
2. Pick B on a four choice test, as this gives you an 11% increase,
as human testers can’t randomise.
3. Avoid the previous choice gives you an 8% increase, again as
human testers can’t randomise.
4. Choose the longest item. Even on the supposedly
unbeatable SAT exams.
5. Eliminate outliers.
6. Always guess, as instinct can sometimes kick in to
produce the right answer.
Conclusion
We live in an age of constant assessment, that is a shame.
However, to live in an age of inefficient assessment is even worse, as smart,
coached and tutored students can ‘game’ the system. For assessment designers it
may be worth looking at these and other ‘game’ strategies to eliminate the
weaknesses of your tests. First randomise the right answers. Humans just cannot
randomise. Then look for the common biases. One final point. Randomisation is an algorithm. This is a case where algorithmic power of computers can eliminate human cognitive bias and weak competence.
10 comments:
Yep, usually B or C or the longest distractor in my experience . People who write tests (especially in elearning) often don't really fully understand the subject they are 'teaching' (or instructionally designing if you believe the BS job title hype).
It's a great shame that elearners are often being taught by people who have done the equivalent of cramming the night before an exam.
Still, it's all about the money and the graphics, isn't it?
Like you say, the problem is often in the design of the question and this requires more awareness and skill on the part of setters. Randomisation is useful but not universally good. Where there is a logical order to options (e.g. numbers) it is fairer to provide them in the logical order.
There are measures that can make MCQs less gameable. An underused strategy is negative marking to discourage guessing. I particularly like the idea of confidence-based marking (confident answers are required for top marks but there is punitive downgrading for confident+wrong selections). I think there is resistance because it openly acknowledges that there is gaming and answers are often guesswork.
Rob - certain truth in this but some of the worst tests in his sample were from Universities by professional lecturers and subject matter experts. They were the easiest to 'game'. It's a skill, whatever your background.
Indeed. The MCQ Is often the first port of call for most people when writing tests for the first time. I've got over 20 yrs experience in learning design and I must say that it can sometimes still take a lot of time to write an effective MC item. Throw confidence testing into the mix and there goes a whole heap of time.
The Guardian daily quizzes have some really skilled item writers on board.
How many educators have had any professional development in assessment design - certainly not most University teachers. However, it is worth looking at the practice in problem based medical education courses -a lot of MCQ in medicine that is really well designed and it certainly changed my opinion on MCQs. Try putting "design medicine mcqs for medical professionals" into Scholar.
Agree. The bottom line is that this is a skill and needs some training. It simply reinforces my opinion that teaching is not an art but a set of practices and competences.
It is getting better in Higher Ed. Many Academics are now more willing to learn how to teach/assess and their institutions are offering resources that help them to do this.
Hallelujah.
I had some very good training in the 80s on testing from a guy who worked in assessment for international aviation. He charged 1.5 days per question. It's all in the planning and user testing.
"teaching is not an art but a set of practices and competences." Donald, this reflects my own opinion that you don't need to know about cognitive psychology or have a grad diploma in Teaching and Learning to be a good teacher - just a good bag of tricks. (This is true for the design of good objectives tests also). Have you written elsewhere on this?
I have but not wholly convinced by that definition Bryan. Doesn't it beg the question 'What practices?'. I've seen widespread bad practices in education - whole word teaching, for example which lasted for 2 decades, learning styles etc. Some knowledge of psychology helps, especially knowledge of how memory works. This can stop the endless lecturing, cognitive overload etc.
Post a Comment