All of the Above - how to cheat Multiple Choice questions

Forget objectives, start with questions

I never liked the 'start with the learning objectives' advice on designing content, and preferred to start with writing and polishing these objectives as test items. This seemed that much more real, practical and relevant than forced 'Mager-like' objectives. Get the assessment right and the rest seemed to follow. This, I suspect, is why students much prefer to get past test papers to establish the real contents of a course.

Of course, writing good test items is far more difficult than many imagine, which is why many tests and not really tests of understanding, merely tests of recall. An interesting way of coming at this problem is to do some reverse engineering. Ask how students can cheat their way through a test.

There's the usual crib notes. The best I heard was, remove the lable from a bottle of orange juice, writing the crib notes on the back of the label.paste it back on, then drink orange to reveal the notes. But let's assume that pure cheating is out. What are you left with?

Second-guessing the test designer

Many multiple choice questions are poorly written. What better way to expose these errors than write a crib sheet for learners? So here goes with my 20 ways to cheat Multiple Choice tests:

1.    Skip the hard questions, mark them with a cross, and go back to them. This means you’ll not lose marks for unanswered easy questions.

2.    If in doubt choose ‘C’, poor questions designers do not truly randomise the right options and have a bias towards ‘C’. Next best is ‘B’.

3.    If in doubt choose the ‘longest option’. Question designers often cannot make a right option any shorter, but have complete freedom with wrong options.

4.    Look for similarities in options and eliminate outliers (in bold) e.g. 4p-q, 2p+q, 4p+q, 3p+q.

5.    Now note that there’s only one ‘-‘, which makes 4p+q more likely. Look for these internal patterns.

6.    ‘All of the above’ is likely to be correct. For it to be correct the writer has to design options that were all correct, so, if you can’t spot any wrong answers, or see that two or more are correct, it increases the probability of ‘All of the above’ being correct. Similarly with ‘None of the above’.

7.    Choose a middle order option i.e. out of 100, 150. 200, 250, choose 150 or 200. Designers tend to have a bias, where right answers tend to be lower than the highest and higher than the lowest option.

8.    For questions that demand an ‘except’ or ‘not’, mark each option with a T for true and F for false against each option. And underline the word ‘not’ as it’s sometimes missed.

9.    If there’s a typo or punctuation error, the option is likely to be wrong. Writers tend to proofread correct answers only.

10.  Look for grammatical agreement between the question and its options; ‘An.....’ and words starting with vowels or agreement between subject, object or verb.

11.  Go with your first impression. The more you read, the more you tend to read into the wrong options.

12.  If you’re stuck, go with the ‘Least bad rule’. Eliminate least likely answers first.

13.  Look for clues about answers from other questions. Designers often, unintentionally, put clues, even answers, to questions in other questions.

14.  If you’ve never heard of the answer, it’s likely to be made up and incorrect.

15.  First cover the options and try to answer. Prevents being misled by clever wrong options.

16.  If two options are opposites, one is likely to be correct. Designers first made up option is likely to be the correct option’s opposite.

17.  Favour options with careful qualifiers, such as ‘sometimes, occasionally etc.’ as tested knowledge usually has more finite than absolute qualities.

18.  Conversely, be wary of options with absolute qualifiers, such as ‘always, never etc’. As these are often too definite to be reasonably correct.

19.  Always guess, unless there is a penalty. It’s a 1 in 4 chance, so don’t give it up.

20.  Eliminate obvious answer on 4 options then guess, don’t fail to answer. This reduces the odds from ‘1 in 4’ to ‘1 in 3’. Far better than just guessing or not answering, depending on any penalty scores for wrong answers.

None of the above

This crib sheet can be used by question designers to improve their tests. Good students put themselves in the shoes of the test designer to improve their chance, so the more you know about their techniques, the better designer you’ll be.

I still see binary option questions with ‘Try again’ logic, grammatical disagreement and stupid options. Some time back in this blog, after they refused to respond when I emailed the mistakes through, I had a go at BBC Bitesize’s science tests, as they were riddled with these errors. 140 comments later, it still pops up on the home page when you go to BBC Bitesize through Google.

Writing good multiple choice questions is not easy. What’s easy is simply extracting all the nouns, objects and quantities, then testing for recall. The trick is to push beyond this to test understanding. It’s not the ‘what’ but the ‘why’ that often matters but remains untested.

Why questions matter

Professor Dylan Wiliam, Deputy Director of the Institute of Education, Professor of Educational Assessment gave a brilliant ALT talk (Seb Schmoller put me on to this) on what calls ‘hinge questions’, questions that literally diagnose poor understanding. He explains how one can use these questions as powerful verbal test items in a classroom, where it is difficult to diagnose 30 kids quickly. This is a technique every teacher should learn.

The ball sitting on a table is not moving. It’s not moving because:

A. No forces are pushing or pulling on the ball.

B. Gravity is pulling down, but the table is in the way.

C. The table pushes up with the same force that gravity pulls down.

D. Gravity is holding it on to the table.

E. There’s a force inside the ball keeping it from rolling off the table.

This question not only catches common misconceptions, it diagnoses between those who have understood the ‘physics’. C is correct.

What can we do to preserve the ozone layer?

A. Reduce the amount of carbon dioxide in the air

B. Reduce the greenhouse effect

C. Stop cutting down the rainforest

D. Properly dispose of air conditioners and fridges

Looks like the designer ran out of options and put the last item in to make up the numbers, but it’s actually the right answer. Interestingly, this was one of dozens of mistakes in BBC Bitesize,

Which of the following is NOT an effect of burning fossil fuels?

A. global warning

B. ozone depletion

C. acid rain

D. smog

E. fog (right according to BBC)

Ozone depletion is also correct as the result of CFCs which are completely artificial (they did not exist in nature prior to synthesis by humans). They were used in air conditioning/cooling units, as aerosol spray propellants prior to the 1980s, and in the cleaning processes of delicate electronic equipment. They are not the result of burning fossil fuels.

Wiliam’s point about the imortance of classroom questioning, is that; 

“The variability at teacher level is about four times the variability at school level. If you get one of the best teachers, you will learn in six months what an average teacher will take a year to teach you. If you get one of the worst teachers, that same learning will take you two years. There’s a four-fold difference in the speed of learning created by the most and the least effective teachers. And it’s not class size, it’s not between class grouping, it’s not within class grouping – it’s the quality of the teacher.”

This led him to determine what separates good from bad teachers.

“And actually, new teachers are actually pretty bad. You don’t really learn to teach at all well until you’re six or seven years into the profession. And some recent data from Australia shows that the amount of value added by teachers actually carries on increasing for about twenty years.”

And here’s a brilliant paragraph.

“The key concept here—the big trap—is that teachers do not create learning. That’s true teachers do not create learning, and yet most teachers behave as if they do. Learners create learning. Teachers create the conditions under which learning can take place. Our schools don’t function like that, which is why somebody once joked that schools are places where kids go to watch teachers work.”

The solution, given the fact that reducing clss sizes is incredibly expensive, is to use diagnistic 'hinge' questions. This accelerates the teacher's knowledge of the state of learning of the learners and accelerates the learning.

Dylan William also co-authored, with professor Paul Blackgave, the brilliant Inside the Black Box: Raising Standards through Classroom Assessment, another ‘should be compulsory’ text for educators. This makes the obvious point that far too much teaching is simply 'chalk and talk'. If you don't believe this walk into any school and you'll see it in practice. It's a manifesto for more formative assessment.

Why questions matter

What matters? All of the above and more. Questions really do matter in learning. 

Questions and curiosity

First, they stimulate curiosity. Almost all of my learning as an adult has this dynamic. Something intigues me and I follow it up as I'm curious to find the answer. This is the great joy of having the internet as a resource. It has made this type of inquiry and research possible.

Questions and diagnosis

Good qestions diagnose your strengths and weaknesses. You don't know what you don't know and questions uncover the often uncomfortable truth that you know less than you thought you know.

Questions and improvement

Questions and searching for answers are fundamental to the process of learning. Roger Schank has been using this apporach in all sorts of contexts, and this truly structured Socratic approach, works well when used by a skilled practitioner. 

Questions and motivation

To create the conditions for learning, as opposed to just delivering content, questions are the true stimulus. 

Yet, despite these advantages, few have the real skills to either construct or deliver formative or summative feedback at the level necessary for true learning. It's a real skill.

 

BBC Bitesize: Warning – may damage your child’s mind!

End of the year and my kids are doing some homework – in this case the BBC KS3 (11-14) bitesize science tests in biology, chemistry and physics. Three sets of short tests and they’re howling with laughter. “This is just a load of crappola, dad” says Callum, then called me over every time he hit a howler, and that was pretty often. 

 JUST PLAIN WRONG 

Let’s start with the stuff that’s simply WRONG. 

Q The products of photosynthesis are:

carbon dioxide and water (right according to Beeb!)

glucose and oxygen

light and dark 

Initially Callum thought he had learnt this the wrong way round, but no, the Beeb clearly have no idea what the ‘products’ of photosynthesis are – right answer is ‘glucose and oxygen’. 

OK, to err is human, we all make mistakes, but it doesn’t get much better in chemistry. 

Q A liquid has a pH of 1 – what does this mean? 

it must be sodium hydroxide solution 

it is strongly acidic 

it is weakly acidic (right according to Beeb!) 

Wrong again, the right answer is that ‘it is strongly acidic’. 

Q Which state is shown in this diagram? Diagram shows particles all touching each other in a beaker. 

solid

liquid (right according to Beeb!)

gas 

Completely wrong, ‘solid’ should be the right answer. The particles are tightly packed and all touching each other. 

Q Which ammeter will have the biggest reading 

 Series circuit showing: ammeter – bulb - bulb – ammeter. ammeter 1 ammeter 2 (right according to Beeb!) they will read the same 

To quote the Beeb’s own words in the revision section, “It doesn't matter whether you connect the ammeter on the left or right of the lamp, it will still give the same reading". 

Q The diagram shows the waveform of two notes played on a musical instrument. Which sound is louder, A or B? 

Two waves (A & B) are shown. A has a higher amplitude but same frequency as B. A B (right according to Beeb!) 

WRONG – it is A. 

TWO RIGHT ANSWERS Then there’s the questions with two possible right answers. 

Q What natural process breaks large rocks into smaller ones? 

erosion

sedimentation

metamorphosis

crystallisation

weathering (right according to Beeb!) 

Erosion involves movement, and often results in the breaking down of large rocks into smaller ones, e.g. in river beds, cliff erosion etc. 

Q Which of the following is NOT an effect of burning fossil fuels? 

global warming

ozone depletion

acid rain

smog fog (right according to Beeb!) 

Ozone depletion is also correct as the result of CFCs which are completely artificial (they did not exist in nature prior to synthesis by humans). They were used in air conditioning/cooling units, as aersol spray propellants prior to the 1980s, and in the cleaning processes of delicate electronic equipment. They are not the result of burning fossil fuels. 

Q What needs to be done to this circuit so that the lamp lights up? 

Series circuit shown open switch and a bulb. close the switch add another lamp add a cell (right according to Beeb!) 

 Again the first answer is also correct, and as it appears first, many choose it before reading on. 

SPELLING, CAPITALISATION and MISSING WORDS 

Q Which part of the skelton protects the lungs? Skelton? 

Q What is the main reason why need protein in our diet? Missing word. 

Q  influenza, flu, is caused by? No capital at start of sentence. 

Q Which of the following is not a use of artificial satellites?? Double punctuation. 

Q Click the diagram that shows correctly how light beams and is reflected by a convex mirror. Sentence doesn’t make sense. 

STUPID OPTIONS This one is annoying as it makes the already awful tests invalid by giving the learner stupid and obviously wrong options. 

Q Which of the following is not a fossil fuel? 

velociraptor (STUPID OPTION)

coal

oi

l 

Q What is magma? 

a chocolate ice cream (STUPID OPTION)

molten rock

bubbles of gas 

Q What can happen to a plant growing in soil without minerals? 

it grows really well

it grows poorly if at all

it leaves to find somewhere else to grow (STUPID OPTION) 

Q What are the tiny air sacs in the lungs called? 

ravioli (STUPID OPTION)

bronchioles

alveoli 

Then there’s one that is so stupid you can’t get it wrong. 

Q What does the hazard symbol mean? Sign with word ‘CORROSIVE’ on it! 

corrosive (IMPOSSIBLE TO GET WRONG)

irritant

harmful 

TOO ADVANCED OR JUST CONFUSING To give KS3 kids questions that are too advanced or plain confusing is another all-too-common fault. Then there’s the downright confusing. 

Q Which of the following is not a way that we can reduce the use of fossil fuels? 

leaving the TV on standby all night

walking instead of going in a car,

turning the lights off when we leave a room

This becomes an exercise in logic for the learner with its double negative in the question and, in option 3, a triple negative! 

Q What does the heat energy supplied to an ice cube do when the ice is melting? 

it warms it up 

it makes the ice particles expand it breaks some of the bonds in the ice * 

 The first option is arguably correct, it does warm it up, and as it is the first option, many go for it. 

Q Which contains more heat energy, 

100 grams of water at 0 degrees C 

or 

100 grams of ice at the same temperature? 

Strange question – absolutely designed to confuse at this level. They claim the water but a curious learner may question whether water can be water at 0 degrees C. 

Can it get any worse? Yes it does. This question is actually impossible to answer: 

Q What is the value of the clockwise moment on the wooden bar in N/cm, if each coin As half the question is missing, it is impossible! I kid you not. I could go on, as there’s lots of other errors, but I’ve already shown that the content is full of wrong answers, confusing (at times impossible) questions and poor English. 

The writers show a lack of basic science, as well as a lack of assessment design skills. Should these people be allowed near our children? You can go through the BBC learning content and find this slapdash approach in almost everything they do. Forget the idea that BBC=QUALITY. As a licence payer it makes me mildly annoyed, as an educator it makes me furious.

Skype learning – 7 great benefits

You can always spot a fabulous technology when it can be used as a verb, like email, text, tweet. I’ll ‘Skype’ you, is one of those wonderful verbs. Over the last two years I’ve been doing voluntary Maths and Science tuition for kids that find these subjects difficult. It’s been a mix of face-to-face and Skype. So what follows is a short comparison between these two techniques.

Free!

Skype is one of the wonders of the web, mainly because it’s free. Who would have thought that videoconferencing would cost nothing and that any old Joe from any old computer, and many phones, could do it for free? In these frugal times that’s a gift from the heavens.

Death of distance

Skype is a classic ‘death of distance’ technology. It quite simply frees us from the tyranny of location and cost. Both teacher and student can be literally anywhere. There’s nothing worse for a young person than having to trek round to a tutor’s house in the dark to do some ‘learning’. Booting up Skype is so much quicker and easier.

Increased focus

Skype forces both teacher and student to focus. This may be because you feel that you’re using up valuable online resource, even though it’s free. In any case, there is real sustained attention, which I think is better than a one-to-one face-to-face session. You are far less likely to drift off-task, as either teacher or learner. That means more learning in less time.

True dialogue

The fact that you’re not sitting next to them, and leaning into them, gives them time to breath, think, reflect and respond in their own time. It’s far more measured, with properly paced, dialogue, as the teacher is less likely to talk over the student and more likely to wait until they give a thoughtful response. Dylan William has shown that teachers tend to ask questions then jump in too early when the student fails to respond. I’ve found his 3 second rule (wait 3 secs before saying anything) much easier when online. Both sides take greater care to participate in a structured and constructive dialogue. Of course, you can be even more structured using the message service, which forces you to wait until the other person has responded with a written (and therefore considered) answer.

Prevents peer problems

But it’s the subtler issue of peer distance that really surprised me. Let’s face it, adults and teachers are not the peers of teenagers. We’re the opposite of peers, in the sense that whatever’s cool for us is the opposite of cool for them. I’ve found that being online, brings with it a healthy form of psychological distance that prevents peer problems. It’s more of an adult to adult conversation in the sense that the technology is a leveller. It puts you both on the same psychological plane.

Shared learning resources

When it comes to doing things, like setting a problem, responding with an answer, illustrating a point with a diagram or downloading a past paper or online resource such as BBC Bitesize, you can do so while keeping the Skype channel open. No need to have the video on, just have the new window full screen and the audio dialogue can continue. This is great as you both have your full attention on the content, not the psychological noise you get in face-to-face sessions.

Extras

You can record your Skype sessions for free, integrate with outlook, create alerts, use whiteboards, tutoring tools (just click on ‘Conversation’ then ‘Extras)’. There’s literally dozens of tools that allow do anything from customise to the sharing of files and resources.

Conclusion

Some of these virtues are simple and clear; it’s free and frees you from the tyranny of location. Others advantages are more psychological; increased attention, better dialogue and levelling out of peer effects. Lastly, there’s the practical advantages around shared online resources. Bottom line: Skype’s a vastly underused tool that’s made for learning.

Libraries are barriers to reading

100 Classic Books on Nintendo

Who would have thought? I’ve just seen the ‘100 Classic Books’ title advertised on prime time TV, just after Big Brother, for the Nintendo DS. Brain Training was a hinge product.  It changed the entire games market. Nothing will ever be the same again. But this is even bolder.

Of course, the traditionalists will be waving their reading glasses in horror, as usual. But to turn books into a fetish is simply to deny learning and access by those who need it most. Real books are great, but let’s not confuse the medium with the content. Just as journalists and newspaper owners fail to realise they’re in the ‘news’ not the ‘newspaper’ business, so book fans and publishers fail to realise that this is about reading, not books. Books are simple a piece of technology. A damn good piece of technology, but one that has some strengths and lots of weaknesses. In time its weaknesses will outweigh its current strengths.

Books destroy trees, need to be expensively transported and stored in expensive libraries and retail outlets. Sure they’re portable, but only one or two at a time, not a 100 or 1000? You can’t search them, and they’re difficult to bookmark, highlight, hyperlink or comment without defacing the product. In time, and it may be a long time, books will be read on screens.

Let’s face it, 100 books for less than £20 is 20p per book and the advantages are portability, storage and bookmarking. William Shakespeare, Jane Austen, Charlotte Bronte, Charles Dickens, Mark Twain, R L Stevenson, Jack London, Joseph Conrad, Thomas Hardy, Jonathan Swift, George Elliot, Edgar Allen Poe, Lewis Carroll, Jules Verne and lots more, available on a games console. It’s all good.

This is the shape of things to come, just a tiny glimpse of the possibilities in learning. Nintendo have taken a leaf out of Amazon’s and Sony’s book, with Amazon’s Kindle and Sony’s Reader both available with massive downloadable libraries. My brother-in-law, a busy man, who’s always on the move, swears by the latter.

We can now see where this can lead us, or more specifically lead our children. Why lock up knowledge and the ability to learn in libraries and schools, when we can publish and distribute it at marginal cost to everyone. As long as we publish in open standards, the devices will just keep on coming. Leave the device design to the experts, like Bill Joy, co-founder of Sun, I believe that Moore's Law will produce $10 devices by 2020, possibly a lot earlier – we just need to focus on free content.

Fill their boots with books

In my book, this is a groundbreaking movement in education. It is not beyond the wits of government to be bold here and recommend an entire ‘digital canon’ for every child in the country.

Take 1000 or 10,000 books, all of the BBC Bitesize content (we the public have paid for it, surely we own it), lots of e-learning, at all levels, language learning, and give it away to every schoolchild for free. Just hand over the entire canon, all GCSE and A-level subjects and lots of juicy extras. The cost would be a tiny fraction of the overall education budget. In fact, I think it can be done at no cost at all.

Libraries as expensive warehouses

How? This may sound like a contradiction – encourage reading by closing the most costly libraries. There are lots of them. The cost of borrowing a book in some public libraries is greater than the cost of the book itself. This may be hard to believe, but it’s true. Divide the actual cost of the library by the number of borrows per year – it’s shockingly high. I don’t mean all libraries or university libraries, just costly public libraries.

Public libraries are no longer encourage reading. In the age of digital abundance, and cheap books, they’re an expensive obstacle to reading. Libraries spend inordinate amounts of time trying to fine people and recover books that people just find too inconvenient to take back. They stop reading from libraries as they criminalise readers. Librarians have become debt collectors.

My local library in Brighton is a beautiful, frightfully expensive, award winning building, but inside is a scrappy warehouse of cheap shelving and a very sparse book collection. Many libraries are just like this, more like bad second-hand book or charity shops. They can’t hope to match the demand-led approach of a real bookshop.

What the planners had to do, to make the idea fly, was include a CD and DVD lending facility. In other words it had to become a Blockbuster to survive. This is the cul-de-sac that the modern library faces, as in the age of digital distribution and downloads; it’s a service that is heading towards vanishing point.

It wouldn’t be so bad if they actually took a business-like view of the world. This New Year I stepped into the local library in my parent-in-laws’ home town, in Scotland. The library had fewer customers than staff, and I simply wanted to use the internet facility they advertised in the window. I was told that I had to be a local resident. Even the offer of payment was rejected.

Then there are the book wardens – sorry librarians. Let’s be honest, they’re mostly just warehouse workers ordering, stacking, handing out, taking in and stacking again. Yet they cost the earth. As graduates (in stacking?) they demand salaries way beyond what the job requires. And many are seriously deficient on the customer care side. The main cost of any public library is the inflated salary costs. This is why the borrowing cost per book in many libraries has become absurd.

OK, I’m sure there’s a few tramps out there and those earnest parents who drag their children along every Saturday, when they’d much rather be playing football or playing computer games, who’ll be seeing this as an affront to civilisation, so I’ll try another tack.

Close down a whole swathe of libraries and encourage, even subsidise, the big bookshops, such as Borders or Waterstones, to expand their activities. They have all the best sites, good coffee, helpful and knowledgeable staff, and better book collections. Give us all some tax breaks on buying books.

Robot teachers – endlessly patient, CPD updates in seconds

The article in the New York Times will no doubt drive traditionalists to apoplexy, so there must be something in this report about the rise of robots in schools. Imagine a teacher that is endlessly patient, always teaches the correct skill in the correct way, provides lot of constructive feedback, can receive CPD updates in seconds, never gets ill and costs less than one month's teacher’s salary? That’s the long-term promise.

What makes all this possible are advances in AI, motion tracking and language recognition. We have already seen what Microsoft have done with Natal in terms of speech, gesture and motion recognition. Plonk this into a robot and we’re really getting somewhere. The point in the short term is not to replace teachers but to replace the teaching of SOME TEACHING TASKS.

The focus, for the moment, is on early years education, playing to the ‘cute’ factor. This makes sense. We could see significant advances in early numeracy, literacy and second language skills in schools with software that provides guidance superior to that of many teachers. In addition, they can be updated, wirelessly, in seconds and can even learn as they teach.

The basic premise is sound and was shown convincingly by Nass and Reeves in The Media Equation – we treat technology as humans if it has the right affective behaviours – good timing, being polite, co-operative etc.. This is how computer games work. With the right movement, sounds and behaviour, avatars and robots seem real. Tens of millions use and experience this phenomenon every day. There’s even a touch of this in your ATM, where you’d rather deal with a hole in a wall than a live teller.

Robots don’t get hangovers, don’t take holidays, never discriminate on grounds of gender, race or accent. They’re patient, scalable and consistent. The ideal teacher!

Robots & language learning

The initial trials have been in learning tasks in autism and English as a second language. S Korea sees English as a key skill in terms of growth and its Institute of Science and Technology has developed Engey, to teach English. Their goal is to have an effective robot, that is better than the average teacher, in 3-5 years. This is part of a general push in robotics that sees robots do things humans do in areas such as production, military, health and education. Hundreds of robots are already in S Korea's 8,400 kindergartens and the plan is to have one in every Kindergarten by 2013.

The University of California has been doing studies with a robot called RUBI, teaching Finnish. Initial studies show that the children do as well in tests as children taught by real teachers on specific language tasks. The retention was significantly better after 12 weeks with a reduction in errors of over 25%. Another interesting finding is that the robots need not look like real people, in fact hi-fidelity seems to be a little ‘creepy’ for kids. Although for an amazingly life like robot watch this.

CES 2010 featured a wonderful talking robot that follows you around the house and teaches you languages. It was remarkably sophisticated, with voice recognition, face recognition, picture recognition (show it a picture and it will say the word in your chosen language).l

Robots & autism

In a collaborative Japanese/US research project, children with autism have been shown to respond positively to synchronised behaviour from a robot. This is used to move the child on to other types of social interaction. In the University of Connecticut, a French robot is being used to with autistic children using mimicry to establish trust. Have a look at Beatbot's Keepon robot designed for kids with autism.

Robots and personalised learning

Personalised learning can also be realised through one on one interaction and the robot engaging in conversations and learning from the learner. Work of this kind has been going on at the Georgia Institute of Technology, with a robot called Simon. The improvements in AI and natural language processing have led to results in the robotic world that promise one to one tuition in the future.

Robots & physical tasks

There’s also the teaching of physical tasks, such as setting a table, where Honda Labs have taught older children to complete the task without the aid of teachers. Robots can already complete physical manufacturing tasks way beyond the physical capability, speed and accuracy of a human. We’ve had 25 years of robotic surgery, with robots being used to do surgery at a distance, unmanned surgery and to minimise invasion. In May 2006 the first AI doctor-conducted unassisted robotic surgery on a 34 year old male to correct heart arrhythmia. The results were rated as better than an above-average human surgeon. The machine had a database of 10,000 similar operations, and so, in the words of its designers, was "more than qualified to operate on any patient." The designers believe that robots can replace half of all surgeons within 15 years. In January 2009, the first all-robotic-assisted kidney transplant was performed at in the US by Dr. Stuart Geffner. The same team performed eight more fully robotic-assisted kidney transplants over the next six months.

Conclusion

It is only natural that robots, which have replaced highly skilled tasks in manufacturing, should be considered for teaching. Automating repetitive, difficult and dangerous tasks has always been technology's trump card. If we know one thing about teaching, it’s that it is difficult and demanding, leading to unnatural levels of stress and illness. If we can, at the very least, relieve the pressure on teachers, that is surely a noble aim. In its own way, simple robotic, screen programmes like BBC Bitesize and e-learning have already automated a lot of education and training. Robots promise to personalise this process. Every passing month sees improvements in movement, gesture and language recognition, with the technology appearing in the games world this year by Christmas. I have no doubt that robo-teaching will be common in schools in my lifetime.

OK Jimmy Wales – what's next after Wikipedia?

Wikipedia is a miracle and Jimmy Wales walks on digital water. So it was great to both see him speak, and get to speak to him, at Learning Without Frontiers this week. A truly 21^st century phenomenon (started 2001) and thorn in the side of those who think that knowledge is the domain of libraries and educational institutions, Wikipedia is BIG, with over 3.5 million articles in English, appearing in 262 languages (not all are fully populated).What’s more, it’s a fantastic legacy, as important as the publication of any book in history, as it has an astounding past (crowdsourced) and a fecund future, in terms of content, access, growth and impact. So what answers did Jimmy Wales have to the following questions?

How many?

Last month there were 408 million unique users. Think about that for a minute. Only China and India have more people.

Who’s looking at what?

Turns out different countries look at different things. The Japanese are obsessed with ‘pop’. Germans have ‘geography’ as their top topic (should we be worried?) and Spain ‘science and technology’. On the whole, however, pop, sex, history, geography, science and health are the big topics. Did you know that the LOST scriptwriting team had to use LOSTpedia to check when new references came up as the whole thing became too complicated to track?

Who creates the content?

Wikipedians are 87% male, average age 26, highly educated (almost all graduates) and the majority do not have a partner or children. Jimmy described them as “intelligent, obsessed guys with too much time on their hands”. Now some could see this as a bit of a problem, but hey, is it our fault guys? Get on there girls.

Wikipedia in China

When Jimmy was in China he was in a restaurant and Wikipedia appeared on a menu. This happened several times and people sent him menus from all over China with Wikipedia dishes on the menu. He guessed that people searched on Google for translations for dishes and since Wikipedia comes up often, it was carried over blindly onto the menus.

More seriously, China banned Wikipedia, but freed up the site around Olympics time, and now only block sensitive pages such as Taiwan and Tiananmen Square. It’s still the one country where whole classes of students have never heard the term ‘Wikipedia’. Everywhere else, the majority have not only heard of it but used it regularly.

Who hates Wikipdedia?

So what did Jimmy think about the ‘haters’, mostly academics? As he explained, they mostly don’t understand what Wikipedia is, in terms of construction, editing and discussion. Sure things are wrong, at times, but as he explained, on the whole, it’s pretty good, and as good as other traditional sources of printed knowledge. To those who say it’s too editorialised, his reply was that you can’t accept all contributions for entry and not have an editorial process. It can’t be completely open. On the whole Wikipedia is built by smart people who care.

What next?

I asked Jimmy whether he ever thought Wikipedia would create an education version, as teachers are not scalable and a step by step instructional adjunct with self-assessment tools would make it more relevant to education. He misunderstood the question a little and referred me to Wikibooks and explained that there’s too many national accreditation boards to consider. That didn’t stop him forging ahead with Wikipedia. Simply go round them. Ignore them. Let users and creators decide on content.

This is important, as Wikipedia broke the back of the encylopedia market, then broke the illusory monopoly that publishers and academics had on knowledge. But more than this, it showed that human beings are decent, altruistic beings who know a good and worthy thing when they see it, and are willing to help create things in education outside of the institutional structures.

I suspect that the next big educational resource will come from another source. Wikipedia is what it is, we need something similar but different.

Solution 1: Wiki textbooks

You can create textbooks through wikis, and use collaborative web-based creation and distribution for quality educational content. The problem here, seems to be the fondness for the ‘book’ metaphor i.e. Wikibooks etc. We don’t want books, we want web-based content.

CK12 is a possible breakthrough, as Jimmy Wales is on the board, and it’s well funded. You can use, edit and customise their textbooks. It’s pretty neat with good drag and drop creation tools, but again, it’s the ‘textbook’ metaphor that limits its usefulness.

Solution 2: Questions and answers wiki

Quora may be the sort of thing that will work. It’s created, organised and edited through crowdsourcing, but organises knowledge as answers to questions (which may in themselves be edited). This puts a more natural front-end onto a knowledge base, as queries are almost always framed as questions, not keywords. However, one question, one answer fails to create the dialogue and opportunities for structured learning.

Solution 3: Wiki Self-paced content

Take a structured, subject based resource, similar to BBC Bitesize, and allow it to grow and edit through crowdsourcing, with an editorial eye that knows good questions, good answers and there are opportunities to answer questions by the learner. Note that I’m not suggesting an expansion of Bitesize. That’s defaulted into little bits of animation add-ons.

Solution 3: Self assessment tools

This, I believe is the key to unlocking the open source knowledge market. Educational institutions have a stranglehold on education, making it incredibly expensive. That stranglehold is reinforced by the noose of accreditation. If we can free assessment from institutional control, we free up education for all. A populated open source Assessment tool that allows you to create, edit and use assessments would be a boon to learners and organisations.

Solution 4 All of above

Over the next ten years I see these fledgling wiki-led, open source movements produce resources in learning that are as powerful as Wikipedia. It needs a combination of good content and assessments. It also needs a credible open source brand, like Wikipedia. But Jimmy Wales, is not on this tack. To be fair he’s changed the world forever with Wikipedia, it would be a bit much to expect him to do it twice! If anyone is interested, contact me.

Top 10 stupid mistakes in design of Multiple Choice questions

Some online learning designers write their assessment items first, as they can match the assessment to the objectives or competences before being distracted by the detail in content. This avoids the trap of writing test items that simply test atomic facts and words from the presented text.

Many test items are quite simply not fit for purpose, as they don’t really assess, test the wrong thing or are so badly designed that they mislead or annoy learners. Bad test item writers tend to simply pluck out key words from the text and test the recognition or the meaning of those terms. Good test item writers, and this is a skill that few subject matter experts possess, understand good design and the need to test deeper understanding.

Here’s a few rules to follow if you’re new to this game:

1. Test understanding, not facts

What’s easy is to simply extract all the nouns, objects and quantities, then testing for recall. The trick is to push beyond this to test understanding. It’s not the ‘what’ but the ‘why’ that often matters but remains untested. What’s hard is to write questions that really do test the learner’s actual knowledge and ability to apply that knowledge. Professor Dylan Wiliam, calls great questions ‘hinge questions’, as they literally diagnose poor understanding. He explains how one can use these questions as powerful verbal test items in a classroom, where it is difficult to diagnose 30 kids quickly but the principle applies just as well to online learning. Here’s an example:

The ball sitting on a table is not moving. It’s not moving because:

A. No forces are pushing or pulling on the ball.

B. Gravity is pulling down, but the table is in the way.

C. The table pushes up with the same force that gravity pulls down.

D. Gravity is holding it on to the table.

E. There’s a force inside the ball keeping it from rolling off the table.

This question not only catches common misconceptions, it diagnoses between those who have understood the ‘physics’ and those who have not. There are forces at play, two in fact, so C is correct. The other options introduce common misconceptions about the absence of forces in stationary objects (A), single forces (B & D) or inner forces (E).

There are four components that make great questions:

Make them THINK

Give helpful FEEDBACK

Keep it CONVERSATIONAL

Be POSITIVE

A really bad question will allow the learner to guess the right answer or simply test recognition from a list. A good question will make the learner think, a great question will make them look away, even close their eyes to recall what they know, do something in their head and move towards the answer. It will push the learner.

Let’s try this:

A bat and ball cost £1.10.

The bat costs one pound more than the ball.

How much does the ball cost?

20p

15p

10p

5p       

This question really is a test of a learner’s grasp of algebra and ‘ratios’ in maths, it does a great job, as it really requires you to know how to apply a principle in mathematical thinking. The answer is not 10p as many choose. Think about it. If the ball was 10p, the bat, if it were a pound more, would be £1.10p, making a total of £1.20. The right answer is 5p. If the ball was 5p, the bat, if it were a pound more, would be £1.05p, making a total of £1.10. That’s a great question.

Remember also, that a great question also needs great feedback. In fact, it is better to call these, not questions, but ‘test items’ as you need, not only the question but the feedback. So let’s think about explanatory feedback for this question, that allows the learner to learn from the question.

A bat and ball cost £1.10.

The bat costs one pound more than the ball.

How much does the ball cost?

20p

15p

10p  

5p   
   

20p    ->  Try again.

15p    ->  Try again.   

10p    ->  Try again.

5p      ->  Correct.

This doesn’t really give any help to the learner who has jumped to the wrong conclusion or hasn’t thought deeply enough about the question. We need to provide HELPFUL feedback.

A bat and ball cost £1.10.

The bat costs one pound more than the ball.

How much does the ball cost?

20p

15p

10p  

5p      

20p    ->  20p+120p=140p  Try again.

15p    ->  15p+115p=130p  Try again.   

10p    ->  10p+110p=120p  Try again.

5p      ->  Correct. 5p+105p=110p

The feedback here tries to unpack the nature of the problem for the learner and gives an explanation as to why they are wrong.

Now let’s add another touch, that human element. Make it CONVERSATIONAL.  In conversation, you wouldn’t ask a question like this and simply feedback the statement ‘10p+110p=120p Try again.’ You’d make it a bit more conversational and friendly.

20p    ->  If  the ball was 20p, and the bat is £1 more at 120p. The ball (20p) plus bat (120p) gives a total of 140p. Try again.

15p    ->  If  the ball was 15p, and the bat is £1 more at 115p. The ball (15p) plus bat (115p) gives a total of 130p. Try again.

10p    ->  If  the ball was 10p, and the bat is £1 more at 110p, the ball (10p) plus bat (110p) gives a total of 120p. Try again.

5p    ->  Correct. If  the ball was 5p, and the bat is £1 more at 105p, the ball (5p) plus bat (105p) gives a total of 110p

Now be a little more POSITIVE:

20p    ->  Sorry. Most people get this wrong on first attempt. Let’s see why this is wrong. If the ball was 20p, and the bat is £1 more at 120p. The ball (20p) plus bat (120p) gives a total of 140p. Think of the ratio of the cost between the ball and bat within the 110p cost. Try again.

15p    ->  Sorry. Most people get this wrong on first attempt. If  the ball was 15p, and the bat is £1 more at 115p. The ball (15p) plus bat (115p) gives a total of 130p. Think of the ratio of the cost between the ball and bat within the 110p cost. Try again.

10p    ->  Most people get this wrong first time around and choose this answer. If  the ball was 10p, and the bat is £1 more at 110p, the ball (10p) plus bat (110p) gives a total of 120p. Think of the ratio of the cost between the ball and bat within the 110p cost. Try again.

5p    ->  Well done. If the ball was 5p, and the bat is £1 more at 105p, the ball (5p) plus bat (105p) gives a total of 110p.

You get the idea. Make test items meaningful, not mundane, and be helpful, fulsome and conversational in the feedback. This type of approach to test items will make the learner feel good and propel them forward, rather than act as barriers or make them feel like failures.

2. Binary Choice language

We don’t ask each other questions in normal language using the terms True and false, so consider more natural terms such as Yes/No, Agree/Disagree

Margaret Thatcher was the only British woman Prime Minister.

􏰁 Yes 􏰁 No      or       􏰁 Agree  􏰁 Disagree

3. Binary choice mistakes

Binary choice questions are often too long or turn into complex tests of logic by having two or more ideas expressed. 
Using NOT in a binary choice simply leads to a logic test, so best avoided. Finally, and I have seen this, asking the user to try again is stupid. 


4.  Edit out unnecessary words in options

How many earths would fit into the sun?

approximately 1000

approximately 10,000

approximately 100,000

approximately 1,000,000

This is a good question, a real test of your knowledge of the relative sizes of the earth and sun. However, take out the repeated word of each of the options, as there is no need to make the learner go to that extra effort in reading the same word five times.

Approximately how many earths would fit into the sun?

1000

10,000

100,000

1,000,000

And with meaningful feedback….

1000 ->  Sorry. It is easy to underestimate the size of the sun. The Sun has a radius 100 times that of the Earth. Try again.

10,000 ->  Sorry. It is easy to underestimate the size of the sun. The Sun has a radius 100 times that of the Earth. Try again.

100,000 ->  Sorry. It is easy to underestimate the size of the sun. The Sun has a radius 100 times that of the Earth. Try again.

1,000,000 ->  That’s right. In fact, you could fit 1.3 million Earths inside the Sun! It’s huge.

5. Make question grammatically agree with all answers

In golf, a one under par is a:

Eagle


Birdie


Bogie


Albatross

The word ‘a’ agrees with two of the options but not the other two. It’s grammatically wrong, as well as almost giving away the right answer!

What do you call one under par in golf?

Eagle

Birdie


Bogie


Albatross

6. Avoid negative questions

Avoid having NOT in the question, (Which of the following is not…..) unless they really are testing a competence than involves a common error or misconception. The ‘not’ can be easily missed by the learner, so capitalize if used. These questions often simply reinforce irrelevant negative information, rather than testing much that is useful.

7. Avoid making the longest option the answer

Go through the assessment items and click on just the longest option. You’d be surprised to see how often these are weighted towards the correct answer, as designers often write this first, then come up with easier, shorter distractors. Poundstone showed that this was common, even in high stakes tests. Options must be roughly the same length, of similar grammar and random in order of presentation.

8. Avoid ‘All of the above’ and ‘None of the above’

These are often imprecise tests and you don’t know what actually misled the learner if they get it wrong. They are also easy to write, so there is a tendency for poor test writers to have these as their ‘right’ answer, a fact shown by Poundstone’s analysis of thousands of test items.

9. All options should be believable.

Answers that look obviously wrong are pointless or even worse, patronizing and condescending. These are two questions from BBC Bitesize:

What is magma?

A chocolate ice cream

Molten rock

Bubbles of gas

What are the tiny air sacs in the lungs called?

Ravioli

Bronchioles

Alveoli

I rest my case. A lesser but similar error is to make one or more of the options too obviously right or wrong.

10. Test your test items

First test your test items with an expert, namely an experienced test-item writer. It is a skill that takes some time to master. Just because you know your subject, does not mean that you know how to assess and write good test items. Second, try the questions out with real target learners and ask them to ‘voice’ what they’re thinking. They invariable say things like ‘I don’t understand this question… What do you mean by…”. Don’t argue, just change it.

Conclusion

Questions encourage, not just the acquisition of learning but also critical thinking. Questions stimulate curiosity. Questions can intrigue and pull the learner towards exploring a subject in more detail. Good questions can be learning experiences in themselves not just test items. They can be used to stimulate thought but also, through good feedback, explain to the learner why they have a misconception, even act as a way of getting a point across. Questions drive learners forward. Good questions diagnose strengths and weaknesses. You don't know what you don't know and questions uncover the often uncomfortable truth that you know less than you thought you know. Finally, questions can assess and determine whether a learner is competent against certain criteria.

Questions really do matter in learning, that’s why it is important to design them so that they are fit for purpose. Professional examination bodies all too frequently write questions that are poorly designed and in some cases, unbelievably, impossible to answer. Experts in their subjects, often write poor test items, as it is difficult to put yourself in the shoes of a novice and assessment skills are different from subject matter expertise. In online learning, even professional vendors often produce badly designed test items, due to a lack of interactive design expertise. In many ways the test items are more important than the content.