Reasons to abandon multiple-choice questions

Long the staple of e-learning and a huge range of low and high stakes tests, the MCQ should be laid to rest or at least used sparingly. Their huge popularity has been due to several factors. First they are an artefact, a hangover from cardboard templates placed over squares to identify 'Xs". The tools for online learning also make them easy to write. This does not make it easy to write them well, in the same way that giving someone Word doesn't make them a novelist or poet.

But it has several flaws…

1. Probability

25% of getting it right on the standard four-option item, makes it less than taxing. True false, really a two option MCQ, is of course worse. I say 25% by research but Rodriguez (2005) shows that three options are optimal, then it goes to 33%.

2. Unreal

You rarely, if ever in the real world, have to choose things from short lists. This makes the test item somewhat odd and artificial, disassociated from reality.  They seem dissonant, as this is not the way our brains work (we don't cognitively select from four item lists in recall or automaticity). They are also weak on recall and therefore weak on the transfer of knowledge to the real world.

3. Distractors distract

It is too easy to remember the distractor, as opposed to the right answer. The fact that they are designed to distract makes them candidates for retention and so MCQs can become counterproductive. The research shows that four option MCQs often contain spurious and extraneous distractors that just add cognitive load and the chance of false recall.

4. Can be cheated

Pick longest item, second-guess the designer. Look for opposites and internal logic of distractor options. There are credible cheat-lists for multiple choice – Poundstone’s research shows that these approaches increase your chance of getting better scores. (20 cheats here)

5. Surface misleading

Take these two questions.

What is the Capital of Lithuania? Tallin, Vilnius, Riga. Minsk

What is the Capital of Lithuania? Berlin, Vilnius, Warsaw, Helsinki

Surface differences in options make these very different test items. And it is easy to introduce these surface differences, reducing the validity of the test items and test. MCQs often have huge variances depending on the options presented.

6. Difficult to write

I have written a ton of MCQs over 35 years – believe me they are seriously difficult to write. It is easy to select the noun from the text and come up with three other nouns. What is difficult it to test, is real understanding.

7. Little effort

This is the big one. As Roediger and McDaniel state in their book Make It Stick, choosing from a list requires little cognitive effort. You choose from a limited set of options, and do not use effortful recall (which is in itself increases retention).

Conclusion
Multiple-choice is not a terrible test item but it has had its day as the primary test item in online learning. We’re still designing test items in lock-step because the tools encourage us to do so, ignoring morepowerful open-response questions that require recall and the powerful act of writing/typing, which in itself, reinforces learning. New tools, such as WildFire, the AI-driven content creation service, focuses on open-response and effortful learning, for all of these seven reasons and more. The learner has to make more effort as that means deeper processing and higher retention.

Taming the AI & Assessment Conflict: 21 Practical Resolutions

There is a battle underway, with assessors in education and institutions on one side and Generative AI on the other. It often descends into a cat and mouse game but there are too many mice, and the mice are winning. Worse still is when it descends into a tug-of war, tech showdown or brutal legal collision.

We clearly need to harmonise assessment with our evolving technology and navigate our way through AI and assessment to avoid friction, with some practical solutions to minimise disruption. Assessment is at a crossroads.

Legal confrontations between assessors and the assessed will do great damage to institutions. When Sandra Borch, Norway’s Minister of research and higher education cracked down on plagiarism, taking a student to the Supreme Court, another student uncovered serious plagiarism on her Master’s theses – she had to resign. A more distasteful case was Claudine Gay, President of Harvard, who had to resign after an attack by a right-wing politician, Bill Ackerman. The whole thing turned toxic as people uncovered plagiarism in the work of his academic wife. As many US kids have lawyers as parents, a slew of cases are hitting the courts putting the reputations of schools and colleges at risk. This is not the way forward.

Problem

Assessment stands right in the middle of the earthquake zone as the two tectonic plates of AI and traditional education collide and rub up against each other. This always happens with new technology; printing, photocopying, calculators, internet, smartphones… now AI.

We are currently in the position where the mass use of AI is common, because it is a fabulous tool, but is being used on the SLY. There is widespread use of unsanctioned AI tools to save time. Learners and employees are using it in their hundreds of millions, yet educational institutions and organisations are holding out by ignoring the issue, or doing little more than issuing a policy document. AI is therefore seeping into organisations like water, a rising tide that never ebbs.

The problem has just gotten way more challenging as AI agents are here (automation). The Claude Sonnet 3.5 upgrade has just gone 'agentic', in a very practical way. It can use computers the way people do. It can look at your screen, go find stuff, analyse stuff, complete a chain of tasks, move your cursor, click on buttons and type text. To be clear, it understands and interacts with your computer just as you are doing now. This shift is in ability means it can to do open-ended functions like; sit tests, do assignments, even open research.

In truth…

We need to be more honest about these problems and stop shouting down from moral high horses, such as ‘academic integrity’. Human nature determines that if people find they are being played, or have an imperative that is different, they will take paths of least resistance. E.O. Wilson’s observation that we have “ Palaeolithic minds, Medieval institutions & Godlike technology” leads people to take shortcuts, through fear of failure, financial consequences, even panic. It is pointless to continually say 'They shall not pass' if the system has below par teaching and assessment is poorly constructed. We cannot simply blame students for the systemic failures of a system. Education must surely be a domian where teachers and learners are in some sort of harmony.

It doesn't help that we delude ourselves about the past. Cheating has been and still is common. We’re kidding ourselves if we think parents don’t do stuff for their kids at school and in universities. Essay mills and individuals writing assessments, even Master’s theses, exist in their tens of thousands in Nairobi and other places, feeding European and US middle-class kids with essays. Silk cloths with full essays go back to Confucian times. Technology can be bougt on the internet with button cameras and full comms into invisible earpieces. Cheating is everywhere. That's not to condone it, just to recognise tat it is ALWAYS a problem, nout just an AI problem.

In truth, we also have to be honest as accept that assessment is far too ‘text’ based. Much of it does not assess real skills or performance – even critical thinking. Writing an essay in an exam does not test critical thinking. No one writes critically starting top left, finishing bottom right – that’s why students memorise essays and regurgitate them in exams. Essay setting is easy, actual assessment is hard. We also have to be honest and accept that most educators designing and delivering assessment know little about it.

In the workplace, few take assessment seriously. At best it is multiple choice or e-learning thinly peppered with MCQs. L&D doesn’t take assessment seriously because they are not driven by credentials, nor do make much effort to evaluate training. With MCQs, you can guess, (1 in 4), distractors are often poor or simply distract, are difficult to write, easy to design badly, often too factual or unreal, require little cognitive effort and can be cheated, (longest, opposites etc.). An additional problem is that online authoring tools lock us into MCQs.

Assessments are odd. People settle for 70-80% (often an arbitrary threshold) as tests are seen as an end-point. They should have pedagogic import and give the learners momentum, yet there is nothing meaningful on improvement in most assessment and marking. Even with high scorers, full competence rarely the aim as a high mark is seen as enough, not full competence. The aim is to pass the test not master the subject. 

Plagiarism checkers do not work, so DO NOT use detectors. Neither should you depend on your gut – that is just as bad, if not worse. There are too many false positives and they consistently accuse non-native speakers of cheating. Students KNOW this tech better than you. They will always be one step ahead and even if they are not, there will be places and tools they can use to get round you.

Neither does setting traps for the mice, like "Include in your work icitations from <fictional name>", as when employed, once the trap is revealed, the learners can use the tech to reveal the fictional trap.

In a study by Scarfe et al., GenAI submissions were seeded into the exam system for five undergraduate modules across all years of BSc Psychology in a UK University. 94% of the AI submissions went undetected, with the AI submissions getting grades a half grade boundary higher than real students.

In a recent survey of Harvard students, some had made different course choice because of AI, other reported a sense of purposelessness in their education. They were struck by the fact that they were often learning what will be done differently in a world of AI. It is not just assessment that needs to change but also what is taught to be assessed. Assessment is a means to an end, assessing what is known, learnt or taught. If what we need to know, learn to teach changes, so should assessment.

Solutions

Calculators generate numbers, GenAI generates text. We now live in a post-generative AI world where this is the norm. Most writing is also now digital, so why are so many exams written? 

Most writing in the workplace is not a postmodern critique of Macbeth but fairly brief, bureaucratic and banal, getting things done by email, comms, plans, docs and reports. Management is currently laden with admin and GenAI promises to free us from admin to focus on what matters - the goal. It is here to stay because there is a massive need in the real world to raise productivity on speed and quality and not nget bogged down on redrafting or pretending you are Proust.  Why expect everyone to be writers of brilliant prose, when the goal is to simply get things done.

1. We have to move beyond text-only assessment into more performance-based assessments. Kids go to school at 5 and come out up to 20 years later having did little else other than read, write and comment on text. There is this illusion that one can assess skills through text – which is plainly ridiculous. Accept that people use GenAI to improve their writing beyond their own threshold. Encourage them to use AI to help them make their writing morce concise through summarisations. Allow them to critique their own work through AI.

2. Build 'pedagogy' into creating assessments with AI. We have done this by taking the research, on say transfer and action, then building that into the AI assessment creation process. You get better, more relevant assessment items, along with relevant rubrics.

3. Also build good assessment design practices into creating assessments with AI. There are clear DOs and DON’Ts in the design of assessment items. Build these into the AI creation process. Go further and match assessments to quality assessments standards. Believe me, this can be done in AI.

4. Match assessments more closely to what was actually taught. This alignment can be done using AI, including the identification of gaps, representative coverage, weaknesses on emphasis identified. The documents and transcripts used in teaching and/or the curriculum, can be used by AI to create better quality assessments.

5. Do more pre-assessments. David Asubel said “The most important single factor influencing learning is what the learner already knows.” I totally agree, yet it is rarely done. This gives assessment real pedagogic import – it propels or feeds forward into the learning process and helps teachers. These can be created quickly by AI.

6. Let's have more retrieval practice. This can be created quickly by teachers, even learners themsleves. We know that this works better than underlining notes and highlighting. Making leatrners or learners themselves making the effort to recall ideas and solutions intheir own minds helps get stuff into long-term memory.

7. Move away from MCQs towards short open text, assessed by AI. Open text is intrinsically superior as it demands recall from memory rather than identification and discrimination (correct answer in MCQs is there on the paper or screen). Open response more accurately reflects actual knowledge and skills.

8. Move away from authoring tools that lock you into fixed, templated MCQ assessment items. They also template things into rather annoying cartoon content with speech bubbles etc. Here's 25 ways I think bad design makes content and assessments suck.

9. Use more scenario and simulation assessment. They match real world skills, have more relevance and can set more sophisticated assessments on decision making and other skills. AI can create such scenario and sims assessment and the content you need to populate the scenarios.

10. On formative assessment, 'test out' more. Testing out means allowing people to progress if they pass the text. they may be able to skip modules, even the entire course. This should be the norm in compliance training, or training full stop, where people get the same courses year after year.

11. Get AI to create mnemonics and question-based flashcards for learners to self-assess, practice and revise and create personalised spaced-practice assessment, so they can get learning embedded. 

12. On formative assessment, use more AI designed and delivered adaptive and personalised learning. Adaptive learning can be 1) PRE-COURSE: Student data or pre-tests define pathways. (Never use learning styles or preferences). 2) IN-COURSE: Continuous adaption which needs specialised AI software (this is difficult but systems can do it). 3) POST-COURSE: With shared data across courses/programmes, also Adaptive assessment, Adaptive retention such a personalised spaced practice and performance support.

13. AI created Avatars can be built into assessments. These can be customers in sales or customer care training; employees in leadership. management and specific skills such as recruitment interviewing; or patients in medical education, where you can interact with people in assessments to provide realism.

14. Automate marking. Most lecturers, teachers and trainers have heavy workloads, many rightly complain about this, so focus on teaching not marking. Automated marking will also give you insights into individual performance and gaps. The Henkel study (2024), in a series of experiments in different domains at Grade levels 5-16 (Key Stage 2/3/4), showed that AI was as good at marking as humans.

15. Use audio to deliver assessment results, along with feedback and encouragement. This is more personal and motivating for the learner. It also forces the assessor to be more articulate and precise.

16. Use AI post-assignment or post-assessment techniques, such as generated audio questions that interrogate the learners understanding of their own work. Their audio answers could be transcribed by AI, even assessed by AI. 

17. Situates GenAi chatbots alongside the student assignment draft. Teachers have visibility of student-GenAi chatbot conversations when the work is submitted. Teacher prompts, mark schemes and other contextual information inform GenAi chatbot responses.

18. Make assessments more accessible on language and content. Far too many assessments have overly academic language or assessment items that are to abstract and can be turned into better expressed and relevant prose and problems. Translate overly-academic language to readable and vocational using AI. Critique and translate into more readable prose.

19. AI has revolutionised accessibility thorough text-to-speech and speech to text. It has now provided text and speech chatbots and automatically created podcasts (free NotebookLM) AI has also given us live captioning and real-time transcription. For dyslexics (5–15% of population), T2S & S2T, spell checks, Grammar Assistants, Predictive Text and voice dictation have been incredibly useful in reducing fear and embarrassment. AI can do wonders in making assessment more accessible.

20. Use AI for data analysis on your assessment data. It is as simple as loading up a spreadsheet and asking questions you want answered.

21. Stop being so utopian. Most people at school and University will not become researchers and academics. Don’t assess them as if that is ‘their’ goal.

22. What are the skills left over for education to focus on after you get GenAI into common use both in education, the workplace and life? The pat answer is too often - soft skills. I disagree. Move deeper into hard expertise and skills, with a broad perspective, that can be enhanced, magnified, even executed and automated by AI.

Radical thought

AI could disrupt the current system of higher education, which often prioritises signaling over actual learning, as argued in Bryan Caplan's The Case Against Education. Caplan suggests that many students pursue degrees not to gain knowledge or skills (human capital) but to signal qualities like conscientiousness and dedication to employers.

If building human capital were the primary goal, cheating would make little sense—it undermines the purpose of learning. However, in a signal-focused system, cheating becomes a way to appear diligent and capable without actually possessing those traits. AI, by making mass cheating more accessible, could devalue degrees, particularly those in softer disciplines, by stripping away their signaling power. Without this value, students might reconsider the utility of pursuing such education and shift toward activities that genuinely enhance their knowledge and skills.

Universities will resist this shift. As institutions that benefit from the current model, they are likely to find ways to preserve the signaling value of degrees rather than embrace a reset. Despite this, students should reflect on how to maximise the real value of their education. Four years dedicated to study is a rare chance to gain lasting knowledge and skills. By focusing on genuine learning instead of merely signaling, students can gain a meaningful advantage over peers who treat education as a hollow credential.

Conclusion

AI is moving steadily away from prompting to automation. This is happening in writing, coding spreadsheet analysis, image creation even moving image creation. It is happening with avatars and in real-time advanced dialogue as speech.

Just like calculators generated numbers GenAI generates text. We need to recognise this and change what we expect of learners. Turning it into a cat and mouse game will not work, there are too many mice and they're winning.

PS
This is a sort of summary of my talk in Berlin at the ATP European Assessment Conference.

 

Why almost all Multiple Choice Questions are badly designed...

Multiple-choice questions are everywhere, from simple retrieval tests to high stakes exams. They normally (not always) contain FOUR options; one right answer and three distractors. But who decided this? Is it more convention than researched practice?

Research

Rodriquez (2005) studied 27 research papers on multiple-choice questions and found that the optimal number of options is THREE not four. Vyas (2008) showed the same was true in medical education. But God is in the detail and their findings surfaced some interesting phenomena.

Four/five options increases the effort the learner has to make but this does not increase the reliability of the test

Four increases the effort needed to write questions and as distractors are the most difficult part of the test item to identify and write, they were often very weak.

Reduction in the number of options from four to three (surprisingly) increased the reliability of test scores.

Tests were shorter leaving more time for teaching.

Cheating MCQs
In my 20 tips on how to increase your score in multiple choice questions, I mention the fact that there's often one distractor that's obviously wrong, another reason for abandoning that often difficult to write fourth distractor.

Weaknesses in MCQs
Of course, multiple-choice is, in itself, weaker than open-input, which is why we can go one step further and have open response, either single words or short answers. Natural Language Processing allows AI not only to create such questions automatically but also provide accurate student scores. AI can also create both of these question types (also MCQs if desired) automatically, saving organisations time and money. This is surely the way forward in online learning. Beyond this is voiced input, again a step forward, and AI also allows this type of input in online learning. If you are interested in learning more about this, see WildFire.

Conclusion

So, you can safely reduce the number of MCQ options from five/four to THREE and not reduce the reliability of the tests. Indeed, there is evidence that it improves the reliability of tests. Not only that it saves organisations, teachers and learners time.

Bibliography

Rodriquez (2005) “Three Options Are Optimal for Multiple-Choice Items: A Meta-Analysis of 80 Years of Research”, Educational Measurement: Issues and Practice, Volume 24, Number 2, June 2005, pp. 3-13.

Vyas R (2008) Medical Education. The National medical Journal of India, Volume 21, No 3

Why almost everything we think about online learning may be wrong and what to do about it…

One thing that research in cognitive psychology has gifted to us over the last decade or so, is clear evidence that learners are delusional when it comes to judgements about their own learning. The big name in the field is Bjork, along with many other high quality researchers, who say that learning is “quite misunderstood (by learners)…. we have a flawed model of how we learn and remember”. There’s often a negative correlation between people’s judgements of their learning, what they think they have learnt and how they think they learn best - and what they’ve actually learnt and the way they can actually optimise their learning. In short, our own perceptions of learning are seriously wrong. This is why engagement, fun, learner surveys and happy sheets are such bad measures of what is actually learnt and the enemy of optimal learning strategies.

Desirable difficulty

Most learning is illusory, as it is too easy. Learning requires Desirable (accomplishable) and difficult learning that requires real effort for high-retention to take place. This is why so much online learning fails. To simply click on faces and see speech bubbles of text, drag and drop labels, choose true or false, even multiple-choice questions, rarely constitutes desirable difficulty. This is click-through learning.

The solution is to provide effortful, retrieval. This means moving beyond the traditional model of text/graphics punctuated by multiple-choice, towards cognitive effort, namely retrieval through open input. This effortful learning gives significant increases in long-term retention and recall. Online learning needs to adopt these techniques if it is to remain credible.

Retrieval 

Retrieve and recall what you need to know Bjork (1975) results in much higher levels of retention. Rather than read, re-read and underline, look away and try to retrieve and recall what you need to know. Rather than click on True or False or an option in a short list (MCQ), look away, think, generate, recall and come up with the answer. The key point here is that research has shown that retrieval is a memory modifier and makes your memory more recallable. Counter-intuitively, retrieval is much more powerful than being presented with the information. In other words it is more powerful than the original ‘teaching’ event.

Take a learning experience that you have probably been through many, many times – the airline safety demonstration. Try to think through what you have to do in the right order – find life jacket, put over head, then what….. not easy is it? Ah yes… inflate it through the blow tube… then there’s the whistle. No. Many choose the ‘inflate’ option but to inflate it inside the aircraft is a BIG no, no... and, in fact, you pull a toggle to inflate. In fact, airlines should set up a spot in the airport, where you actually sit down then have to DO the whole thing. Next time you sit there, watch, then afterwards, close your eyes and retrieve the process, step by step – that also works.

Roediger and Karpicke (2006) researched studying v retrieval testing (without feedback). One week later the retrieval tested group did much better. They also asked them how much you are likely to remember in one week’s time for each method – oddly, the majority of learners got it completely wrong.

Making errors is also a critical component of successful learning. According to Kornell, Hayes and Bjork (2009), generating the wrong thing, then getting it right, leads to stronger learning. The reason is that you are activating the brain’s semantic network. Retrieval testing does better than reading or watching, as it potentiates recall.
So are unsuccessful tests better than presentations? The work by Kornell (2009) shows that even unsuccessful testing is better. Retrieval testing gives you better internal feedback and works even when you get few or no correct answers. Testing, even before you have access to the material, as a learning experience, also helps learning. Once again, almost bizarrely, Heustler and Metcalfe (2012) asked learners what worked best and they were largely wrong.

From Gates (1917) who compared reading and re-reading with retrieval, to Sptzer (1939) who halted forgetting over 2 months with retrieval in 3000 learners, to Roediger (2011) who got a full grade increases with retrieval techniques and McDaniel (2011) who increased attainment in science, the evidence is clear. For a clear summary of this, and detail on the research, this excellent talk by Bjork is pretty good.

Online learning

In online learning the mechanics of this have also been researched. Duchaster and Nungester (1982) showed that although MCQs help you answer MCQs, they are poor in actual retention and recall. Kang (2007) showed that retrieval is superior to MCQs. At the really practical level, Jacoby (1978) showed that typing in retrieved learning was superior, as did MacDaniel (1986) and Hirsham and Bjork (1988) who showed that even typing in some missing letters sufficed. Richland (2005) did real world experiments that also proved efficacy.
We have the tools in Natural Language Processing and AI to do this, so technology has at last caught up with pedagogy. Let's not plough the same furrow we've plowed for the last 35 years. Time to move on.

Conclusion

I wrote, in a rather tongue in cheek manner (25 ways you which your e-learning sucks), about why I think most current e-learning is click-through and therefore low retention eye candy. This research shows that our methods of online learning are sub-optimal. The problem we face is that immediate success often means long-term failure. More focus should be given to retrieval, NOT presentation or clicking on items and multiple-choice. We need to be presented with desirable difficulties, through partial or complete open input. This is exactly what we’ve spent the last two years building with WildFire.

AI-driven speech may revolutionise online learning

Over the last year or so we built the world’s first AI content creation service, WildFire, which creates online learning in minutes not months, at a much lower cost and with high retention. The reason for claiming it is high retention, is that we largely abandoned multiple-choice-questions for open-input, making the learner think, recall and actively input their thoughts. It is this ‘effortful’ learning that really matters in learning. This worked well and we have delivered online learning on factual knowledge, high-end academic content, processes, procedures and management content to a range of audiences in large organisations, from apprentices to high-end clinicians, in finance, healthcare, travel and manufacturing. Having seen how well open-input worked, we turned our attention to the use of AI to go several steps further and improve the interface. What if the learner could simply speak the answers? ... it was a revelation.

Speech

We learn to speak almost effortlessly, whereas, writing takes many years. So why not exploit what we do everyday in our lives - use speech input. It was thought that women spoke much more than men, a myth started in The Female Brain, by Louann Brizendine, who claimed that, whereas women spoke on average 20,000 words a day, it was only 6,000 for men. This proved to be nonsense. An actual study, at the University of Arizona by Mehl (2007), using an electronic recording device to sample everyday speech from 396 people found that we speak, on average, around 16,000 words a day, with no significant difference between men and women.

This is much greater than the average for writing. So it makes sense to use speech in learning. Consider also that if spelling is not part of the learning, you eliminate problems around misspelling, especially for those who are nervous on that score or who may have dyslexia. On top of this you do not have to make the physical effort to move a cursor around the screen into a field, then physically type.

Retention

However, it is interesting to compare different forms of input in terms of retention. So, you think of an answer, then:

   CHOOSE your answer from a list (multiple choice)

   TYPE your answer

   SPEAK your answer

CHOOSE (MCQs)

It is clear that simply clicking on an already provided answer from a list is the least effective of the three. The answer is there in front of your eyes, you have a 25% chance of getting it right without knowing anything, questions are often designed so that you can guess and the distractors are often remembered rather then the correct answers. It makes you wonder why the online learning industry is so wedded to MCQs.

TYPE

This has the advantage of making you recall the answer into your brain first (a powerful reinforcement event), then actively type in the answer, another reinforcement event, without having been given the answer or suffering from the drawbacks of simply choosing from a pre-written list. We have found this to be a much more powerful way of learning.

SPEAK

Things get interesting here, as you are communicating directly, without any of the artificiality of choosing from a list or typing. My initial impression (not based on any studies) is that this may be even better. Being hands free, your attention and cognitive focus is entirely on thinking and expressing your thoughts. None of your cognitive bandwidth is taken up by moving the cursor, typing and letter-by-letter spelling. You get a focus on meaning but there’s an additional advantage, as you get more of a flow and the learning is faster.

Podcasts

Using another form of AI, text to speech, we can also, automatically, create podcasts. This is built into the service. Simply tick a box and your online learning will create a podcast of the module or page by page speech. This is a useful supplement to the active learning.

Context

In addition, as we have audio only learning, including navigation, using the words, NEXT, BACK, GO and SCROLL, so we can place the learning experience within VR, which we have done, instantly and cheaply. We know that context helps retention, so speech input allows a further level of retention to be achieved. This is getting interesting in say, training fror healthcare professionals in a hospital or cabin crew inside an aircraft.

Conclusion

Simultaneously, using different forms of AI, we hope to have increased the efficacy of online learning by the:

1. Superfast creation of content

2. Higher retention open-input

3. Higher retention speech input

4. Automatically created podcasts

5. Full 3D VR delivery

All at lower costs and far greater speed than traditional and expensive methods. If you are interested we can show you all of this by Skype. Contact us here.

Bibliography

Mehl M (2007). Are Women Really More Talkative Than Men? Science ,Vol. 317, Issue 5834, pp. 82

AI revolutionises Higher Education in China Open. Huge project gets UNESCO Prize

After writing a book ‘AI for Learning’, I have given a lot of talks and podcasts on the need to use smart software to make people smarter. That means using the technology of the age – AI and data. In the book (p228), and in many of these talks, I explained how China is forging ahead with AI in this field. Unlike the West, China has focus, investment and a view that access to education needs to be cheaper, faster, smarter and with a massive increases in access. 

Meanwhile, we continue with a view that Higher Education needs to remain scarce and expensive, very expensive. We put more attention into AI and Ethics than real projects. Our Universities and colleges do little more than write reports on AI for learning. That’s a shame.

Meanwhile the Open University of China has been awarded a UNESCO Prize for its use of AI to empower rural learners. Their ‘One College Student Per Village’ is an ambitious and inspiring initiative that puts equitable access at the heart of their offer. This is all about improving access to education for the poor. Running since 2004, financed by the Chinese Ministry of Education to tackle access problems, it does far more than reach out with infrastructure. AI lies at the heart of their efforts to provide scalability.

Scalability

In its efforts to provide quality learning experiences, the OUC set up over 500 cloud-based classrooms and smart classrooms in poorer areas in 31 provinces, municipalities and autonomous regions. The trick was to make the courses demand-led by asking what local people wanted and providing largely practical, vocational courses. They also built the courses to be accessible on mobile devices for farmers and those in rural professions and places. The numbers are impressive:

·      29 programmes (using AI) 

·      825,827 learners enrolled

·      529,321 graduated

·      1,500 OUC study centres

·      300 online courses

·      100,000 mini-lectures

·      all open to general public

AI for Learning

But the secret, sweet without the sour sauce, is AI. The learning is personalised using personal and aggregated data. This adaptive learning means that different students take different paths through the courses, a bit like the SatNav or GPS in your car, go off course, and it re-sequences content and provides feedback to get you back on course. I’ve been working with this for five years – believe me it works.

The really clever stuff is the use of AI to recognise text (text to speech), as well as semantic analysis of answers. This allows open input from students, as opposed to MCQs. I’ve also been working with this for some time in WildFire. This approach allows learners to answer or ask questions which are automatically recognised through semantic analysis, then feedback automatically provided by the system. This feedback (should really be called feedforward) is what oils the wheels of learning and provides real scalability. The immediate feedback with learning opportunities means that the system does not depend as much on human tutors. Semantic analysis of learner answers is something we’ve implemented. It is powerful and pedagogically superior to MCQs as it is more realistic, requires greater cognitive effort and can be more diagnostic for the purpose of feedback.

They also use AI for knowledge mapping, automatic content generation and smart chatbots for 24/7 online learner support. This use of AI for content creation is something we’ve been doing at WildFire. It reduces the cost of learning per student, as new content can be created quickly from documents, PowerPoints and videos. The point is to create content quickly, cheaply but using AI for semantic analysis and retrieval practice for high retention.

AI and assessment

The automation of assessment and essay marking is what allows them to scale sophisticated learning to so many people free from the tyranny of time, place and expensive human effort. Automating much of the assessment allows human tutors to focus on closing knowledge and skills gaps, rather than marking. As Li Ganged, a tutor at the OUC says, “Automated essay scoring is efficient in that I don’t have to mark these assignments myself but I can get a clear picture of where learners need help.” This idea of using AI to create assessments on all of your content at little extra cost is also something we've been doing using AI.

Conclusion

Another feature of this initiative, something our rigid system can’t really handle is the mix of programmes, degrees, diplomas and short courses. The focus on vocational training is also something we desperately need but underfund in favour of longer degree courses. We have to move beyond our sclerotic system of high cost content creation, high cost delivery and dependence on physical campuses. There’s too much at stake here for us to focus on education of the few at the expense of the many. Who would have thought that China is leading the way. They’ve just clocked up over 18% of economic growth but it’s not just about that, it’s the simple fact that they are leading the world in educational innovation.

Is SaaS in learning dead?

Our LMS seem antiquated, a bare learning landscape of frustration, overloaded with outdated content a stockpile of cartoonish modules and tedious slideshows dotted with MCQs.

The market seems weary of simply renewing licences for software that has low use figures, often just slabbing out compliance content with no actual measurement of its productivity or impact. Measuring bums, even on virtual seats is to measure the wrong end of the learner. 

These systems get locked in as a matter of habit not impact. But dreams don’t last forever and pricing on LMS software, content libraries and other learning services has got fiercely competitive. Many are seeing revenues plateau or fall. Management is being swapped out, share-prices falling, revenues no longer predictable, profits still showing overhang on 3 year licence fees. It has all become fiercely competitive.

AI is disruptor

Having been involved in a lot of AI projects for over a decade, since early 2014, the assumption has always been that people need to build some sort of AI SaaS product, sit back and watch the money pour in. A recent Tweet questioning this, by the much admired John Rush, went viral on this point.

Yet AI is already bringing services directly to your users, be it learners or employees in any sector at any level. With ChatGPTo1 reason has now appeared. AI now gives massive agancy to the individual separate from your corporate systems. Like a supercar at saloon prices it has overtaken many services on the fast lane. SaaS services get locked into fixed depreciating code. These inflexible code sets and fixed products make progress impossible, whereas AI seems to release new astounding updates on a weekly basis, so quick, it is hard to keep up.

AI is not really a business change, it is a change in mindset. You have to think of of power being plugged directoy into actual learners and employees. It releases agancy and motivation. Once you realise that everyone is using it, you need to see that its direct impact bypassing your existing systems. 

AI on the SLY

The studies on productivity and impact are clear, that’s why most are using AI on the SLY. Individuals are increasing their productivity, making their lives easier, while organisations, locked into legacy systems are trying to ignore it or fighting it wit negativity. That’s because it needs a shift in mindset, to accept the probability game, understanding that everything has flaws, everyone is biased, especially people and processes. Sure AI may occasionally come up with the wrong answer but so do people and at least AI is fast! In fact, for most education, learning and business uses, hallucination is not a problem at all, as it has been pushed away out to the periphery. The appearance of reasoned ‘thinking’ has meant a whole load of new educational and organisational uses have been folded in to the omelette. And you don’t make an omelette without breaking eggs.

Productivity

Traditional Saas product in learning may be struggling but it also means that straight to user services in real time are alive and kicking ass.

One group of companies that are doing well are those that use AI to simply increase internal productivity. This gives them edge and the financial room to increase productivity or respond to pricing pressure. In one case we’ve seen astounding increases in productivity on process, both massive savings in time but also increases in quality. This gives you time to focus on customers and quality. How do you make your learning company more valuable? Get lean. Get agile. AI allows you build software fast. AI allows you to fast-track productivity increases. AI gives you quality increases fast. AI unleashes life in an organisation.

Performance support

Performance support, which is what 99% of AI is, is now in the hands of users. There’s tons of rooms for good delivery here, as organisations need to deliver performance support based on their own motherlode of content. In steps AI with RAG and other techniques to do precisely that - superbly well. Suddenly, you’re not relying on generic content, that everyone’s using, giving your organisation no edge at all. You can focus and lever what you already own. 

Conclusion

We’re still locked into the lumbering LMS, often a dark dungeon of disappointment, also over-engineered content, often out of date and hopelessly cliched and generic, with its cartoon libraries and slideshows peppered with MCQs. There is a sense that the old era is dying in the vine, coming to an end.

 

AI builds e-learning at 10% of cost, in minutes not months, with higher retention and recall

Paying 20k an hour for e-learning content, over many momnths, that is laden with noisy pages of text/graphics, punctuated by low retention multiple-choice questions? AI can help you to build content at 10% of the cost, in minutes not months, with higher retention and recall.

Problem

The problem with the traditional online learning bespoke content paradigm is the tools. They push vendors and buyers into producing content that has ten major flaws:

1. Expensive to produce

2. Too long to produce

3. Difficulties with SMEs

4. Media (but not mind) rich

5. Weak Multiple-Choice

6. Low effort learning

7. Pavlovian gamification

8. Impersonal learning

9. Low retention and recall

10. No practice

1. Expensive to produce

I ran a large, bespoke, content company. It was very successful but that was back in the day and it used the tools of the day. Yet the content produced today costs and looks much as it was 30 years ago, despite the fact that computers are faster, better and cheaper and online. Why does it still cost 20k an hour to produce e-learning content? Because we’re still in the old paradigm of traditional authoring tools and a mindset besotted with appearance, not learning. Imagine reducing that cost by 90% through a radically different approach, using AI and automation. You can.

2. Takes too long to produce

E-learning projects can take up to six months or longer, with lots of process and angst. It’s a highly iterative process, and takes a huge amount of management, definition, design, development and delivery time to produce anything. Imagine doing all of this in minutes, not months. You can.

3. Difficulties with SMEs

By far the most difficult step in the production of online learning is getting the knowledge and expertise from the mind of the subject matter expert (SME) across and into the course. It’s a tricky process and often full of angst and recrimination. Imagine taking SME content – any document, PowerPoint or video and turning it automatically into online learning. You can.

4. Media (but not mind) rich

As computers have allowed us to deliver media rich experiences, we have, often blindly, ignored the research by Mayer, Clark and many others, showing that media-rich is not always mind-rich. This has resulted in an often garish concoction of movement, graphics, cartoons and animation, that inhibits, rather than enhances learning. There’s maybe not enough ‘edu’ and too much ‘tainment’ in ‘edu-tainment’. We need to pay attention to the research, reduce cognitive overload and focus on the learning. Less is more.

5. Weak Multiple Choice Questions

Long the staple of online learning, yet when in real life does anyone choose an option from a list, as if the mind was a simplistic ATM, choosing from menus? Besotted with MCQs, we have produced low-effort, low retention and low recall learning. It doesn’t have to be this way. Use more effortful, open response learning. The research shows it is superior.

6. Low effort learning

The illusion of mastery is what much online learning produces, the feeling that you’ve learnt things through light-touch exposure and occasional selections from lists. In truth, real learning is learning by doing, real effort, not page turning and exposure to fancy media. Recent reseach turns traditional onine learning on its head. Make the learner make the effort – that’s what results in high retention and recall.

7. Trivial Pavlovian gamification

Does gamification play Pavlov with learners? You can focus on the sort of gamification that Demis Hassabis uses in AI learning – repeated deliberate practice. Don’t make it too easy to sail through, allow learners to fail, make them work, make them do things until they get 100% competence…. That’s true gamification.

8. Impersonal

Good online learning is always a balance between directed and open-learning. There needs to be structure, often quite directed, but there also has to be the opportunity for support, expansion and curiosity. Allow the learner to explore by providing automatically generated links out to extra content. Make your course porous. This is possible with AI.

9. Low retention and recall

Over the last ten years, evidence has emerged (summarized in Make It Stick) that effortful learning matters, that open response is superior to multiple choice and that deliberate and practice matters. We have the ability to use AI to embody and deliver practice based on contemporary learning theory.

10. No practice

The once only, sheep-dip experience was the target of online learning, with its anytime, anywhere offer. Yet online learning simply replaced one type of sheep-dip (offline) with another (online). The fact that they rarely delivered opportunities for reinforcement and practice, was the same in both camps. AI, the algorithmic delivery of simple and effective online learning, through effortful and deliberate practice, can change this.

Conclusion

We can now, for certain types of leaning, produce content at 10% of the cost, in minutes not months, with higher retention and recall. We can avoid the trap of whizz-bang graphics, weak MCQs and Pavlovian ‘collect the coins’ gamification. The new approach, using AI, creates e-learning which is effortful and allows the learner to expand on their learning with access to external resources and further opportunities for practice. It’s called WildFire.

WildFire takes any document, PowerPoint or video and turns it into online learning, within minutes, using AI. More than this, it uses effortful open-response learning to increase retention and recall. Beyond this it automatically curates links to content beyond the course and, in real time, can create online learning from this content. It is unique. For more information see the WildFire website.