AI assessment apocalypse – a 5-step solution

Some describe the assessment issue in higher education as apocalyptic, destroying the very fabric of higher education. This is exaggerated. This piece offers an alternative,  to turning learning and assessment into a toxic cat-and-mouse game, where there are many more mice, and the mice are winning, It is about better and more authentic assessment.

There is no silver bullet, as this is a multivariant problem involving student motivations, teaching, institutional practices and technology. Those who simply shout ‘bring back in-person exams’ are ignoring the causes and not offering adequate solutions. The solution involves several steps.

Step 1: Stop the blame

The first step is to admit the serious nature and scale of the problem but also accept it is not the students’ fault. There is a bias in teaching, research and assessment, towards the teacher and researcher. They are the means to an end, not the end in itself. The profession all too ready to blame students and dismiss AI, when these problems were explicit before AI hit the scene. It is hopelessly utopian to expect learners not to use AI. 

Rather than accuse, the solution has to be involve not tempting them with shortcuts that resut from an accusatory environment and poor assessment. Redesign the system to allow more time for teaching and eliminate the temptation that may lead to a toxic environment of accusations, false positives and expulsions; a life-changing disaster for any young person. 

It may also be a life-changing disaster for the faculty member or administrator who ends up making a false accusation. This has already happened with a Minister for Higher Education and leaders of major educational institutions, accused of plagiarism themsleves, and removed.

Most students do not want to cheat. However, when the pressure is overwhelming, from the perception of peers (everyone is doing it, I’d be a fool not to) and parents, tteaching not as good as it could be, assessments poorly designed; students will take available shortcuts. Step 1 is to recognise that cheating is normal and that in high stakes exams, people will take high stakes risks, so don’t blame the students.

We also need to cool down on ths idea that using idea is simply congitive surrender, destroying the leaner's ability to learn. There is a fundamental flaw in most debate about cognitive surrender to AI. The argument that we should be keeping learning difficult, is very different from the idea of useful, deliberate difficulty. As I said earlier, dull lectures, poor teaching, obscure content and accessibility is a big problem that requires more focus on teaching.

Students use AI because they find it useful in learning, to find things out, expand on concepts, unravel things they find difficult, test themselves, produce flashcards for revision and practice. To ban AI would be to throw the baby out with the bathwater, and the bath.

Step 2: No silver bullet

Cheating has always been rampant in education. It was there before AI, with a range of techniques and technologies. I wrote about this in my book ‘Learning Technologies’. There was a whole section on cheating technology from Confucian silk cheat sheets to repurposed calculators, false arms and even surgical implants. Cheating has been an intrinsic feature of educational assessment. As long as there are exams, people will try to take shortcuts.

Let’s take a cold, hard look at in-person essay-based exams. For generations, smart students have looked at past papers, worked out the probability of topics appearing in their exam, pre-writing essays, then memorising them for regurgitation in the exam. The assumption was that we were testing critical thinking. Yet no one who has ever written anything using critical thinking would claim that a piece of writing, written in pen from top left to bottom right of page, without redrafting, reordering and rewriting, even approximated critical thinking. Critical thinking is an internal dialogue where you think, reconsider, revise, seriously reorder and rewrite, as you proceed. This is as far away from regurgitating essays in exams as you can get.

Even in formative essay assessments, students would readily beg, steal and borrow essays from each other, get help from their graduate parents or pay essay mills. These mills were huge enterprises with tens of thousands employed in Nairobi, China and elsewhere, where the well-educated poor provided essays and dissertations for the rich. It was generally ignored by the system (no real moral outrage, as with AI) even though everyone knew it was endemic, especially among students who were studying in their second language. Why? This became a lucrative source of income, the real reason for sliding the problem under the carpet. AI suddenly became one big essay mill, free or cheap, and everyone had access to its services. The revenues of the known cheat companies plummeted.

This is why the current emergency over assessment is really the surfacing of an old and existing problem. We can pretend it is all about AI but AI has merely surfaced a deep, existing problem. It is not fundamentally an AI problem, it is a system and human nehavu=ioural problem. A lot of cheating is the artefact of of the existing teaching and assessment processes and design. It is the same problem that pushes parents to help students with their assignments, hire tutors and pay for exam prep.

Step 2 is to recognise that in-person exams may help, and are not to be scoffed at, but they are not the whole solution, not the silver bullet.

Step 3: Create a Hub

Policies are, at best sticking plasters, at worst they exacerbate the problem. They are certainly not a solution to a large and evolving problem, as they suck up collective effort, are often ignored, then just sit there unrevised and unloved until out-of-date in relation to the advances and uses of AI. 

A policy is one thing, strategy another. Rather than rant and rail around academic integrity, or blaming students, one must understand the problem, then come up withe ‘workable’ solutions. 

There is a lot of hand-wringing and ethical hubris centred around words like integrity, responsible, ethical, trust and so on. This form of abstraction comes easy to academics and administrators but it does not tackle the problems head-on. This is a practical problem that needs workable, pragmatic design solutions.

AI was the fastedst adopted technology in the history of our species and has continued to get better, as it learns. The solution, therefore, needs to involve a process, not a single policy or event. That process needs to be owned and maintained by the institution or cluster of institutions, even nationally. At first, this needs to be a one-stop-shop for advice, tools and services on assessment. This can be part of a wider hub for the use of AI in general, by all in the institution; administrators, researchers, teachers and students. A technology that is globally universal, used by almost all students to learn, warrants this level of attention. Create that hub and keep it up to date.

Step 4: Multimodal assessment

In many subjects, if you depend on just writing as proof of learning, you do not have an AI problem, you have a learning design and assessment problem. When writing is treated as the sole proof of learning for everything, AI exposure isn’t the flaw. the flaw is in the assessment design.

In the real world, all jobs involve the doing of things, dealing with people, using tools, practical tasks. 80% of jobs in the world are deskless, and those that are deskless are being automated by AI. If you do want to switch towards skills that just focus on expression through text, then other forms of teaching and assessment are necessary.

We are now on the other side of  the Gutenberg Parenthesis, where more is available in multiple media formats, from which one can teach and learn. teachers actually speak and listen, we now listen to audiobooks and podcasts, recorded lectures, videos and audio dialogue using AI.

Multimodal assessment is the optimisation of assessment by moving beyond text, now made possible, as AI has become truly multimodal. Models have integrated all media types and can ingest text, audio, images and video, as well as output these media from your text. This offers you the opportunity to free assessment from the tyranny of pure text.

One useful shift, in a world where listening and speaking to others is likely to be more useful than simply writing, is to record oral assessments and use AI to grade them. The argument against oral exams is that they take too much time, but transcription and automatic grading can speed up the process. You also eliminate the stress and problems of worrying about who used AI. Hartmann's (2025) reoriented an upper-level humanities course around oral exams and tracked the time to show that oral exams can verify student understanding directly and, importantly, that they may not take more instructor time than essay.“ Instructor time investment proved comparable to traditional paper grading” with oral exams taking 13 hours , compared to 15 for grading papers. The main point made by the paper was that oral assessment can be integrated back into your existing courses. They may even force your students to do the work, through the idea that they will be properly assessed. 

Video based assessment of performance is also becoming possible as AI can recognise what the person is doing in an uploaded video. This swings effort away from detecting AI, towards designing assessments that assess student performance. All outputs from learners can be ingested and interpreted by AI; text, audio, images and video. 

Digial portfolios are also an option for gathering evidence across the course. What is often missed is that is shows the student thinking, the ability to recall the foundational knowledge that allows thought to flourish and build a case. It shows skills, not in just writing, but a fuller form of expression and doing. This is not to say it is right for all subjects and skills but portfolios and oral exams should be part of the teaching and assessment toolkit.

A multimodal approach also offers a solution to other problems: accessibility and dyslexia. Dyslexics love AI, as it has for years offered a text to speech option. Others find that rewriting and summarising through AI translates the content into something they can understand more readily than often abstruse academic language. In other words, AI is often used by learners to simply ‘access’ and understand content. So widen out your assessment options and think beyond just text.

Step 5: Automate assessment

Automating assessment has become possible in many cases. This is not to say that all assessment should be automated, only ‘optimally’ automated. There can still be expert validation and quality checking. This has the additional advantage of freeing up busy teaching time for actual teaching.

The simple generation and marking of quizzes in formative assessment can clearly be automated. Students do this routinely. They instinctively know the ‘test effect’ works, so build their own quizzes and flashcards with spaced-practice, using AI. Many now use specialist tools like NotebookLM and ChatGPT’s education features, to help them improve the productivity of their own learning. Learning is also being integrated into tools like Google Translate, so that you can practice learning a language through role play and immersion. There is a strong argument for automating much formative assessment via platforms which gives data back to teachers about individual student performance.

Summative assessments have to have strong input by faculty but the questions, rubrics and marking can often be automated by AI. Automating marking is the single most effective way to free up time for teaching, research and other activities. This also includes the marking and feedback of essays.

We can also automate much more detailed feedback. As Dylan Wiliam has been saying for years, far too much assessment has no forward-looking pedagogy. It is seen by students as an end-point, when it should be feeding forward. AI can do this. Have experts in the loop by all means but look at ways to automate the bulk of the work. This as a field that is advancing rapidly, as AI capability progresses.

Step 6: Test Centres

The Opposite of Cheating by Tricia Bertrand Gallant presents a different set of perspectives. She and fellow author David Rettinger, flip the argument and start, not from the institutional, but student perspective.

She argues, based on flipping the debate towards students’ needs, that testing should be the responsibility of separate and shared ‘Test Centres’. These would provide assessment expertise and the ability to design and manage the delivery of assessments. This removes the pressure on faculty, who, on the whole, do not have the necessary expertise on assessment or its delivery. These centres would look at automating as much as possible, while being careful about verification and standards. It is clear that as AI improves, and it is at a blistering pace, so the automation of assessment and marking will become easier, better and cheaper.

Assessment is a rapidly evolving problem that needs this rapid and adaptive response. This is not a final solution as such, but a new approach to a growing problem that focuses expertise, while relieving the system of the unbearable pressure of producing and policing assessment. Teachers are not cops.

The question is whether this should be a single, clustered or national initiative. Huge savings would be possible if it were organised by the sector nationally. This is unlikely, as there is no real legal or political mechanisms for such a strategic approach. Tertiary education institutions are not known for their sharing, so even clusters are unlikely. That does not invalidate the strengt hof the idea.

These test centres could be physical but more sensibly virtual. This would allow testing at any time, on any subject. It is bizarre that one can only get tested on one day of the year, resits often not available for months, even a year later. Imagine of this were true of driving? A Test Centre should allow testing on demand. 

Conclusion

Oddly, the AI prwessure is forcing tertiary education to rethink, and reassess its own assessment. This is long overdue. It is acting as a catalyst for reshaping the role of the teacher and learner in relation to technology, accepting that AI is here to stay. The alternative is to boil like the proverbial frog, fail to respond with anything other than a policy document, constantly accusing students and/or institution for failing to properly assess learners. This is the road to ruin and regret, not the road to success.

Bibliography

Hartmann, C. (2025). Oral exams for a generative AI world: Managing concerns and logistics for undergraduate humanities instruction. College Teaching.

Learning theorist gets Nobel Prize….

When people use the word ‘AI’ these days they rarely understand the breadth of the field. One horse may have won the Derby by a golden mile recently, GenAI, but there’s a ton of other stuff in the race.

In the pre GenAI days, way back in 2014-2021 I used to regularly talk about Alphafold as an astonishing, measurable breakthrough for our species. This one tool alone remains a monumental achievement and by far the most important person in the tripartite award is Demis Hassabis.

AlphaFold, developed by DeepMind in 2020, predicts protein structure prediction. It both accelerates and opens up a vast array of research opportunities. They thrashed the competition in the CASP14 competition, outperforming the other 100 other teams, with a gargantuan leap in the field. It literally shocked everyone.

DeepMind had released a database containing over 200 million protein structures. This includes structures for nearly all cataloged proteins known to science. This database is FREE to the global scientific community, democratising access to high-quality protein structures.

The productivity gain is mindblowing. The traditional methods using incredibly expensive equipment and expertise took years for just one protein. Alphafold does it in hours. This allows researchers to focus on further experimentation, not groundwork. It has literally saved centuries of research.

For example, during the COVID pandemic, AlphaFold predicted structures of proteins related to the SARS-CoV-2 virus. This led to the rapid development of treatments and vaccines. This is generally true in this important, and some feel, neglected field. 

Back to Demis Hassabis, the British entrepreneur, neuroscientist and Artificial intelligence researcher. A chess prodigy and games designer, he was the lead programmer and co-designer of Theme Park, well known in the games world. After a spell as an academic publishing a series of papers, he started an AI company based on his understanding of how the brain and memory works. That company, DeepMind, was sold in 2014 to Google for $628 million.

Learning (memory) theory

Hassabis focused on the hippocampus, as that is where episodic memory is consolidated. He found, through a study of five brain-damaged patients, that memory loss, caused by damage to the hippocampus, was accompanied by loss of imagination (the ability to plan and think into the future). This was a fascinating insight, as he then realised that the process of reinforcement, was the real force in learning, practice makes perfect. This link between episodic memory and imagination was backed up by other studies in brain scanning and experiments with rats. He proposed a ‘scene construction’ model for recalling memories, which on scale sees the mind as a simulation engine. This focus on the reinforcement and consolidation of learnt practice, deliberate practice, as it is known, when captured and executed algorithmically, generates expertise. This led to him setting up a machine learning AI company in 2010 - Deepmind.

Deep Learning algorithms become experts

DeepMind focused on deep learning algorithms that could take on complex tasks, and here’s the rub - without prior knowledge and training. This is the key point – AI that can ‘learn’ to do anything. They stunned the AI community when their system played a number of computer games and became expert gamers. In Breakout their system not only got as good as any human, it devised a technique of breaking round the edge and attacking from above that humans had not encountered. The achievement was astonishing, as the software didn’t know about these games when it started. It looked at the display, seeing how the scoring worked and just learning from trial and error. Deep Learning takes some aspects of human learning, but combines deep learning with reinforcement learning, called deep reinforcement learning to solve problems. 

AlphaGo beat the Go World Champions Lee Sedol in Seoul 5-1, the game that is the Holy Grail in AI, reckoned to be the most complicated games we play, the pinnacle of games. Lee Sedolm was playing for humanity. The number of possible moves is greater than the number of atoms in the universe. It is trained by many games played by good amateurs. Deep neural networks that mimic the brain, with enormous computing power, trained to perform a task, can go beyond human capabilities. In game two it made moves that no human would and became creative. It learns and goes on learning. Far from seeing this as a defeat Lee Sedol saw it as a wonderful experience and GO has never been so popular.

Conclusion

One of the leading companies in the world, where humans have created some of the smartest software in the world, built that success on the back of learning theory, going back to Hebb and his successors. This should matter to learning professionals as AI now plays a significant role in learning. Software ‘learns’, or can be ‘trained’ using data. In addition to human teachers and learners, we now have software teachers and software that learns. It is not that a machine can beat a human but that it can learn to do even better. It is a sign of things to come, a sign of as yet unknown but astounding things to come in learning. The cutting edge of AI is the cutting edge of learning. His Nobel Prize is well deserved, as it is of such great benefit to the future of our species.

Motivation - in the blink of an AI…. an underestimated dimension of Generative AI

In the blink of an AI, I’ve seen people go from 0 to 60, motivated, keen to learn, creative and more productive. 

In just one example, I have seen Claude have a huge and immediate impact on a small business. Honestly, this is a very real example, with someone I know well. After we showed the CEO what AI could do, he stayed up for hours that night played with it and discovered use after use. I have never seen him more excited about something that he was told would improve his business. For all the networking, business mentors and support he’s had, this blew all of that out of the water and, like a fountain, it has continued to flow. Described by him as “like having a new skilled employee for free” it is the gift that keeps on giving. Make no mistake, this will lead to growth in his business. The sheer enthusiasm of the CEO was infectious. It is particularly satisfying to see the people at the top get the fever as it cascades downward. Not just by example but by giving permission to others to use it in their own jobs. The dam bursts and the water flows.

We need to reflect on dimensions of Generative AI, other than its functionality, and productivity gains, as there is another greater prize to be won – motivation. Dig deeper and we find behavioural change lies at the root of its global use.

High and low agency

There’s a huge difference between high and low agency people in organisations. It’s the difference between those standing still on an escalator and those that briskly walk, even when it’s going down. You can often see and feel this when you deal with its people. High agency in organisations have individuals or teams who have significant autonomy and control over their work.

People are empowered to take ownership of their tasks and projects, and AI scaffolds this activity. They can make decisions, influence outcomes, and take initiative without excessive oversight or restrictive rules. High agency AI environments also foster creativity and innovation, as individuals feel free to experiment and explore new ideas without fear of failure or micromanagement. Of course, with greater freedom comes greater responsibility. High agency implies that individuals are accountable for their actions and outcomes, but this can also lead to a stronger sense of ownership and engagement.

What you tend to see in high agency environments are individuals with the power and autonomy to control their own work, which leads to higher job satisfaction, performance and well-being. Low-agency environments often lead to dissatisfaction, disengagement, and higher turnover rates.

AI and Intrinsic motivation

Self-Determination Theory (SDT) gives credence to this idea of agency. It shows that people have intrinsic motivations to act when they feel autonomous, competent, and connected to others. High agency aligns with these needs, leading to higher motivation and satisfaction.

Everyone has that ‘Holy shit’ feeling when using AI for the first time, feeling More specifically and  Self-determination theory, as defined by Edward Deci and Richard Ryan in their book ‘Self-Determination and Intrinsic Motivation in Human Behaviour’, see the active self, being in control, as the primary driver behind personal growth and fulfilment. It sees intrinsic, not extrinsic motivation as the driver for personal satisfaction and success. Your own need for growth that drives other personal needs. This means growing in competence as well as feeling related or connected to other people. 

The theory has three components:

1. Autonomy - being in control, able to take action that results in actual change. 
2. Competence - learning knowledge and skills to achieve more autonomy
3. Connection or relatedness - feeling attached to other people

AI provides autonomy for people by freeing them to do things felt they were never capable of, so gives immediate agency. It also gives that feeling of rapidly increasing competence, of learning quickly to confirm that feeling of autonomy. On top of this, I’d argue that dialogue with a LLM is like speaking to another person or expert (see research by Nass and Reeves).

AI and wellbeing

Another dimension of motivation in AI, is wellbeing. In ‘Lost Connections: Uncovering the Real Causes of Depression – and the Unexpected Solutions’, Johann Hari, sees the causes of depression and anxiety, not as simply the result of chemical imbalances in the brain but largely social and environmental. Depression, anxiety and unhappiness at work stem from various forms of disconnection in people's lives; disconnection from meaningful work, helplessness, meaningful values, status and respect, a hopeful or secure future and a  disconnection from a sense of meaning. AI can partly help (it is by no means the sole solution) by reconnecting the individual with meaningful work, rebooting intrinsic motivation through a strong sense of productivity and achievement.

We know that personal agency matters in terms of job satisfaction, wellbeing and staff retention. We also know that agency matters in learning. People learn faster when they feel a sense of agency, growth and achievement. AI as door to learning

Low floor, High ceiling, Wide walls 

Intrinsic motivation is amplified by the ease of the interface, along with speed and breadth of results. Donald Norman said good technology should be invisible. The future of online learning is that it will be smart & that these smarts will disappear. The invisible hand of AI will transform why, what and how we learn. But it was Seymour Papert who defined what this should look and feel like in practice. Papert's concept of low floor, high ceiling and wide walls are wholly relevant to AI in both tasks and learning, foundational ideas in the design of productive and learning environments. 

Low Floor is the ease with which a beginner can start using a tool or engaging in an activity. The entry point is astonishingly simple in AI, a chatbot letterbox or voice, simple and accessible, allowing novices, even children or those with no prior experience, to begin learning without feeling overwhelmed. 

High Ceiling offers complexity and depth for those who wish to explore further. It means that learners can continue to build on their knowledge through dialogue, taking on more challenging tasks as their skills develop, something available and evolving fast in AI, with multimodality, coding, data analysis, agents and additional technology built into and around Generative AI.

Finally, we need Wide Walls, a depth, breadth and diversity of knowledge and paths that learners can take within a productive process, learning task or environment. Generative AI seems to have a Degree in every subject, speaking dozens and dozens of languages. It suggests that there are many different ways to engage with the material, catering to various interests and forms of expression. Do you want full exposition, brief summary, checklist? Or writing at the right level in a certain style, even poem or story, in any language? As an image, animation or video? 

Generative AI has, and continues to release these three features. A low floor, high ceiling, wide walls interface promotes inclusivity, creativity, and personalised learning. It encourages the design of educational tools and activities that are accessible to beginners (low floor), offer room for advanced exploration (high ceiling), and provide multiple ways to engage and express creativity (wide walls).

Conclusion

Let me add one other thing. Adding AI to improve productivity, quality and meet individual and organisational objectives, especially if you build proprietary AI applications, adds value for everyone. If you are an individual, you will feel better using this technology, find that a sense of release in being able to do things quicker, have less stress and be more productive. It gives people a release of energy and purpose that many other deliberate interventions, such as courses and edicts from above do not. It has an immediacy, with instant results and gives a sense of wonder.

This is why most use of AI in organisations is ‘on the sly’. Organisations veer towards top-down bureaucratic solutions to try to solve problems, which are often cumbersome, requiring difficult skills to master. We finally have a technology that allows one to be more productive, learn faster and feel better.

Resilence training - where it came from and why it went so badly wrong

My bullshit word for the last couple of years has been 'Resilience'. It is what David Graeber, in his brilliant book, Bullshit Jobs, called making shit up to make money from assumed misery. If you have to attend a hokey conference or conference talk on 'Resilience' you don't and will never have it... to be fair, if you make it through a Resilience training course that should suffice!

Workforce learning professionals are in a state of perpetual angst. They feel they are not listened to and don’t have a voice at the top table. This is true HR and L&D have never had any sustainable influence to board level. Hardly surprising when we deliver courses on things neither the business nor its employees ever asked for. I have never, ever heard any normal person say what they need is a ‘course’ on ‘resilience’. It is something supplied by L&D not demanded by organisations, a chimera to make us look caring and important. This has been a worrying trend in workplace learning the delivery of courses based on abstract nouns that no one ever asked for. We supply things we think are relevant, rather than look at what the business demands in terms of goals.

Having beaten into people that they have all sorts of mental deficits, through billions spent on DEI, ESG and wellbeing training courses and initiatives, thrashing employees like piñatas, telling them they are weak and have deficits that need cured by courses. To remedy this, apart from the endless groundhog debates on the future of L&D at conferences, we come up with abstract concepts around which conference sessions and courses have to be built. The current obsession is with ‘Resilience’. These are too often bouts of over-earnest classroom courses or weird e-learning. All of this despite the overwhelming evidence, over many years, that this does not work, it continues. A ‘roll of the eyes’ is the most common reaction when you ask people what they think of all this.

So, as HR has turned into defending the organisation against itself, they then had the temerity to demand that we all need to man and woman up – we need more resilience. It’s like slapping people repeatedly on the face then them telling them to ‘pull themselves together’ before carrying on with the slapping. If your organisation is so dysfunctional that you need to train people to deal with that dysfunction - that speaks volumes about your organisation. Training people to deal with dysfunction is not going to fix it.

Curious history of ‘Resilience’ training

Resilience has deep roots in psychiatry, especially Freud and his daughter Anna Freud, on how individuals cope with trauma and adversity, who I discussed in detail in a recent podcast and whose theories are literally flights of fancy. also Bowlby and Erikson (wrong on most counts) pushed this forward in the 50s and 60s. But it was in the 70s and 80s, that Emmy Werner and Ruth Smith did longitudinal studies on children in adverse conditions. Their work, especially the Kauai Longitudinal Study, highlighted the factors that contributed to resilience in at-risk and sick children.

In training, resilience emerged from the positive psychology movement in the late 1990s when Martin Seligman emphasised the importance of building strengths and well-being, rather than just treating mental illness. Also discussed in detail in this podcast. He backtracked somewhat and more recent evidence shows that the training and wellbeing programmes are not effective at all.

Antifragility 

Modern Resilience training is a mishmash of all of this but there is one book that people thought promoted resilience but was in fact an attack on resilience and resilience training. That book was Antifragile: Things That Gain from Disorder by Nassim Nicholas Taleb. 

Taleb was a derivatives trader then hedge-fund manager and anyone who has actually read the book will know that he hates resilience training. “The fragile want tranquility, the antifragile grows from disorder, and the robust doesn’t care too much.” This is a man who is fiercely critical of certain elites and experts, particularly those he believes are detached from real-world consequences and has resonated resonate with populist sentiments.

He defines resilience as a mistake in that it promotes the ability to resist shocks but stay the same. A resilient system can withstand stress without significant damage but does not necessarily improve from the experience. By contrast, antifragile systems thrive and grow stronger in the face of stress and adversity. Taleb advocates for antifragility over mere resilience because antifragile systems benefit from disorder and challenges.

Taleb argues that resilience training, focuses on helping individuals or systems return to their baseline state after a disruption, an approach that misses the opportunity to leverage stressors for growth and improvement. It creates a false sense of security encouraging individuals and organizations to believe they are adequately prepared for challenges when, in fact, they are only prepared to endure them, not to improve from them.

We need to deliberate expose people to manageable levels of stress and variability to stimulate to build stronger, more adaptable capabilities. People need to continually seek out challenges that push their boundaries and enhance their capabilities so they can survive disruptions but actively using them as catalysts for innovation, embracing and leveraging stressors and challenges to achieve growth and improvement. In practice he doesn’t like HR, L&D as they are part of the bureaucracy of institutions promoting rules and rigidity, fixed outlooks and fixed career paths. Individuals should rather seek out challenges, embrace uncertainty and new experiences that push their boundaries and expand their capabilities.

Conclusion

And so we end up with a hotchpotch of stuff wrapped up into a disjointed PowerPoint and call it Resilience training. We need to stop building empires around ‘big words’ and get back to training competences to solve the skills shortages that all employers report.

What does the learning game have to learn from the beautiful game - football? Data really matters...

Most professional sports employ data to improve performance. Yet football (soccer), in data terms, is not so much the beautiful game as a rather messy and random affair or in statistical terms – stochastic. This refers to the level of unpredictability and the influence of random factors on its outcomes. Football is often considered highly stochastic compared to some other sports due its low scoring, flow with few stoppages and more unpredictable events and player performance and other variables.

It's important to note that all sports have some level of stochasticity but comparatively, sports like tennis and basketball have higher scoring and more frequent scoring opportunities, which can reduce the impact of a single random event. Baseball and cricket, with their more structured and turn-based nature, allow for more consistent application of skill and strategy, though they still have elements of unpredictability.

Unlike many sports, such as basketball, American football and baseball, in soccer the ball changes sides so often it is difficult to identify patterns in the numbers. That is not to say they don’t exist. As usual, the data, although messy, reveals some surprising facts:

1. Corners don’t matter that much. Mourino was amazed when English supporters cheered corners, as he knew they rarely led to goals. The stats support this. There is no correlation between corners and goals – the correlation is essentially zero.

2. Then there’s an old myth that teams are at their most vulnerable after scoring a goal. Teams are not more vulnerable immediately after scoring goal. In fact, the numbers show that this is the least likely time that a goal will be conceded.

3. Coin toss is the most significant factor in penalty-shootout success. 60% of all penalty shootouts have been won by coin toss winners. Goalkeepers who mess about on the line and hold their hands high to look bigger also have an effect, making a miss more likely. Standing 10 cms to one side also has a significant, almost unconscious effect on the goalscorer, making one side look more tempting.

4. It’s also a game of turnovers. The vast amount of moves never go beyond four passes. This has huge consequences – ‘pressing’ matters, especially in final third of field. Avoiding turnovers is perhaps the most important tactic in football.

These were just a few of the secrets revealed by Chris Anderson and David Sally, two academics, from Cornell and Dartmouth, in their book The Numbers Game – Why Everything You Know About Football is Wrong.

Artificial Intelligence

A new tool has caused a bit of a splash, called Tactic AI. A paper in Nature confirms its use in the taking of corners – although, As I say above, this is an odd focus as other tactics are more valuable. Google have worked for four years at Liverpool FC. Yet it is in other areas that data matters more, in scouting and transfers. Brighton (my home team) are lowest in Premiership on corners won but have one of the best track records in transfers, as they use data more widely. Brighton have sold on a nearly decent Premiership team to rest of Premiership: Sanchez, Curucalla, White, Bissouma, Ciaicedo, MacAllister, Trossard, Burn, Maupay, Knockaert... for getting on to a half a billion. These are key players in these other top teams.

Bias

Seasoned managers, coaches, trainers, players often get it wrong because, in football, our cognitive biases exaggerate individual events. We exaggerate the positives and what is obvious and seen at the expense of the hidden, subtle and negative. A good example is defending. Mancini may have been the greatest defender ever because of what he never did – tackle. We prize tackling, yet it is often a weakness not a strength. We think that corners matter when they don’t. Similarly in education, we prize the opinions of seasoned practitioners over the data: exams, uniforms, one hour lectures, one hour lessons and all sorts of specious things just because they are part of the traditional game. Yet, what good teachers don't do really matters. This is why guided coaching and tons of deliberate and variable practice matter in sports but is rarely taken seriously in education.

Soccer and learning

If a sport like football, which is random and chaotic, can benefit from data and algorithms that guide action such as buying players, picking players, strategy, and tactics, then surely something far more predictable, such as learning, will benefit from such an approach? What we can learn is that data about the ‘players’ is vital, what they do, when they do it and what leads to positive outcomes. It is this focus on the performance of the people that really counts, a personalised approach to learners, that is so often missing in learning.

Education gathers wrong data

Education has, perhaps, been gathering the wrong data – bums on seats, contact time, course completion, results of summative assessments, even happy sheets. What is missing is the more fine-grained data about what works and doesn’t work. Data about the learner’s progress. Here we can lever data, through algorithms to improve each student’s performance as they take a learning journey. We need the sort of data that a satnav uses to identify where they start, where they’re going and, when they go off-piste, how to get them back on track. In modern sports going over videos of a team's performance and those of the opposition has become normal, as has the gathering of stats. What has most often led to the goals you've scored this season? It may not be the quality of the striker but what wing is better, the feeder players from midfield, the importance of dead-ball opportunities.

Just as the ‘nay-sayers’ in football claimed that the numbers would have no role to play in performance, as it was all down to good coaches, trainers and scouts, so education claims that it is all down to good teachers. This is a stupid, silver-bullet response to a complex set of problems. It is partly down to good teachers but aided by good data, learners have the most to gain from other interventions. Education needs to take a far more critical look at pedagogic change and admit that critical analysis leads to better outcomes. This means using data, especially personal data, in real time to improve learner performance

 

Personalised tutors - a dumb rich kid is more likely to graduate from college than a smart poor one

A dumb rich kid is more likely to graduate from college than a smart poor one and traditional teaching has not solved the problem of gaps in attainment. Scotland, my own country, is a great example, where the whole curriculum was up-ended and nothing has been gained. In truth, we now have a solution that has been around for some time. I have been involved in such systems fo decades.

AI chatbot tutors, such as Khanmigo, are now being tested in schools but not for the first time. The Gates Foundation has been at this for over 8 years and I was involved in trials from 2015 onwards, and even earlier with SCHOLAR, where we showed a grade increase among users.. We know this works. From Bloom’s famous paper onwards, the simple fact that detailed feedback to get learners through problems they encounter as they learn works. 

“It will enable every student in the United States, and eventually on the planet, to effectively have a world-class personal tutor” says Salman Khan. Gates, who has provide $10 million to Khan agrees, “The AIs will get to that ability, to be as good a tutor as any human ever could” at a recent conference.

 

We see in ChatGPT, Bard and other systems increased capability in accuracy, provenance and feedback, along with guardrailing. To criticise such systems for early errors now seems churlish. They’re getting better very fast.

 

Variety of tutor types

We are already seeing a variety of teacher-type systems emerge, as I outlined in my book AI for Learning.

Adaptive, personalised learning means adapting the online experience to the individual’s needs as they learn, in the way a personal tutor would intervene. The aim is to provide, what many teachers provide, a learning experience that is tailored to the needs of you as an individual learner.

The Curious Case of Benjamin Bloom

Benjamin Bloom is best know for his taxonomy of learning (now shown to be weak and simplistic), wrote a far less read paper, The 2 Sigma Problem, which compared the lecture, formative feedback lecture and one-to-one tuition. It is a landmark in adaptive learning. Taking the ‘straight lecture’ as the mean, he found an 84% increase in mastery above the mean for a ‘formative feedback’ approach to teaching and an astonishing 98% increase in mastery for ‘one-to-one tuition’. Google’s Peter Norvig famously said that if you only have to read one paper to support online learning, this is it. In other words, the increase in efficacy for tailored  one-to-one, because of the increase in on-task learning, is huge. This paper deserves to be read by anyone looking at improving the efficacy of learning as it shows hugely significant improvements by simply altering the way teachers interact with learners. Online learning has to date mostly delivered fairly linear and non-adaptive experiences, whether it’s through self-paced structured learning, scenario-based learning, simulations or informal learning. But we are now in the position of having technology, especially AI, that can deliver what Bloom called ‘one-to-one learning’.

Adaption can be many things but at the heart of the process is a decision to present something to the learner based on what the system knows about the learners, learning or context.

 

Pre-course adaptive

Macro-decisions

You can adapt a learning journey at the macro level, recommending skills, courses, even careers based on your individual needs.

 

Pre-test

‘Pre-test’ the learner, to create a prior profile, before staring the course, then present relevant content. The adaptive software makes a decision based on data specific to that individual. You may start with personal data, such as educational background, competence in previous courses and so on. This is a highly deterministic approach that has limited personalisation and learning benefits but may prevent many from taking unnecessary courses.

 

Test-out

Allow learners to ‘test-out’ at points in the course to save them time on progression. This short-circuits unnecessary work but has limited benefits in terms of varied learning for individuals.

 

Preference (be careful)

One can ask or test the learner for their learning style or media preference. Unfortunately, research has shown that false constructs such as learning styles, which do not exist, make no difference on learning outcomes. Personality type is another, although one must be careful with poorly validated outputs from the likes of Myers-Briggs, which are ill-advised. The OCEAN model is much better validated. One can also use learner opinions, although this is also fraught with danger. Learners are often quite mistaken, not only about what they have learnt but also optimal strategies for learning. So, it is possible to use all sorts of personal data to determine how and what someone should be taught but one has to be very, very careful.

 

Within-course adaptive

Micro-adaptive courses adjust frequently during a course to determine different routes based on their preferences, what the learner has done or based on specially designed algorithms. A lot of the early adaptive software within courses uses re-sequencing, this is much more sophisticated with Generative AI. The idea is that most learning goes wrong when things are presented that are either too easy, too hard or not relevant for the learner at that moment. One can us the idea of desirable difficulty here to determine a learning experience that is challenging enough to keep the learner driving forward.

 

Algorithm-based

It is worth introducing AI at this point, as it is having a profound effect on all areas of human endeavour. It is inevitable, in my view, that this will also happen in the learning game. Adaptive learning is how the large tech companies deliver to your timeline on Facebook/Twitter, sell to you on Amazon, get you to watch stuff on Netflix. They use an array of techniques based on data they gather, statistics, data mining and AI techniques to improve the delivery of their service to you as an individual. Evidence that AI and adaptive techniques will work in learning, especially in adaption, is there on every device on almost every service we use online. Education is just a bit of a slow learner.

 

Decisions may be based simply on what the system thinks your level of capability is at that moment, based on formative assessment and other factors. The regular testing of learners, not only improves retention, it gathers useful data about what the system knows about the learner. Failure is not a problem here. Indeed, evidence suggests that making mistakes may be critical to good learning strategies.

 

Decisions within a course use an algorithm with complex data needs. This provides a much more powerful method for dynamic decision making. At this more fine-grained level, every screen can be regarded as a fresh adaption at that specific point in the course.

 

AI techniques can, of course, be used in systems that learn and improve as they go. Such systems are often trained using data at the start and then use data as they go to improve the system. The more learners use the system, the better it becomes.

 

Confidence adaption

Another measure, common in adaptive systems, is the measurement of confidence. You may be asked a question then also asked how confident you are of your answer.

 

Learning theory 

Good learning theory can also be baked into the algorithms, such as retrieval, interleaving and spaced practice. Care can be taken over cognitive load and even personalised performance support provided adapting to an individual’s availability and schedule. Duolingo is sensitive to these needs and provides spaced-practice, aware of the fact that you may have not done anything recently and forgotten stuff. Embodying good learning theory and practice may be what is needed to introduce often counterintuitive methods into teaching, that are resisted by human teachers. This is at the heart of systems being developed using Generative AI, the baking-in of good learning theory, such as good design, deliberate practice, spaced practice, interleaving, seeing learning as a process not an event.

 

Across courses adaptive

Aggregated data

Aggregated data from a learner’ performance on a previous or previous courses can be used. As can aggregated data of all students who have taken the course. One has to be careful here, as one cohort may have started at a different level of competence than another cohort. There may also be differences on other skills, such as reading comprehension, background knowledge, English as a second language and so on.

 

Adaptive across curricula

Adaptive software can be applied within a course, across a set of courses but also across an entire curriculum. The idea is that personalisation becomes more targeted, the more you use the system and that competences identified earlier may help determine later sequencing.

 

Post-course adaptive

Adaptive assessment systems

There’s also adaptive assessment, where test items are presented, based on your performance on previous questions. They often start with a mean test item then select harder or easier items as the learner progresses. This can be built into Generative AI assessment.

 

Memory retention systems

Some adaptive systems focus on memory retrieval, retention and recall. They present content, often in a spaced-practice pattern and repeat, remediate and retest to increase retention. These can be powerful systems for the consolidation of learning and can be produced using Generative AI.

 

Performance support adaption

Moving beyond courses to performance support, delivering learning when you need it, is another form of adaptive delivery that can be sensitive to your individual needs as well as context. These have been delivered within the workflow, often embedded in social communications systems, sometimes as chatbots. Such systems are being developed as we speak.

 

Conclusion

There are many forms of adaptive learning, in terms of the points of intervention, basis of adaption, technology and purpose. If you want to experience one that is accessible and free, try Duolingo.

ASU trials

Earlier trials, with more rules-based but sophisticated systems proved the case years ago. AI in general, and adaptive learning systems in particular, will have enormous long-term effect on teaching, learner attainment and student drop-out. This was confirmed by the results from courses run at Arizona State University from  2015. 

One course, Biology 100, delivered as blended learning, was examined in detail. The students did the adaptive work then brought that knowledge to class, where group work and teaching took place – a flipped classroom model. This data was presented at the Educause Learning Initiative in San Antonio in February and is impressive.

Aims
The aim of this technology enhanced teaching system was to:
• increase attainment
• reduce in dropout rates
• maintain student motivation
• increase teacher effectiveness

It is not easy to juggle all three at the same time but ASU want these undergraduate courses to be a success on all three fronts, as they are seen as the foundation for sustainable progress by students as they move through a full degree course.

1. Higher attainment

A dumb rich kid is more likely to graduate from college than a smart poor one. So, these increases in attainment are therefore hugely significant, especially for students from low income backgrounds, in high enrolment courses. Many interventions in education show razor thin improvements. These are significant, not just on overall attainment rates but, just as importantly, the way this squeezes dropout rates. It’s a double dividend.

2. Lower dropout 

A key indicator is the immediate impact on drop-out. It can be catastrophic for the students and, as funding follows students, also the institution. Between 41-45% of those who enrol in US colleges drop out. Given the 1.3 trillion student debt problem and the fact that these students dropout, but still carry the burden of that debt, this is a catastrophic level of failure. In the UK it is 16%. As we can see increase overall attainment and you squeeze dropout and failure. Too many teachers and institutions are coasting with predictable dropout and failure rates. This can change. The fall in drop out rate for the most experienced instructor was also greater than for other instructors. In fact the fall was dramatic.

3. Experienced instructor effect

An interesting effect emerged from the data. Both attainment and lower dropout were better with the most experienced instructor. Most instructors take two years until their class grades rise to a stable level. In this trial the most experienced instructor achieved greater attainment rises (13%), as well as the greatest fall in dropout rates (18%).

4. Usability

Adaptive learning systems do not follow the usual linear path. This often makes the adaptive interface look different and navigation difficult. The danger is that students don't know what to do next or feel lost. In this case ASU saw good student acceptance across the board. 

5. Creating content
One of the difficulties in adaptive, AI-driven systems, is the creation of ustable content. By content, I mean content, structures, assessment items and so on. We created a suite of tools that allow instructors to create a network of content, working back from objectives. Automatic help with layout and conversion of content is also used. Once done, this creates a complex network of learning content that students vector through, each student taking a different path, depending on their on-going performance. The system is like a satnav, always trying to get students to their destination, even when they go off course.

6. Teacher dashboards

Beyond these results lie something even more promising. The  system slews off detailed and useful data on every student, as well as analyses of that data. Different dashboards give unprecedented insights, in real-time, of student performance. This allows the instructor to help those in need. The promise here, is of continuous improvement, badly needed in education. We could be looking at an approach that not only improves the performance of teachers but also of the system itself, the consequence being on-going improvement in attainment, dropout and motivation in students.

7. Automatic course improvement
Adaptive systems take an AI approach, where the system uses its own data to automatically readjust the course to make it better. Poor content, badly designed questions and so on, are identified by the system itself and automatically adjusted. So, as the courses get better, as they will, the student results are likely to get better.

8. Useful across the curriculum
By way of contrast, ASU is also running a US History course, very different from Biology. Similar results are being reported. The platform is content agnostic and has been designed to run any course. Evidence has already emerged that this approach works in both STEM and humanities courses.

9. Personalisation works
Underlying this approach is the idea that all learners are different and that one-size-fits-all, largely linear courses, delivered largely by lectures, do not deliver to this need. It is precisely this dimension, the real-time adjustment of the learning to the needs of the individual that produce the reults, as well as the increase in the teacher’s ability to know and adjust their teaching to the class and individual student needs through real-time data.

10. Student’s want more

Over 80% of students on this first experience of an adaptive course, said they wanted to use this approach in other modules and courses. This is heartening, as without their acceptance, it is difficult to see this approach working well.

Conclusion

We have been here before. These systems work. They are already powerful teachers and will become Universal Teachers in any subject. The sooner we invest and get on with the task, the better for learners.

Is handwriting better than typing for note taking? Surprisingly, it's not!

Karl Marx wrote a short summary of every book he read and many scholars and successful people refer to note taking as the secret of their success. I once shared a platform with Richard Branson, where he put his entire business success down to his lifetime habit of taking notes. Apart from being dyslectic, he made the simple point that we forget most of the good ideas we come up with, so taking notes prevents forgetting. He attributed almost all of his business ideas and successes to note taking.

I am also an obsessive note taker and have dozens of black notebooks which have helped me learn and plan over the years. I am often astonished, when speaking to large audiences of learning professionals, how few take notes, when the forgetting curve has been established, since Ebbinghaus in 1885, as one of best known and researched pieces of learning science.

Of course, note taking has always been a staple for learners, especially in Higher Education and the research is clear on their efficacy. Generative note taking and the use of such notes significantly enhances learning. Yet, as technology has become more ubiquitous in learning, the ways in which learners can take notes have expanded. In a study of 577 college students, Morehead (2019), it was found that notes were almost always taken, in notebooks and laptops. Smartphones are also increasingly used to grab images of whole slides, useful when graphs and diagrams are presented but also for the main test points. Students often chose different and combined methods for different courses and contexts. Unfortunately, they don’t always know how and when to optimise their note taking.

That brings me to one of the great myths in learning theory, the idea that it has been proven, without doubt, that hand written notes result in greater learning outcomes than typing.

It is an often deeply held belief among educators that, for learners, handwriting is better than typing. You can see why it is so enthusiastically embraced by those who don't really like this pesky new technology, and that good old fashioned pens and pencils trump the computer. But there’s a problem - it’s not true.

The study that got everyone in such a traditional tizz, by Mueller and Oppenheimer, came out in 2014, with the grand title of ‘The Pen is Mightier Than the Keyboard: Advantages of Longhand Over Laptop Note Taking’. This eye catching title got tons of publicity and a willing audience of believers. It is strange how a study is enthusiastically taken up and remembered when it confirms one biases.

Most note-taking literature pre-dates computers, so the study hypothesised that typing led to shallower processing and that typing tended to encourage more verbatim note-taking. In three studies, it compared learners who watched the same TED videos:

1. Laptop versus longhand performance.

2. Laptop versus longhand performance (students instructed to avoid verbatim not taking)

3. Laptop versus longhand performance (study of notes was included before testing)

In all three cases they noted the superior performance on conceptual questions by longhand note takers

But…

Few picked up on the replication study in 2019. In this study, researchers replicated and expanded the earlier work by using the same videos but adding a group that took notes on an eWriter and a group that took no notes. The researchers also tested students on the content of the videos two days after watching to examine the effect of different note-taking styles on retention. In one version of the experiment, they allowed participants to study their notes before the test to imitate more closely how students use class notes to study for assessments.

When it came to conceptual questions, longhand did not outperform typing. Indeed, in one test, the laptop, eWriter and no notes groups actually outperformed the handwriting group on conceptual questions. In general, when learners were allowed to study their notes, all advantages just disappeared for the retention test.

In truth, this study does not prove it either way, as the results seemed to reverse. But the idea that there is a significant difference is not proven.

Then Voyer et al. 2022, a meta-anaylsis that explored the effect of longhand and digital note taking on performance, showed no effect of method of note taking on performance under controlled conditions. It considered 77 effect sizes from 39 samples in 36 articles, showing no effect on note taking approach.

It would seem that writing notes in your own words, and studying your notes, matter more than the methods used to write your notes. This makes sense, as the cognitive effort involved in studying are likely to outweigh the initial method of capture. It is not note taking that matters but effortful learning.

Digital note taking has the clear advantage of being capable of being edited, formatted, stored, printed, searched and transmitted anywhere across the internet and devices. This blog piece is a good example. It also allows tools such as spellcheck and grammar checks to be applied, citations automatically formatted, images and video imported. Not much text is written in longhand these days.

This debate focuses on one issue, the method of note talking but the more important issue is to move beyond note taking to actual learning. Here we know that underlining, highlighting and rereading are not efficient learning strategies. One needs to move towards effortful, generative learning, deliberate, retrieval and spaced practice. Note taking is not an end in itself, merely the start of a learning journey. It is an important bridge to more effortful learning.

Bibliography

Voyer, D., Ronis, S.T. and Byers, N., 2022. The effect of notetaking method on academic performance: A systematic review and meta-analysis. Contemporary Educational Psychology, 68, p.102025.

Morehead, K., Dunlosky, J., Rawson, K.A., Blasiman, R. and Hollis, R.B., 2019. Note-taking habits of 21st century college students: implications for student learning, memory, and achievement. Memory, 27(6), pp.807-819.

Morehead, K., Dunlosky, J. and Rawson, K.A., 2019. How much mightier is the pen than the keyboard for note-taking? A replication and extension of Mueller and Oppenheimer (2014). Educational Psychology Review, 31(3), pp.753-780.

Mueller, P.A. and Oppenheimer, D.M., 2014. The pen is mightier than the keyboard: Advantages of longhand over laptop note taking. Psychological science, 25(6), pp.1159-1168.