Assessment at a crossroads - don't catch, change behaviour

Assessment within institutions is at a crossroads, squeezed between human nature and AI. In an educational system that has to cope with the massification of Higher Education, one that must scale and commoditise the delivery of learning and assessment, you are far more likely to have large numbers of learners who see the process as transactional. If you are transactional with them, they will be transactional with you.

Two major approaches have emerged as solutions to the cheating problem:

1. Catch

2. Change behaviour

We can choose to turn this into a cat and mouse game where the goal is to ‘catch’ miscreants. This may deter some from attempting to cheat but has severe downsides, as it turns assessment into a toxic process, of deliberate evasion, accusation, often with disastrous consequences for learners, and possibly the institutions themselves. AI detection tools are the main line of defence but have been an abject failure, as research uncovers their inadequacies. 

Or, we can create system that changes behaviour, where the process of learning and assessment are entwined in such a way that the natural flow of learning is assessed, using AI. Desirable difficulty must be built into such learning and assessment systems, where direct answers or solutions are avoided. The academic integrity problem will not be solved by posturing around the word ‘integrity’, it will be solved by a pragmatic approach to delivering usable learning and assessment tools.

Ai is detrimental to learning when learners use AI to outsourcing intrinsic cognitive work, bypassing desirable difficulty, for example, asking AI to simply write an essay, summarise a paper, give solutions , do your homework and so on.

AI is beneficial to learning if used to outsource extraneous load, freeing working memory to focus on real work of learning. If it designs-in ‘desirable difficulty’, so that students can learn by generating their own thoughts and actions, to truly retain knowledge and skills, it will encourage strong learning and assessment.

STUDI

Studi instils a sense of agency and accountability for the learner’s own learning by providing learning pathways that keep them in the flow of learning, by always having a variety of types of support, through pedagogy-driven chatbots, embedded in the learning process.

This is a form of respectful independence, where learning is seen as a process not an event. Embedded AI, as pedagogic chatbots, captures all the qualities of teacher support, to teach and assesses the process of learning, not wait until they are about to leave the institution, when it us often too late. Teachers still have full visibility of learners and their progress, not through complex dashboards but a unique, single overview image.

STUDI manages learning and assessments, individual students with how many active activities and assessments each student has and their details. For Higher Education, There’s access to a digital content, with modules, papers and citations. The platform tracks students’ progress in reading, interrogating and understanding recommended papers.

My Bots

The AI kicks in with pedagogical bots. This gives the teacher the opportunity to choose the types of pedagogy they want for their students, from Socratic questioning to scaffolded coaching. These include Tutors, Coaches, Facilitators, Instructors, Simplifiers and Mentors. They can be topic-specific.

You can also create your own bots, defining how they teach, question, assess and support your students. To create a pedagogical bot, you give it a name, define its pedagogical purpose, give the main thrust of the bot’s purpose, then add detail, describing, in detail how you want the bot to interact with learners. 

You can also control verbosity and tone, make them private or public. A feature that adds an element of desirable difficulty, is the ability NOT give full answers but ask questions. These bots can be cloned by others to save time.

The students have to review and comment on the papers they read, so formative or continuous assessment is the primary approach to assessment. The idea is to change the behaviour of the students by avoiding over-reliance on end-point assessment. By then it is often too late and pushes the learner towards cheating. STUDI allows teachers to assess as they learn.

Feedback

Teachers add feedback to these reviews, making the process more of a feedforward process. 

Feedback is the lubricating oil of teaching and learning. Yet feedback in online learning is mostly absent. Some effort is made to get learners to do effortful learning, usually multiple-choice questions or tasks which are returned to the teacher or lecturer. What is so often missing is good FEEDBACK.

Feedback is, to be precise, an accelerator. It speeds up learning and reduces the amount of time spent teaching. It motivates and propels learners forward, pulling them away from cheating. Most of the frustration experienced by learners is poor, slow or inadequate feedback; the embarrassment of being asked questions in a classroom in front of others, even one-to-one by a human tutor, the fear of asking questions in a classroom or in a Zoom session, as you’d feel stupid, the lack of opportunity to ask for clarification or ask questions in a Zoom lesson, classroom or lecture, the email reply that takes days to come back.

Feedback is a process not an event. When people learn online the structure of periods, lectures, classrooms and timetables disappears. You need to work harder to keep learners on task, feedback is the spark and stimulus that gets them to the next stage. Feedback is the process that propels online learning. Technology can deliver both teaching and the feedback. 

Learning analytics

Students are tracked with a RAG system showing; GREEN (On track), AMBER (At risk) and RED (Critical). And here is where the system really makes student progress visible in a way few others do.

In one single image, each student is a node and the papers, types of assets, activities and assessments students have attempted are shown radiating from that node. Your entire class can be shown in ONE image, which you can zoom in and out. Individual student records can be examined in detail showing progress, AI providing a summary. You can click on a student or paper node to examine their activity, also get stats on total number of students, papers and percentage completed.

Engagement is tracked through fine-grained search activity, with an ‘understanding’ score, based on keywords used and review quality. This can be examined down to the level of an individual paper or activity. The point is to assess using relevant metrics, such as their understanding, engagement depth and search activity.

Creating assessments is easy, as you can upload content, select your pedagogy bot, from the bots you chose and created earlier, then choose the repositories you want to allow content access. In Higher Education this can be sources such as arXiv, DOAG,PLOS etc. 

Students can upload their assessments, and you can set the acceptable file types to keep the process manageable. Importantly, grey publications can be added, as these are important in many subjects, especially those with vocational dimensions.

Embedded Chatbots

AI springs into action with the pedagogic chatbots, not as a crutch or method of cognitive offloading, but an embedded tutor, coach or mentor, encouraging the student to think, move slowly through the process and learn from that process, rather than get straight answers. Desirable difficulty is built into the system and the learner is constantly being asked to think, provide ideas and build their skills.

The beauty of this system is the embedding of chatbots into the learning process, on the same screen. Embedding pedagogical chatbots into the learning workflow is what makes this a single process for the learner. The bots have pedagogical intent, so avoid the dangers of cognitive offloading or surrender.

One proof of the success of this approach is the disappearance of student queries to teachers. We have found that these plummet, giving teachers more time to teach. The system offers a learning workflow that allows the students to proceed with support, allowing teachers to supervise their learning, as opposed to cajole them into action.

AI assessment apocalypse – a 5-step solution

Some describe the assessment issue in higher education as apocalyptic, destroying the very fabric of higher education. This is exaggerated. This piece offers an alternative,  to turning learning and assessment into a toxic cat-and-mouse game, where there are many more mice, and the mice are winning, It is about better and more authentic assessment.

There is no silver bullet, as this is a multivariant problem involving student motivations, teaching, institutional practices and technology. Those who simply shout ‘bring back in-person exams’ are ignoring the causes and not offering adequate solutions. The solution involves several steps.

Step 1: Stop the blame

The first step is to admit the serious nature and scale of the problem but also accept it is not the students’ fault. There is a bias in teaching, research and assessment, towards the teacher and researcher. They are the means to an end, not the end in itself. The profession all too ready to blame students and dismiss AI, when these problems were explicit before AI hit the scene. It is hopelessly utopian to expect learners not to use AI. 

Rather than accuse, the solution has to be involve not tempting them with shortcuts that resut from an accusatory environment and poor assessment. Redesign the system to allow more time for teaching and eliminate the temptation that may lead to a toxic environment of accusations, false positives and expulsions; a life-changing disaster for any young person. 

It may also be a life-changing disaster for the faculty member or administrator who ends up making a false accusation. This has already happened with a Minister for Higher Education and leaders of major educational institutions, accused of plagiarism themsleves, and removed.

Most students do not want to cheat. However, when the pressure is overwhelming, from the perception of peers (everyone is doing it, I’d be a fool not to) and parents, tteaching not as good as it could be, assessments poorly designed; students will take available shortcuts. Step 1 is to recognise that cheating is normal and that in high stakes exams, people will take high stakes risks, so don’t blame the students.

We also need to cool down on ths idea that using idea is simply congitive surrender, destroying the leaner's ability to learn. There is a fundamental flaw in most debate about cognitive surrender to AI. The argument that we should be keeping learning difficult, is very different from the idea of useful, deliberate difficulty. As I said earlier, dull lectures, poor teaching, obscure content and accessibility is a big problem that requires more focus on teaching.

Students use AI because they find it useful in learning, to find things out, expand on concepts, unravel things they find difficult, test themselves, produce flashcards for revision and practice. To ban AI would be to throw the baby out with the bathwater, and the bath.

Step 2: No silver bullet

Cheating has always been rampant in education. It was there before AI, with a range of techniques and technologies. I wrote about this in my book ‘Learning Technologies’. There was a whole section on cheating technology from Confucian silk cheat sheets to repurposed calculators, false arms and even surgical implants. Cheating has been an intrinsic feature of educational assessment. As long as there are exams, people will try to take shortcuts.

Let’s take a cold, hard look at in-person essay-based exams. For generations, smart students have looked at past papers, worked out the probability of topics appearing in their exam, pre-writing essays, then memorising them for regurgitation in the exam. The assumption was that we were testing critical thinking. Yet no one who has ever written anything using critical thinking would claim that a piece of writing, written in pen from top left to bottom right of page, without redrafting, reordering and rewriting, even approximated critical thinking. Critical thinking is an internal dialogue where you think, reconsider, revise, seriously reorder and rewrite, as you proceed. This is as far away from regurgitating essays in exams as you can get.

Even in formative essay assessments, students would readily beg, steal and borrow essays from each other, get help from their graduate parents or pay essay mills. These mills were huge enterprises with tens of thousands employed in Nairobi, China and elsewhere, where the well-educated poor provided essays and dissertations for the rich. It was generally ignored by the system (no real moral outrage, as with AI) even though everyone knew it was endemic, especially among students who were studying in their second language. Why? This became a lucrative source of income, the real reason for sliding the problem under the carpet. AI suddenly became one big essay mill, free or cheap, and everyone had access to its services. The revenues of the known cheat companies plummeted.

This is why the current emergency over assessment is really the surfacing of an old and existing problem. We can pretend it is all about AI but AI has merely surfaced a deep, existing problem. It is not fundamentally an AI problem, it is a system and human nehavu=ioural problem. A lot of cheating is the artefact of of the existing teaching and assessment processes and design. It is the same problem that pushes parents to help students with their assignments, hire tutors and pay for exam prep.

Step 2 is to recognise that in-person exams may help, and are not to be scoffed at, but they are not the whole solution, not the silver bullet.

Step 3: Create a Hub

Policies are, at best sticking plasters, at worst they exacerbate the problem. They are certainly not a solution to a large and evolving problem, as they suck up collective effort, are often ignored, then just sit there unrevised and unloved until out-of-date in relation to the advances and uses of AI. 

A policy is one thing, strategy another. Rather than rant and rail around academic integrity, or blaming students, one must understand the problem, then come up withe ‘workable’ solutions. 

There is a lot of hand-wringing and ethical hubris centred around words like integrity, responsible, ethical, trust and so on. This form of abstraction comes easy to academics and administrators but it does not tackle the problems head-on. This is a practical problem that needs workable, pragmatic design solutions.

AI was the fastedst adopted technology in the history of our species and has continued to get better, as it learns. The solution, therefore, needs to involve a process, not a single policy or event. That process needs to be owned and maintained by the institution or cluster of institutions, even nationally. At first, this needs to be a one-stop-shop for advice, tools and services on assessment. This can be part of a wider hub for the use of AI in general, by all in the institution; administrators, researchers, teachers and students. A technology that is globally universal, used by almost all students to learn, warrants this level of attention. Create that hub and keep it up to date.

Step 4: Multimodal assessment

In many subjects, if you depend on just writing as proof of learning, you do not have an AI problem, you have a learning design and assessment problem. When writing is treated as the sole proof of learning for everything, AI exposure isn’t the flaw. the flaw is in the assessment design.

In the real world, all jobs involve the doing of things, dealing with people, using tools, practical tasks. 80% of jobs in the world are deskless, and those that are deskless are being automated by AI. If you do want to switch towards skills that just focus on expression through text, then other forms of teaching and assessment are necessary.

We are now on the other side of  the Gutenberg Parenthesis, where more is available in multiple media formats, from which one can teach and learn. teachers actually speak and listen, we now listen to audiobooks and podcasts, recorded lectures, videos and audio dialogue using AI.

Multimodal assessment is the optimisation of assessment by moving beyond text, now made possible, as AI has become truly multimodal. Models have integrated all media types and can ingest text, audio, images and video, as well as output these media from your text. This offers you the opportunity to free assessment from the tyranny of pure text.

One useful shift, in a world where listening and speaking to others is likely to be more useful than simply writing, is to record oral assessments and use AI to grade them. The argument against oral exams is that they take too much time, but transcription and automatic grading can speed up the process. You also eliminate the stress and problems of worrying about who used AI. Hartmann's (2025) reoriented an upper-level humanities course around oral exams and tracked the time to show that oral exams can verify student understanding directly and, importantly, that they may not take more instructor time than essay.“ Instructor time investment proved comparable to traditional paper grading” with oral exams taking 13 hours , compared to 15 for grading papers. The main point made by the paper was that oral assessment can be integrated back into your existing courses. They may even force your students to do the work, through the idea that they will be properly assessed. 

Video based assessment of performance is also becoming possible as AI can recognise what the person is doing in an uploaded video. This swings effort away from detecting AI, towards designing assessments that assess student performance. All outputs from learners can be ingested and interpreted by AI; text, audio, images and video. 

Digial portfolios are also an option for gathering evidence across the course. What is often missed is that is shows the student thinking, the ability to recall the foundational knowledge that allows thought to flourish and build a case. It shows skills, not in just writing, but a fuller form of expression and doing. This is not to say it is right for all subjects and skills but portfolios and oral exams should be part of the teaching and assessment toolkit.

A multimodal approach also offers a solution to other problems: accessibility and dyslexia. Dyslexics love AI, as it has for years offered a text to speech option. Others find that rewriting and summarising through AI translates the content into something they can understand more readily than often abstruse academic language. In other words, AI is often used by learners to simply ‘access’ and understand content. So widen out your assessment options and think beyond just text.

Step 5: Automate assessment

Automating assessment has become possible in many cases. This is not to say that all assessment should be automated, only ‘optimally’ automated. There can still be expert validation and quality checking. This has the additional advantage of freeing up busy teaching time for actual teaching.

The simple generation and marking of quizzes in formative assessment can clearly be automated. Students do this routinely. They instinctively know the ‘test effect’ works, so build their own quizzes and flashcards with spaced-practice, using AI. Many now use specialist tools like NotebookLM and ChatGPT’s education features, to help them improve the productivity of their own learning. Learning is also being integrated into tools like Google Translate, so that you can practice learning a language through role play and immersion. There is a strong argument for automating much formative assessment via platforms which gives data back to teachers about individual student performance.

Summative assessments have to have strong input by faculty but the questions, rubrics and marking can often be automated by AI. Automating marking is the single most effective way to free up time for teaching, research and other activities. This also includes the marking and feedback of essays.

We can also automate much more detailed feedback. As Dylan Wiliam has been saying for years, far too much assessment has no forward-looking pedagogy. It is seen by students as an end-point, when it should be feeding forward. AI can do this. Have experts in the loop by all means but look at ways to automate the bulk of the work. This as a field that is advancing rapidly, as AI capability progresses.

Step 6: Test Centres

The Opposite of Cheating by Tricia Bertrand Gallant presents a different set of perspectives. She and fellow author David Rettinger, flip the argument and start, not from the institutional, but student perspective.

She argues, based on flipping the debate towards students’ needs, that testing should be the responsibility of separate and shared ‘Test Centres’. These would provide assessment expertise and the ability to design and manage the delivery of assessments. This removes the pressure on faculty, who, on the whole, do not have the necessary expertise on assessment or its delivery. These centres would look at automating as much as possible, while being careful about verification and standards. It is clear that as AI improves, and it is at a blistering pace, so the automation of assessment and marking will become easier, better and cheaper.

Assessment is a rapidly evolving problem that needs this rapid and adaptive response. This is not a final solution as such, but a new approach to a growing problem that focuses expertise, while relieving the system of the unbearable pressure of producing and policing assessment. Teachers are not cops.

The question is whether this should be a single, clustered or national initiative. Huge savings would be possible if it were organised by the sector nationally. This is unlikely, as there is no real legal or political mechanisms for such a strategic approach. Tertiary education institutions are not known for their sharing, so even clusters are unlikely. That does not invalidate the strengt hof the idea.

These test centres could be physical but more sensibly virtual. This would allow testing at any time, on any subject. It is bizarre that one can only get tested on one day of the year, resits often not available for months, even a year later. Imagine of this were true of driving? A Test Centre should allow testing on demand. 

Conclusion

Oddly, the AI prwessure is forcing tertiary education to rethink, and reassess its own assessment. This is long overdue. It is acting as a catalyst for reshaping the role of the teacher and learner in relation to technology, accepting that AI is here to stay. The alternative is to boil like the proverbial frog, fail to respond with anything other than a policy document, constantly accusing students and/or institution for failing to properly assess learners. This is the road to ruin and regret, not the road to success.

Bibliography

Hartmann, C. (2025). Oral exams for a generative AI world: Managing concerns and logistics for undergraduate humanities instruction. College Teaching.

Robot teaching assistants – I’ve changed my mind…

This is wild. AI is much more than just text generation. It has revived robotics with real dialogue. If this hits the market at sub-car costs, it’s a winner.

Domestic goods changed the world forever, making washing and drying clothes and dishes much easier, as did vacuum cleaners and central heating but the gap remained between static tasks and mobile tasks. We still have to put out the rubbish, get things into these machines, dust, clean, all of that other domestic crap.

I’ve been a sceptic of robot teachers for some time but am starting to change my mind. Why couldn’t a domestic robot play a role in child rearing, talk to children, encourage their curiosity, extend their vocabulary, even teach them to be polite, say thanks and generally be nice to others.

A robot that encouraged a student to do the assignments or homework could work. Then there’s helping to learn in or outside of school. A robot that is endlessly patient could at least perform the function of a teaching assistant, ready to help with specific tasks.

I can’t help but conclude that at some point, probably sooner than we think, these robots will be commonplace. As teaching and learning are still largely 'one to many' why shouldn't they be introduced to pay more attention to individual needs. They could be aware of learning difficulties, such as dyslexia, that an individual learner may have, be sensitive to their personality, know where they are in terms of competence on different subjects.

Then there’s the ability to teach physical tasks. We have stripped this out of the curriculum but learning to do things would be very cool, whether it is playing a musical instrument, cooking or magic! Sports coaches?

Imagine a robot exam invigilator that could block all mobile signals, have a full view of all candidates, hand out and collect papers, check from a database of cheat devices and methods? Just a thought!

These robots have degrees in every subject, so asking them for help is not a problem. They can read your handwriting, hear what you say, speak in any language, at any level, 24/7. It’s OK not to want this sort of future but it’s a choice. This future has something to offer for all ages. I'm in.

Worrying data on how students & institutions are dealing with AI

Just published, this report from Cengage shows that students are stuck in institutions that can’t make their minds up about AI. As we’ve also seen workplace users are often in a similar position. This report should be seen in conjunction with the paper just published by the University of Chicago on the use of AI in the workplace showing it’s everywhere, the main resistance being institutional barriers. Back to the Cengage report and some data.

As 52% of students don’t end up in work even remotely related to their degree subject and career advice is woeful, they know, more than the institutions they’re educated in, how the world is changing, understand that AI is important and doing something about it.

They really are using this technology in anger to learn:

55% grads said their educational program did not prepare them to use GenAI tools.

The reasons for this are clear. Education is highly institutionalised in terms of structures, timetables, funding, quality control and embedded practices. It is these deeply embedded practices, strict hierarchies, bottlenecks and bureaucracy that make it difficult to adapt to new realities. One such reality is AI.

Even worse, their reaction is often to simply reject out of hand, categorise it as cheating or get obsessed about fictional ethical issues. After 40 years in the learning technology game I’ve seen it many times before, with email, photocopiers, calculators, computers, internet, word processing, social media, smartphones, recording lectures…

Yet what this data shows is that these students will emerge into a world where the data already shows massive use of AI. It is fast becoming an essential skill, yet little attempt is made to recognise this or do anything about it. Even worse, it is often demonised to the point of being restricted.

59% of employers say AI has caused them to prioritise different skills when hiring.

Even when we see this dramatic swing in the workplace, we still don’t react. Huge numbers of people are using GenAI when applying for jobs or skilling themselves in using the tools. They know it gives them edge. Why not listen to what is happening - read the research, look at the data.

48% of students say courses were offered to help them learn how to use GenAI

To be fair we’re in a twilight zone, or what Dan Fitzpatrick calls a ‘liminal’ zone’, stuck between the old and new. The first reaction in institutions is usually to deliver a ‘course’.  That’s is fair but not the real solution. 

Yet many are making the effort to embed AI in the educational system. This is mostly in the US but also some notable examples in other countries. There’s a slew of Universities in the US that have heartily embraced the technology and prepare their students for an obvious dimension of the future.

55% of colleges/learning programs discouraged GenAI use

Sadly, there’s the laggards, still stuck is caricaturing AI as some sort of cheat code in their game, for that is what much of this has become, a cat and mouse game on assessment. I understand that it takes time to get to grips with new stuff, but the revulsion and constant moralising, is damaging the future prospects of your learners. I expected fear and loathing in Las Vegas, not our Universities.

51% students say pace of technology making them second-guess their career

This is interesting and heartening. Young people recognise that the system is not helping them and are doing it for themselves, rethinking what they need and want to do. Sure they’re worried about their future, but understand that they have to be part of the real world not just the world of just lectures and essays.

Conclusion

It is important to track the data on use and attitudes as this allows us to readjust for the future. It is perhaps utopian or at least impractical, to expect institutions to change fast but the good news is that tech-savvy young people are doing it anyway.

 

AI changes work and will change who, why and how we recruit

Recruitment is messy. I’ve recruited a lot of people myself from cleaners to C-suite. It has always been a messy business of uninformative ads, dodgy recruitment agencies, odd resumes and even odder interviews. I don’t think I was half bad to be honest, as many of the people I recruited went on to be CEOs of companies they started themselves. I didn’t care about aligning their values with the company, or checking their personality traits with Myers-Briggs BS. I wanted smart, driven people and got them. Not easy to manage but that’s the point, I wanted them to manage!

Anyway, I find myself giving a keynote at a recruitment conference in Oslo. The young people organising it are excellent, very thorough, checking my slides, making useful comments. They didn’t mind me challenging the audience.

After explaining that we Scots are closer to Norwegians that they think, my mother never called us children, always ‘bairns’, to this day the Scottish working class and Scandinavian word for children is ‘barn’. We got the Norwegian Vikings, the English the Swedish Vikings. There’s a difference I joked. At another Keynote in Stockholm last year I heard a speaker make a joke about the Norwegians. I assume there’s some English/Irish thing going on here. How many Norwegians does it take to change a lightbulb? Answer: One, he puts it in and the world revolves around him!’ Not bad. I'm sure thee Norwegians tell the same joke about the Swedes!

After doing my bit on AI, stating that, as AI will radically change the nature of ‘work’ it is inevitable that it will change who, why and how we recruit.

In a recent survey by JobSwipe, large numbers admit to using AI to find new jobs:

• 18 – 24: 47.9% 

• 25-34: 34.5% 

• Women 25.8%

• Men 20.6%

And 60.8% don’t see using AI as cheating. They understand that AI makes finding a job more efficient and enhances your opportunities and chances of getting a job. In a famous TikTok one young man aced his interview with Lockheed for a senior rocket scientist job using live transcription of the questions plus ChatGPT for the answers. He crushed interviews and was hired with zero knowledge. Young people are researching your organisation & sector, creating CVs, creating answers to likely questions and creating questions to ask you and writing emails.

They also understand that certain types of roles and jobs are under threat.

• 49% see writing tasks at threat 

• 47% customer services

• 33% coding

That is reflected in the fact that 29% feel they have not done enough to develop their skills to keep up with the changes being made in the workplace.

HirePro in a blind survey found that 30-50% of people cheat at entry-level job assessments

Candidates ahead of game here. So here are my ten recommendations:

1. Up your game

You need to get to grips with this AI stuff, as well as think about technology solutions, such as a secure browser, using proctoring to detect cheating, even a physical proctor monitors candidates' movements when they're taking the assessments online.

2. Use AI tools to search through databases & online profiles

This widens your pool of candidates, many who may not have applied for position directly. You can more accurately match and go some way towards eliminating human bias in process (gender, race, socioeconomic).

3. AI can GENERATE recruitment materials

From Job ads to Job descriptions, Personal requirements, Skills lists,Formal assessments

and emails to failed and successful candidates, Generative AI will increase your efficiency, productivity and quality of recruitment.

4. AI can ANALYSE recruitment data

Scanning and analysing CVs and application essays to spot gaps, strengths &weaknesses, patterns of employment, even possible misinformation. This is not easy and need objectivity. One can even learn from past recruitment cycles to predict the success of candidates and improve hiring decisions.

5. AI tools can assess candidates 

It can assess during interviews, looking at speech patterns, word/vocabulary choice, facial expressions. This can include the transcript for text analysis and sentiment analysis.

6. Far from introducing biases, AI tools can help reduce human biases

What is bias? The benchmark here is our brain, which, as Daniel Kahneman showed is full of innate biases and they are largely uneducable. There are different meanings of ‘bias’ and we tend to forget that statistics IS identification of bias, so in AI bias can be identified and reduced. It gives us more focus on data and metrics.

7. AI can provide timely updates & feedback

Generative AI improves their experience and engagement with company through quick responses, consistent communications, positive messaging and personalised feedback.

8. AI-driven assessments

Create assessments with MCQs, open input with analysis, scenarios, all with AI generated rubrics and marking. You can go further Evaluate candidates' skills and aptitudes. Tools now generate learning in any subject, at any level. You can build these resources in minutes not months.

9. AI can provide onboarding

You can have a Chatbot (before & after joining), generate training content and assessments for compliance and gather data from onboarding for improvement.

10. AI can maintain engagement with candidates

Keep in touch with past & potential future candidates to build a talent pool for future openings. More importantly, improve communications with all applicants, successful or not.

Conclusion

What do these people have in common?

They were the President of Hungary, President of Harvard University, Minister for Higher Education (Norway) and Minister for Education (Brazil) and all lost their jobs through plagiarism. A nuclear bomb is waiting to go off in recruitment. If academics insist on accusing students of cheating if they use use ChatGPT, they in turn should be subjected to the same standards. All it takes is someone to go back to that Master’s thesis, paper, article or book. This could wreak havoc in recruitment and employment

You may be doing a lot of this already but the point is to do it faster and better. AI changes the very nature of work, it will, therefore change who, why and how we recruit. This mean upskilling now to at least be aware of these AI tools, then use them yourself. Resistance is futile!

 

University enrols two AI students - fascinating experiment

An experiment is taking place in a Michigan University (Ferris University), where two AI students have enroled and will take the same route through courses as their fellow human students. The aim is to evaluate the ‘student experience’. Interesting idea.

We’ve had successful teaching assistants since 2016 in Higher Education, along with adaptive, personalised teaching systems that educate everyone uniquely. We also know that current LLMs can crush high stakes exams in HE but this attempts to track the learning experience of students, so can make its own decisions, choose courses and generally have the agency of a real student.

There is a more fundamental idea here, of the mystery shopper or evaluator for any service, whether in education, healthcare, retail, whatever. Run bot customers through the system and see what happens? This is a great way to identify flaws, redundant processes and risks. It can be built to include risk analysis methods to identify and quantify risks, as well as recommend improvements. The choice of 'student experience' is a bit of a get out. Let's not tackle teaching and learning, let's see if they enjoy their gilded cage?

As a reflective experiment this has some merit. Lots is made of the student experience, yet little is done to improve teaching and learning, with the lecture and essay still rock solid as core pedagogies. Placing a proxy learner in context, going through the motions is interesting. 

Careful what you wish for

They have to be careful what they wish for here. The AI will have a flawless digital memory, will not sleep, be super quick at tasks, never distracted, can multitask, network, never gets a hangover. Will is smash the exams? Will it have the urge to cheat?

The set up will be important. Will they be modeled on typical student behaviour, where 40% don't turn up to lectures?

The AI students are called Ann and Fry – odd as they say there’s no genders attributed? They are not robots but will listen through microphones, do assignment and eventually speak . It will have to go through the admissions process, then registration and make decisions on what classes it wants to take.

In HE, it would, I’d imagine, do a hatchet job on lectures (transience effect, cognitive overload, little interaction, poor slide design and poor teaching). It could compare live to recorded lecture experiences and, in its eyes, conclude that the convenience of recorded lectures wins hands down. It could critique the need for long absences during holiday breaks, the odd idea of having to wait for months even a full year to resit an exam. It could notice the number of students consistently absent from lectures.

To be fair, it is being built in partnership with the U.S. Department of Defense, National Security Agency, Department of Homeland Security and Amazon Web Services. That some pretty heavy fire right there.

The lack of research on using AI to teach and learn has been puzzling, compared to studies on productivity in the workplace. They've been noticeable by their absence. Yet we have truckloads of papers, frameworks and report on ethics and AI. I'd like to see two groups, randomised etc. one with, the other without AI, to measure impact on learning and performance. In may ways that would be more useful.

Of course, we all know what will actually happen… you don’t need millions to see the weaknesses already… but more power to their elbow. The US is taking the AI bull by the horns, while others write endless reports on ethics… it’s easy to be a critic much harder to do do real stuff and test against reality.

 

Thorndike (1874 – 1949) – Experimental rigour and why Latin doesn’t give you transferable skills...

Edward Thorndike deserves much more attention than he gets in the history of educational psychology. Inspired by William James, he was an assiduous experimenter, who revolutionised experimental psychology by introducing scientific rigour into his work. This led to ground-breaking work in important learning topics such as ‘transfer’.

Experimental method

Children were the subject of his early experimental research but Harvard stopped him from using human subjects, so he moved to chickens and cats. In fact his earliest work, ‘Animal Intelligence’ (1898), made his name as a psychologist, as much for its scientific method than the results these methods revealed. His method included the now familiar; representative samples, clear explanations of the experimental method so that it could be reproduced, comparative groups under the same controlled conditions and clear quantitative measurement. Educational psychology up to that point, and beyond, had been conducted in a non-scientific, if not misleading manner, Freud and Paiget being just two of many example.

Two laws of learning

His animal experimental work led to the formulation of two laws, based on the behaviourist Stimulus-Response model, which in his view, governed all learning, even in humans. 

First the Law of Effect; that consequences, either rewards or punishments are a necessary condition for learning. This was shown to be wrong by another behaviourist, Tolman.

Second the Law of Exercise; that learned behaviour fades without practice and is strengthened with practice. The driver behind this is instinct. This focus on rewards, punishment and repeated practice was to dominate behaviourist psychology, and research into learning, for decades.

Transfer

An important early finding was his work (with Woodworth) on transfer in 1901, namely the degree to which learning transfers to actual behaviour and performance in the real world. This is still an often ignored issue in education and training. Learning is largely (not always) a means to an end, namely the application of that knowledge or skills. Yet relatively few educators or instructors know or care much about transfer. Thorndike showed that transfer depends on the similarity of the situations or domains. This principle of ‘identical elements’ led him to recommend problem solving and practice in real-world contexts, so that the learning tasks and context matched the real world.

It is a strong argument for context rich learning,  learning by doing and workplace learning. Transfer involves personal, learning and organisational factors, as explained in What Makes Training Work by Weinbauer-Heidel (2018). These factors, when studied in longitudinal studies Gegenfurtner (2013) and literature reviews Gegenfurtner (2009) are largely in line with Thordike’s original theories on transfer. 

This has had a lasting impact on simulations and now the design of workplace learning and learning in the workflow. 

Personally, the learner has to want to follow through to experience practice and transfer and be confident that they can perform. Personal motivation and intention to transfer have been identified as drivers in successful transfer by Gegenfurtner (2013). Learning experiences must make them aware of the value of doing this in practice. Learners need to believe that they can master and apply what they learn in their work, then learn through practice and application. The training also needs to be clear about expectations in terms of application and doing,going beyond theory. True to Thorndike’s views, job relevance and the congruence between what is learned and what you do, has been a constant finding in the research, Botke et al., (2018). Beyond this, organisationally, learners should be expected to practice, with time available and support, especially social support Botke (2018), within the workplace. Opportunities to use what you learn on the job, Gegenfurtner et al. (2009), need to be made clear or linked to the original learning experiences.

Latin

To take a still current example, one still hears claims that Latin helps one learn other Romance languages or improve performance in other subjects, Thorndike predicted that this is unlikely as there are significant differences in grammar. In Search of the Benefits of Latin by Haas and Stern (2003) in the Journal of Educational Psychology (incidentally founded by Thorndike) revealed that Thorndike “did not find any differences in science and maths in students who learned Latin at school and those who did not” and in the Haag and Stern (2000) follow up study, two groups of comparable German students, where one studied Latin, the other English, were assessed after two years, “No differences were found in either verbal or non-verbal IQ or grades in German or Maths”. This had been predicted by Thorndike decades before, namely that transfer needs common ground in the source and target.

Now for the bad news. In language learning, Latin actually makes it worse. The problem with understanding Latin is that you need to pay close attention to word endings; case markers on nouns and time markers on verbs. But in English, and Romance languages, word order and prepositions are more important, endings play a minor role. What Haag and Stern found, predictably, based on Thorndike’s research, was that students who had learned one Romance language first, found it easier to learn another Romance language, than those who had learned Latin. Latin caused incorrect transfer, such as the omission of prepositions and auxiliary verbs in Romance languages. In other words, learning Latin was detrimental to the learning of the new language.

Testing

Thorndike took a particular interest in testing, developing intelligence and subject-specific tests. This was also to have a long-lasting effect on education, not always positive. Cyril Burt, an English psychologist, the first non-American to win the Thorndike Award by the American Psychological Association, went on to impose a regime of standardised tests, such as the 11+ (still used in some parts of the UK). Burt turned out to be a fraud and a cheat, as he falsified data and even the existence of co-workers.

Influence

Thorndike laid down methods for the scientific measurement of behaviour that were to be used by an entire school of psychology, the behaviourists, people like Watson and Skinner. Establishing scientific rigour in experimental psychology was just one of his lasting achievements. His work on transfer was long-lasting, although conveniently ignored, even today.

Bibliography

Thorndike, E. L. (1898). Animal intelligence: An experimental study of the associative processes in animals. New York and London: Macmillan Co.

Thorndike, E. L. (1906). The principles of teaching: Based on psychology. New York: Seiler

Thorndike, E. L., & Columbia University. (1927). The measurement of intelligence. New York: Bureau of Publications, Teacher's College, Columbia University

Thorndike, E. L. (1901). The influence of improvement in one mental function upon the efficiency of other functions. N.Y: MacMillan.

What Makes Training Work by Weinbauer-Heidel (2018). Tredition.

Gegenfurtner, A., (2013). Dimensions of motivation to transfer: A longitudinal analysis of their influence on retention, transfer, and attitude change. Vocations and Learning, 6(2), pp.187-205.

Gegenfurtner, A., Veermans, K., Festner, D. and Gruber, H., (2009). Integrative literature review: Motivation to transfer training: An integrative literature review. Human resource development review, 8(3), pp.403-423.

Botke, J.A., Jansen, P.G., Khapova, S.N. and Tims, M., (2018). Work factors influencing the transfer stages of soft skills training: A literature review. Educational Research Review, 24, pp.130-147.

aag, L. and E. Stern. (2003). In search of the benefits of learning Latin. Journal of Educational Psychology 95(1), 174-178.

Haag, L. and Stern, E., 2000. Non scholae sed vitae discimus? Auf der Suche nach globalen und spezifischen Transfereffekten des Lateinunterrichts. Zeitschrift für Pädagogische Psychologie/German Journal of Educational Psychology.

Reasons to abandon multiple-choice questions

Long the staple of e-learning and a huge range of low and high stakes tests, the MCQ should be laid to rest or at least used sparingly. Their huge popularity has been due to several factors. First they are an artefact, a hangover from cardboard templates placed over squares to identify 'Xs". The tools for online learning also make them easy to write. This does not make it easy to write them well, in the same way that giving someone Word doesn't make them a novelist or poet.

But it has several flaws…

1. Probability

25% of getting it right on the standard four-option item, makes it less than taxing. True false, really a two option MCQ, is of course worse. I say 25% by research but Rodriguez (2005) shows that three options are optimal, then it goes to 33%.

2. Unreal

You rarely, if ever in the real world, have to choose things from short lists. This makes the test item somewhat odd and artificial, disassociated from reality.  They seem dissonant, as this is not the way our brains work (we don't cognitively select from four item lists in recall or automaticity). They are also weak on recall and therefore weak on the transfer of knowledge to the real world.

3. Distractors distract

It is too easy to remember the distractor, as opposed to the right answer. The fact that they are designed to distract makes them candidates for retention and so MCQs can become counterproductive. The research shows that four option MCQs often contain spurious and extraneous distractors that just add cognitive load and the chance of false recall.

4. Can be cheated

Pick longest item, second-guess the designer. Look for opposites and internal logic of distractor options. There are credible cheat-lists for multiple choice – Poundstone’s research shows that these approaches increase your chance of getting better scores. (20 cheats here)

5. Surface misleading

Take these two questions.

What is the Capital of Lithuania? Tallin, Vilnius, Riga. Minsk

What is the Capital of Lithuania? Berlin, Vilnius, Warsaw, Helsinki

Surface differences in options make these very different test items. And it is easy to introduce these surface differences, reducing the validity of the test items and test. MCQs often have huge variances depending on the options presented.

6. Difficult to write

I have written a ton of MCQs over 35 years – believe me they are seriously difficult to write. It is easy to select the noun from the text and come up with three other nouns. What is difficult it to test, is real understanding.

7. Little effort

This is the big one. As Roediger and McDaniel state in their book Make It Stick, choosing from a list requires little cognitive effort. You choose from a limited set of options, and do not use effortful recall (which is in itself increases retention).

Conclusion
Multiple-choice is not a terrible test item but it has had its day as the primary test item in online learning. We’re still designing test items in lock-step because the tools encourage us to do so, ignoring morepowerful open-response questions that require recall and the powerful act of writing/typing, which in itself, reinforces learning. New tools, such as WildFire, the AI-driven content creation service, focuses on open-response and effortful learning, for all of these seven reasons and more. The learner has to make more effort as that means deeper processing and higher retention.

Gagne's 9 dull Commandments - why they cripple learning design...

50 year old theory
It is over 50 years since Gagne, a closet behaviourist, published The Conditions of Learning (1965). In 1968 we got his article Learning Hierarchies, then Domains of Learning in 1972. Gagne’s theory has five categories of learning; Intellectual Skills, Cognitive strategies, Verbal information, Motor skills and Attitudes. OK, I quite like these – better than the oft-quoted Bloom trilogy (1956). Then something horrible happened.

Nine Commandments
He claimed to have found the Nine Commandments of learning. A single method of instruction that applies to all five categories of learning, the secret code for divine instructional design. Follow the linear recipe and learning will surely follow.

1 Gaining attention
2 Stating the objective
3 Stimulating recall of prior learning
4 Presenting the stimulus
5 Providing learning guidance
6 Eliciting performance
7 Providing feedback
8 Assessing performance
9 Enhancing retention and transfer to other contexts

Instructional designers often quote Gagne, and these nine steps in proposals for e-learning and other training courses, but let me present an alternative version of this list:

1 Gaining attention
Normally an overlong animation, coporate intro or dull talking head, rarely an engaging interactive event. You need to grab attention not make the learner sit back in their chair and mind.
2 Stating the objective
Now bore the learner stupid with a list of learning objectives (really trainerspeak). Give the plot away and remind them of how really boring this course is going to be.
3 Stimulating recall of prior learning
Can you think of the last time you considered the details of the Data Protection Act?
4 Presenting the stimulus
Is this a behaviourist I see before me? Yip. Click on Mary, Abdul or Nigel to see wht they think of te data Protection Act - cue speech bubble... or worse some awful game where you collect coins or play the role of Sherlock Holmes....
5 Providing learning guidance
We’ve finally got to some content.
6 Eliciting performance
True/False or Multiple-choice questions each with at least one really stupid option (cheat list for MC here).
7 Providing feedback
Yes/no, right/wrong, correct/incorrect…try again.
8 Assessing performance
Use your short-term memory to choose options in the final multiple-choice quiz.
9 Enhancing retention and transfer to other contexts
Never happens! The course ends here, you’re on your own mate….

Banal and dull
First, much of this is banal – get their attention, elicit performance, give feedback, assess. It’s also an instructional ladder that leads straight to Dullsville, a straightjacket that strips away any sense of build and wonder, almost guaranteed to bore more than enlighten. What other form of presentation would give the game away at the start. Would you go to the cinema and expect to hear the objectives of the film before you start?
It’s time we moved on from this old and now dated theory using what we’ve learnt about the brain and the clever use of media. We have AI-driven approaches such as WildFire and CogBooks that personalise learning.....

And don’t get me started on Maslow, Mager or Kirkpatrick!

10 reasons why 'Leaderboards suck in learning

Gamification has been around for years (over 30) but I fear it has recently descended into a superficial, Pavlovian box of tricks, overhyped by vendors keen to prove how ‘cool’ and ‘contemporary’ they are . In practice, much of it is a bit hokey, and insensitive to learning theory and, I fear, the whole point of the bigger game – learning.

Don’t get me wrong, I like gamification, in the sense of games’ techniques grounded in good learning theory. These include: levels of attainment, catastrophic failure and repeats, effortful learning, mini-sims and deliberate practice. What I witness, however, is the superficial buzz of the arcade; leaderboards, points, board games, sound effects, noise and animated piffle.

Gamification only works when the cognitive load, that gamification always entails, results in greater learning outcomes. It you go for approaches that are superficially buzzy, yet work against learning, in that they draw the learner away from actual learning, realism, context and transfer, you will have failed.  You may even have demotivated, rather than motivated most of your learners.

This is especially true of the one gamification technique I do not like, and do not recommend - the leaderboard. The word betrays itself, embedding as it does the the word ‘Leader’, but there are many other problems:

1. Decoupled from standards

Leaderboards are inappropriate for personal development. Learning is not a race, not the survival of the fittest. Any league table that ranks people into a linear list decouples the learning from real measures of competence. You can list all you want but that list is not anchored in competences just because it is ranked. Let’s say you have 100 learners in five leagues of 20. The entire 100 may be failing but you can still sort them into a table.

2. Leaders problem

One of the problems with leaderboards is something I experienced on a project for a major bank some years ago, where we set up product quizzes on kiosks (like fruit machines) around the country. The leaderboard was quickly dominated by a few fanatics, one site in particular, who would do anything, including cheat, to stay at the top. Leaderboard encourage people to cheat. These ‘leaders’ became a problem. We had to take it down. This is something I’ve seen elsewhere.

3. Zero sum game

In general, social challenges, where individual learners are exposed to group comparisons and scores, can demotivate those who lag behind. The downside is obvious, seeing people move ahead of you makes you think you’re not good enough. For every person that moves up a place there’s someone who moves down so the motivational spur may, in total, be zero. For every leader there’s a laggard. In fact, it may be worse than this, as even those above the median point may feel disappointed. In fact the leaderboard technique tends to ignore the majority in favour of the few at the top.

4. Self determination theory

Far better to let the learner use themselves as the benchmark, set self-targets and self-goals, then encourage progress in that way. This avoids the danger of disaffecting entire groups who do not see their learning in terms of games and learning. Gamification is often best when it is almost invisible and embedded in the learning strategy. This ‘self determination’ theory is well researched and relies more on encouraging intrinsic, as opposed to extrinsic motivation.

5. Short-term gain but no long-term gain

Over time the above effects become exacerbated. You get a short-term rush and gain, followed by a longer tail of disappointment, then disillusionment.

6. Good and bad failure

Visibility of ranked failure may also decrease that learner’s willingness to learn through good failure. It’s like cholesterol, there’s good failure and bad failure. Bad failure is illusory ranking, good failure is that which you learn from. Ranked lists make learners more risk averse.

7. Tests not behaviours

It is rare that assessment within a learning programme or experience matches the actual required behaviours in the real world. The tests are often simplistic, too text based often simple multiple-choice. You can do well on leaderboards by focusing relentlessly on the types of answers required in text-based tests. See my list of ways to cheat on MCQs.

8. Lower retention

When the score or position in a league table become the goal, there is less chance that the content will actually be remembered and recalled. Deeper processing and reflection is less unlikely when your goal is a score and not competence. It’s a case of in one ear, get the score, then out the other.

9. Conflict

Leaderboards may induce conflict rather than co-operation. When winning or moving ahead of your colleagues becomes the goal, collaborative or team effort becomes secondary. So this individualistic approach may destroy any semblance of collaborative or team behaviour that one would normally want to encourage in workplace learning.

10. Transfer

Transfer of skills to the real word is important and leaderboards set up an alternative, distinct and distractive world for the learner – the leaderboard. This diminishes the congruence between what is learned and the real target in the real world, that is the point of transfer.

Conclusion

All in all, leaderboards may seem like a good idea but for one upside, competitive motivation, but they have many identifiable downsides, statistically, motivationally and in terms of learning theory. The workplace is not a sporting venue, not a race for the top. Some areas may be driven by such competitive urges, arguably sales, but one should not universalise this phenomenon into learning. The learning game is not a penny arcade game, it’s about cognitive change – that’s hard and should not be trivialised. I must reiterate that I am NOT against gamification, only its superficial promises. For more see my Don't play Pavlov with gamification - 10 DOs and DON'Ts.