Assessment within institutions is at a crossroads, squeezed between human nature and AI. In an educational system that has to cope with the massification of Higher Education, one that must scale and commoditise the delivery of learning and assessment, you are far more likely to have large numbers of learners who see the process as transactional. If you are transactional with them, they will be transactional with you.
Two major approaches have emerged as solutions to the cheating problem:
1. Catch
2. Change behaviour
We can choose to turn this into a cat and mouse game where the goal is to ‘catch’ miscreants. This may deter some from attempting to cheat but has severe downsides, as it turns assessment into a toxic process, of deliberate evasion, accusation, often with disastrous consequences for learners, and possibly the institutions themselves. AI detection tools are the main line of defence but have been an abject failure, as research uncovers their inadequacies.
Or, we can create system that changes behaviour, where the process of learning and assessment are entwined in such a way that the natural flow of learning is assessed, using AI. Desirable difficulty must be built into such learning and assessment systems, where direct answers or solutions are avoided. The academic integrity problem will not be solved by posturing around the word ‘integrity’, it will be solved by a pragmatic approach to delivering usable learning and assessment tools.
Ai is detrimental to learning when learners use AI to outsourcing intrinsic cognitive work, bypassing desirable difficulty, for example, asking AI to simply write an essay, summarise a paper, give solutions , do your homework and so on.
AI is beneficial to learning if used to outsource extraneous load, freeing working memory to focus on real work of learning. If it designs-in ‘desirable difficulty’, so that students can learn by generating their own thoughts and actions, to truly retain knowledge and skills, it will encourage strong learning and assessment.
STUDI
Studi instils a sense of agency and accountability for the learner’s own learning by providing learning pathways that keep them in the flow of learning, by always having a variety of types of support, through pedagogy-driven chatbots, embedded in the learning process.
This is a form of respectful independence, where learning is seen as a process not an event. Embedded AI, as pedagogic chatbots, captures all the qualities of teacher support, to teach and assesses the process of learning, not wait until they are about to leave the institution, when it us often too late. Teachers still have full visibility of learners and their progress, not through complex dashboards but a unique, single overview image.
STUDI manages learning and assessments, individual students with how many active activities and assessments each student has and their details. For Higher Education, There’s access to a digital content, with modules, papers and citations. The platform tracks students’ progress in reading, interrogating and understanding recommended papers.
My Bots
The AI kicks in with pedagogical bots. This gives the teacher the opportunity to choose the types of pedagogy they want for their students, from Socratic questioning to scaffolded coaching. These include Tutors, Coaches, Facilitators, Instructors, Simplifiers and Mentors. They can be topic-specific.
You can also create your own bots, defining how they teach, question, assess and support your students. To create a pedagogical bot, you give it a name, define its pedagogical purpose, give the main thrust of the bot’s purpose, then add detail, describing, in detail how you want the bot to interact with learners.
You can also control verbosity and tone, make them private or public. A feature that adds an element of desirable difficulty, is the ability NOT give full answers but ask questions. These bots can be cloned by others to save time.
The students have to review and comment on the papers they read, so formative or continuous assessment is the primary approach to assessment. The idea is to change the behaviour of the students by avoiding over-reliance on end-point assessment. By then it is often too late and pushes the learner towards cheating. STUDI allows teachers to assess as they learn.
Feedback
Teachers add feedback to these reviews, making the process more of a feedforward process.
Feedback is the lubricating oil of teaching and learning. Yet feedback in online learning is mostly absent. Some effort is made to get learners to do effortful learning, usually multiple-choice questions or tasks which are returned to the teacher or lecturer. What is so often missing is good FEEDBACK.
Feedback is, to be precise, an accelerator. It speeds up learning and reduces the amount of time spent teaching. It motivates and propels learners forward, pulling them away from cheating. Most of the frustration experienced by learners is poor, slow or inadequate feedback; the embarrassment of being asked questions in a classroom in front of others, even one-to-one by a human tutor, the fear of asking questions in a classroom or in a Zoom session, as you’d feel stupid, the lack of opportunity to ask for clarification or ask questions in a Zoom lesson, classroom or lecture, the email reply that takes days to come back.
Feedback is a process not an event. When people learn online the structure of periods, lectures, classrooms and timetables disappears. You need to work harder to keep learners on task, feedback is the spark and stimulus that gets them to the next stage. Feedback is the process that propels online learning. Technology can deliver both teaching and the feedback.
Learning analytics
Students are tracked with a RAG system showing; GREEN (On track), AMBER (At risk) and RED (Critical). And here is where the system really makes student progress visible in a way few others do.
In one single image, each student is a node and the papers, types of assets, activities and assessments students have attempted are shown radiating from that node. Your entire class can be shown in ONE image, which you can zoom in and out. Individual student records can be examined in detail showing progress, AI providing a summary. You can click on a student or paper node to examine their activity, also get stats on total number of students, papers and percentage completed.
Engagement is tracked through fine-grained search activity, with an ‘understanding’ score, based on keywords used and review quality. This can be examined down to the level of an individual paper or activity. The point is to assess using relevant metrics, such as their understanding, engagement depth and search activity.
Creating assessments is easy, as you can upload content, select your pedagogy bot, from the bots you chose and created earlier, then choose the repositories you want to allow content access. In Higher Education this can be sources such as arXiv, DOAG,PLOS etc.
Students can upload their assessments, and you can set the acceptable file types to keep the process manageable. Importantly, grey publications can be added, as these are important in many subjects, especially those with vocational dimensions.
Embedded Chatbots
AI springs into action with the pedagogic chatbots, not as a crutch or method of cognitive offloading, but an embedded tutor, coach or mentor, encouraging the student to think, move slowly through the process and learn from that process, rather than get straight answers. Desirable difficulty is built into the system and the learner is constantly being asked to think, provide ideas and build their skills.
The beauty of this system is the embedding of chatbots into the learning process, on the same screen. Embedding pedagogical chatbots into the learning workflow is what makes this a single process for the learner. The bots have pedagogical intent, so avoid the dangers of cognitive offloading or surrender.
One proof of the success of this approach is the disappearance of student queries to teachers. We have found that these plummet, giving teachers more time to teach. The system offers a learning workflow that allows the students to proceed with support, allowing teachers to supervise their learning, as opposed to cajole them into action.
