Saturday, November 30, 2019

Video for learning –15 things the research says – some may shock you…

Video for learning is great at some things, not so great at others. Some time back I attended a session on video for learning and found that neither the commissioners nor the video production company had looked at a single piece of research on video and learning. They told us that people loved the work and since then I’ve seen a lot of 'Netflix for learning’ delivery. That’s fine but let’s step back a minute..What I didn’t see was much evidence that people were actually ‘learning’ much. I have been involved in video production for 35 years, from corporate videos delivered on VHS cassettes, interactive videos on Laservision (in many ways way beyond what we currently see), CD-ROMs and now streamed. We even made a feature film The Killer Tongue – lost a pile of money on that one and lesson learnt.
Video is great for emotional impact and so attitudinal shift, showing processes and procedures, moving objects, going inside things down to the micro-level or to places we can’t get to like space or the inside of a nuclear reactor. It can also present people, talking to you. But what works best for LEARNING? The evidence shows that many of the things we do in video are just plain wrong. There’s decades of research on the subject, that mostly remains unread and unloved. For a good summary, read Brahme (2016). So let’s surface a few of these evidence-based pointers.

What can we learn from Netflix?

It wiped out Blockbuster and has become a behemoth in the entertainment industry because it adopted a demand-AI-driven, technological and personalised solution in a world that was stuck in scheduled, supply-side delivery. Netflix is easy to use, searchable and personalized, extensively A/B tested, tiled and has an AI recommendation engine behind the interface. It is also time-shifted, streamed and multi-device. But what really makes it sing is the data-driven algorithmic delivery. This is why learning and video folk need to pay attention to technology, especially AI. That other behemoth, YouTube, the largest learning platform on the planet, is also searchable, personalised and time-shifted.
But Netflix is entertainment, not learning content, so what more do we have to learn, from cognitive science about what makes great learning video?

Episodic v semantic memory

Many can recall famous scenes from their favourite film. Mine is the Rutger Hauer scene from Bladerunner, where he sits, dying on the roof in the rain and gives a moving soliloquy about the joys of life, forgiveness and death. Many recall the scene well but few can remember what he actually said. That is because our episodic memory is strong and video appeals to that form of memory, but video is terrible on semantic knowledge, what we need to know in terms of language. I can remember the scene and literally play it back in my head but I can’t remember the speech. This is why video is not so good at imparting detail and knowledge. There is a big difference between recalling episodes and knowing stuff. Yet so often we see talking heads and text on screen that tries to do exactly that.

Shooting star

Learning is a lasting change in long-term memory and video suffers from the lack of opportunity to encode and consolidate memories. Your working memory lasts about 20 seconds, can only hold three or four things in the mind at one time and without the time to encode it is quickly forgotten Sweller (1988). Our minds move through video like a shooting star, where the memories burn up behind us. Without additional active, effortful learning, we quite simply forget. A big warning here, people think they have learnt from video but as Bjork (2013) and many others have shown, much of this is illusory learning, mistaking the feeling that you’ve learnt things but when tested, you have not.

How long do learners engage?

We know from Guo (2014), who had a large data set of learning video data gathered from MOOCs, that learners drop out in large numbers at around six minutes. It drops dramatically down to 50% at 9-12 minutes and 20% beyond this. Evidence from other studies on attention, using eye-tracking, confirm this quite rapid drop in arousal Risko (2012). Keep videos at 6 minutes or less – the less the better.

Student control of video?

Should you give learners control of video? Yes. In fact giving them 'chapter' points beyond the usual forward, back and pause controls proves useful, as it matches the structure of the learning experience Zhang (2006).

Video, audio, text on screen?

It is vital to reduce cognitive load in video. Mayer (2003) and others have shown that text plus audio plus video on the screen, commonly seen in lecture capture, actually inhibits learning. Don’t put captions, text or scripts on the screen while the narrator or person on the screen is talking. One exception to this (rules are there to be broken) is in language learning, where the ability to jump from voice to text can prove useful. Also avoid complex backgrounds and background music, as they also inhibit learning Brahme (2016).

Talking heads or not?

This comes as a bit of shock to many but talking heads, that staple of lecture capture and MOOCs, is often not such a great idea. It can be useful if the person is a respected and well-known expert and there is evidence that the mere knowledge of them as famous experts increases retention and there is a social learning effect. Inspirational talks, well-delivered with emotional impact can also work as do short introductions, links and goodbyes, to take advantage of the social effect. But there is nothing worse, in terms of learning that an autocued talking head, mechanically delivering a scripted piece to camera in the expectation that people will remember what he or she has said – they won’t Chaohua (2019). Fiorrella and Mayer (2018). We have ample evidence for this in the Khan Academy where you don’t see Salmon Khan’s face, only the working through of examples or images. Cognitively, the face is often just noise, what you want to see is the actual maths, with the hand drawing and writing on screen. For subjects that are semantically rich, getting rid of the face is a good thing as it reduces cognitive load. Social cueing does seem to work, for example, at start and end of lesson and, of course, for subjects where visuals are less useful - story-telling, philosophy and so on.

Recommendations on speech?

Avoid over-formal, TV-news like presentation from autocue. The evidence suggests that  informal is good, enthusiastic is good, personal ‘I’ ‘you’ ‘your’ is good. Speaking between 185–254 words per minute seems to be optimal and no background music Mayer (2008), Brahme (2006).

Perspectives in learning video?

When showing the manipulation of objects by hand or, in general, procedures and processes, video shot from the perspective of the learner, is better that third-person viewpoints. This may upset traditional Directors but it means that you get cognitive congruence Florella (2017).

How does image size affect learning?

This should worry those whose video as a great medium on smartphones. The smaller the screen the less you learn. Or the larger the screen the more you learn. Nass and Reeves (1996) showed 60 video segments to 125 adults on different screen sizes and tested them a week later. Those that had watched the larger screen remembered “significantly more”.

How does video quality affect learning?

Video quality doesn’t appear to matter as much as audio quality. That’s because our vision has evolved to cope with twilight, different light conditions and so on. Our hearing, on the other hand, is designed for close proximity speech Nass and reeves (1996). Their research showed that on attention, memory and evaluation, video quality had “no psychological advantages” whereas audio quality was critical.

How do video cuts and pace affect learning?

Cuts raise attention, about 1 sec after the cut but too many cuts lowers attention
So be measured with cuts and make sure they match the ‘need to know’ learning objectives. It is worth going at a slower pace with cuts, as the learners needs to process information before it goes from working to long-term memory Nass and Reeves (1996).

Should we chunk video?

OK here’s the big one. Learning improves when there are “visual rests” and memory is enhanced when ”people have a chance to stop and think about the information presented". We need to stop and think about that a little. It seems as though micro-chunking video down to smaller segments is a good thing for learning Florella (2019).

What should the relationship be between video and active learning?

This may seem obvious but it is often ignored in practice. After video the effortful learning must be very closely related to video content. A large data mining exercise showed that this is not always the case and that when the two are too loosely related it affects student attainment MacHardy (2015).

What do you do after/between video chunks?

Don’t get them to draw stuff, get them to explain what they think they’ve learnt. You can do multiple choice but this is the weakest of the techniques. Retrieving key concepts is better Szpunar (2013), Roediger (2006), Vural (2013) and typing in actual explanations, that can be semantically interpreted by AI is even better.
Wildfire uses AI to grab the transcript of the video and produce retrieval-based online learning. It does it quickly and cheaply with links to external content, making the learner type in concepts, numbers or full sentences, which are interpreted by AI.


To produce effective videos for learning we must look at learning research. That not only tells us about the cognitive basis of good learning, it also gives us sound recommendations about what we need to do to increase retention, some of it surprising and counterintuitive. You have to break some of the rules of traditional video production and directing to adjust video towards learning. Oh... and rules are there to be sometimes broken... but only with good reason.


Sweller, J., 1988. Cognitive load during problem solving: Effects on learning. Cognitive science12(2), pp.257-285
Bjork, R.A., Dunlosky, J. and Kornell, N., 2013. Self-regulated learning: Beliefs, techniques, and illusions. Annual review of psychology64, pp.417-444.
Brame, C.J., 2016. Effective educational videos: Principles and guidelines for maximizing student learning from video content. CBE—Life Sciences Education15(4), p.es6.
Guo PJ, Kim J, Robin R.  L@S’14 Proceedings of the First ACM Conference on Learning at Scale.New York: ACM; 2014. How video production affects student engagement: an empirical study of MOOC videos; pp. 41–50.
Risko, E.F., Anderson, N., Sarwal, A., Engelhardt, M. and Kingstone, A., 2012. Everyday attention: Variation in mind wandering and memory in a lecture. Applied Cognitive Psychology26(2), pp.234-242.
Zhang, D., Zhou, L., Briggs, R.O. and Nunamaker Jr, J.F., 2006. Instructional video in e-learning: Assessing the impact of interactive video on learning effectiveness. Information & management43(1), pp.15-27.
Mayer, R.E. and Moreno, R., 2003. Nine ways to reduce cognitive load in multimedia learning. Educational psychologist38(1), pp.43-52.
Chaohua, O, Joyner, D, Goel, A., 2019. Developing Videos for Online Learning: A 7-Principle Model. Online Learning
Mayer, R.E., 2008. Applying the science of learning: Evidence-based principles for the design of multimedia instruction. American psychologist63(8), p.760.
Brame, C.J., 2016. Effective educational videos: Principles and guidelines for maximizing student learning from video content. CBE—Life Sciences Education15(4), p.es6.
Fiorella, L., van Gog, T., Hoogerheide, V. and Mayer, R.E., 2017. It’s all a matter of perspective: Viewing first-person video modeling examples promotes learning of an assembly task. Journal of Educational Psychology109(5), p.653.
Reeves, B. and Nass, C.I., 1996. The media equation: How people treat computers, television, and new media like real people and places. Cambridge university press.
Fiorella, L., Stull, A.T., Kuhlmann, S. and Mayer, R.E., 2019. Fostering generative learning from video lessons: Benefits of instructor-generated drawings and learner-generated explanations. Journal of Educational Psychology.
MacHardy Z, Pardos ZA. Evaluating the relevance of educational videos using BKT and big data. In: Santos OC, Boticario JG, Romero C, Pechenizkiy M, Merceron A, Mitros P, Luna JM, Mihaescu C, Moreno P, Hershkovitz A, Ventura S, Desmarais M, editors. Proceedings of the 8th International Conference on Educational Data Mining, Madrid, Spain. 2015
Szpunar, K.K., Khan, N.Y. and Schacter, D.L., 2013. Interpolated memory tests reduce mind wandering and improve learning of online lectures. Proceedings of the National Academy of Sciences110(16), pp.6313-6317.
Roediger III, H.L. and Karpicke, J.D., 2006. The power of testing memory: Basic research and implications for educational practice. Perspectives on psychological science1(3), pp.181-210.
Vural, O.F., 2013. The Impact of a Question-Embedded Video-based Learning Tool on E-learning. Educational Sciences: Theory and Practice13(2), pp.1315-1323.

OEB Berlin… AI, video, learning analytics, data, and schnapps!

Been going for many years… and like the vibe… fruity mixture of tech, academic, government from 73 countries. There's the added attraction of Berlin, a Christmas market right across the street and free drinks at the Marlene Bar!
Gave three talks, all on AI theme… AI and ethics (the overblown hysteria), Learning Analytics (how to and real examples), Video and AI (research and how to). A number of the sponsors were companies that use AI and it was a solid theme this year, rightly, as it has already changed why, what and how we learn.
As is usually the case at conferences I found the smaller presentations and conversations more useful than the keynotes. Great start with Julian Stodd, who was his usual articulate and incisive self. He talked about the weirdness of HR trying to ‘impose’ values and compliance training on people, attacking people’s sense of self and agency. But one phrase that really resonated with me was the ‘humility to listen’. There’s a lot of depth to those three words…
The opening keynotes were a trio of very different fruits. The Max Planc/MIT guy gave a solid talk, showed the Frey and Osborne report (2013), but got the date wrong – it wasn’t 2016 – this matters as it was a paper that predicted the 47% jobs at risk of automation over a decade in the US. We are 6 years in and there is pretty much full employment in the US. Toby Walsh eviscerated this report and talked at this conference two years ago – so we seemed to be going backwards. The Chinese guy was clearly giving a sales pitch but at least he had data and citations to back up his case. Audrey Watters gave her standard  ‘it’s largely agitprop, ideology and propaganda’ replete with soviet posters. Oddly she mentioned being jeered at a summit in Iceland. I was there - it was a very small audience and the first question she was asked (by a woman) was whether she was throwing the baby out with the bathwater, neither was there a 3D cat. She rightly showed some claims that were unsubstantiated but out of context and actually several are evidence-based but Audrey’s was so keen to show that everyone else was ideological that she missed the fact that hers was the most ideological talk of the three. But oh how academe clapped.
The keynotes on the second day (HE session) tackled the future of HE. Professor Shirley Alexander showed the shocking costs, debts and default rates of HE – it is basically out of control on costs. But her solution, literally on the next slide was a huge, spanking new building they’ve just erected and some writing feedback software. I was convinced by nether the erection nor software, which has been around for decades. Bryan Alexander is always up for some fun and opened his talk in a Death Metal voice. Had a great conversation with Bryan about AI afterwards and he did the futurist thing – 3D printing, drones etc…didn’t really see any scalable solutions that tackled the cost issue.
One feature of learning conferences is a general refusal to face up to political issues such as cost and inequality. It is assumed that education is an intrinsic good, no matter what the cost.  No reflection on WHY Brexit, Trump, Gilet Jaunes and other political upheavals are happening, only a firm belief that we keep on doing what we do, no matter the cost. This is myopic. Bryan Caplan tried with his keynote last year, with real evidence, but once again we seemed to have gone backwards. I had a ton of conversations in the bar, in restaurants and over coffees on these issues. A refreshingly straight talk with Mirjam Neelen was one of many.
I liked the practical sessions on learning analytics. It is complex subject but offers a way forward that builds on a platform of data that can be used to describe, analyse, predict and prescribe learning solutions. With smart software (AI), it frees us from the fairly static delivery of media, which online learning has done for over 30 years. Speaking with the wonderfully named Thor and Christian Glahn, we opened up the world of xAPI, LXPs, LRSs and adaptive learning. Here lies some real solutions to the problems posed by the keynotes. 
Sure there are ethical issues and I gave a session explaining that AI is not as good as you think and not as bad as you fear. We went through a menu of ethical issues: Existential, employment, bias, race, gender and transparency. Every man, woman and their dog is setting up and ethics and AI committee, pouring out recommendations and edicts, often based on a thin understanding of ethics and the technology. Many seem designed to give people an excuse to avoid it and do nothing.
Enjoyed Mathew Day’s session on the use of video which is uploaded to the International Space Station, which they use just before they do a task. That’s what I call cosmic, performance support. I was on just before him and showed the evidence in learning theory on why video on its own is rarely enough for deep learning, as well as key evidence on what makes a good learning video, much of it counterintuitive – POV, slower pace, edit points, not so much talking heads, maximum length, adding active learning and so on.
So many interesting chats with people I knew and met for the first time. What I did walk away with was a sense that people are waking up to the possibilities of AI in learning, especially for teaching, Henri Palmer of TUI gave a great case study, showing how one can deliver a large project, super-fast at a fraction of the cost using AI created online content. Great to hear that her team won a Gold Award for that project the night before in London. 
Final dinner in Lutter and Wegner, an old German restaurant was great. Harold did his pitch-perfect Ian Paisley impression at full volume with much clinking of glasses… wine and schnapps. When you’re sitting next to people from Norway, Poland, France, Belgium and Trinidad – you can’t go far wrong.
BIG thanks to Channa, Astrid, Rosa, Rebecca, Harold and the team for inviting me… open people who not only do a great job organising this event but are also open-minded enough to encourage critical thinking…

Saturday, November 16, 2019

Learning - lessons from AI...

AI in one sense means doing what human do when they learn. The field is thick with references to ‘learning’, the most common being machine learning, deep learning and reinforcement learning.
Machine learning uses algorithms and statistical models and applies patterns and inferences to perform tasks, so that it learns from experience.
Deep learning is a machine learning technique that uses layered neural networks with data, supervised, semi-supervised or unsupervised, to perform tasks.
Reinforcement learning operates on maximising reward by exploring existing knowledge and exploring new knowledge. 
Digging deeper there is decision-tree learning, lazy and eager learning, supervised and unsupervised learning, incremental, feature, federated and ontology learning. The noun is used so frequently, in so many contexts, for so many techniques that it has become a general term for software that gets better as it proceeds. This is why it AI is a topic of immediate interest to those in the learning game. 
John McCarthy and Marvin Minsky convened a conference in 1956, whose aim was to 
to proceed on the basis of the conjecture that every aspect of learning (my bold) or any other features of intelligence can in principle be so precisely described that a machine can be made to simulate it”. 
There were successes, like Arthur Samuel’s checkers software, but the promise was never realised and the first AI winter arrived in the 60s. The early 80s saw a resurgence of interest in expert systems and a second winter came. It was only when probability and statistics were literally introduced into the equations that deep learning gave us translation, speech recognition and image recognition. But AI, with exponential growth in processing power, data and devices can now deliver on some of those early promises.
We are now revisiting, and implementing that 1956 objective, with a focus on how AI can be used to accelerate learning. AI makes us rethink learning. It holds the possibility that we do not have to learn some old knowledge and skills, it may help us learn new knowledge and skills, even improve the process and speed of learning. What greater objective that AI helping us to learn; learn to deal with the consequences of this bountiful technology, learn to solve intractable problems, learn to control AI, if necessary. 
Stuart Russell, a major figure in AI, rightly claims in 'Human Compatible', that 
With AI tutors, the potential of each child, no matter how poor can be realised. The cost per child would negligible and that child would live a far richer and more productive life.” I think he’s right. Even if he’s only part right, this is the right direction of travel.

Saturday, November 09, 2019

Doug – a meditation - Things I've learnt from Doug Part II

I’ve never been one for meditation, mindfulness or mentors, never been one for fads… but you know what I've learnt from Doug… to be a little more contented...
I sometimes wake during the night and feel him curled up on the bed, warm asleep. Even better when I open my eyes in the morning and there’s Doug, looking down on me, waiting… you know the day will be all the better for having him around.
Doug, of course, loves a walk. I say a walk, it’s more like being pulled forward by a small locomotive. He springs along like a lamb, checks this, checks that, a judicious sniff here or there, no tree left unexamined, always ahead but often looks back to see that I’m still there. We’re a team, Doug and I. Occasionally he’ll walk behind me, usually mulling over some new scent but we had an ‘incident’ my son and I, when I heard a strange sound behind me, an odd bumping and scraping. Looking round we found that Doug has slipped his harness and was off for a sniff in the bushes. We were literally walking his empty lead and harness for about 30 yards. Lesson learnt.
On walks to Preston Park, we pass the Preston Park Tavern and he veers off, as if on dog autopilot, towards the door of the pub. This is no accident. You see, my sons and I have been taking him to the pub. He loves a flop on the warm, wooden floor, gets the occasional head rub from a punter and generally keeps an eye on what’s going on. But I think what he really loves is that warm, hoppy, beery smell. Good pubs need a dog. I don’t mean bars, I mean pubs. Doug likes pubs, not places with music, too many young people, fishbowls of gin, but pubs and punters. So… you know that old guy in the pub on his own… looking perfectly content… that’s me that is…
Then there’s The Flour Pot, a little cafĂ© up at the Fiveways. We go there, sometimes, after our morning walk, for a coffee and a read of the paper… Doug will snuggle down on the velvety cushion next to me and just watch, head perfectly still, eyes checking out anyone who passes our spot, like a two little searchlights… they have a little jar of dog treats. He likes that.
Now there comes a point in the day, when I sit down and try to stop the tumble of thoughts in the washing machine that is my mind. As I get older I’ve even taken to the occasional nap in the afternoon. Doug has this covered. He loves a nap. And a nap is all the better for doing it together. It helps when Doug flops down on the sofa next to me, his body pushed up against me in a show of solidarity, even better when he places his head gently across my thigh and closes his eyes. That is a moment of pure joy. Sometimes I catch him dream; a quiver, little chew, a wee moan. It’s good to know that we both have dreams. The difference, I suspect, is that his always come true.

For my previous piece on Doug - see here

Sunday, November 03, 2019

Don’t lecture me!

Lectures are an hour long (some astoundingly 2/3 hours) because the Sumerians had a base-60 number system. It is for the convenience of timetabling, not the psychology of attention and retention.
One of the saddest learning stories I’ve ever heard was from the actress Tilda Swinton. She was the only student who turned up to a lecture at Oxford by Raymond Williams where he read out his lecture, from notes, from behind the lectern, and neither of them even acknowledged each other. Studies show quick drop-out from lectures across all subjects. Even at Harvard, where you may be laying out a large five figure sum per year, where GoPro cameras recorded attendance across ten courses, lecture attendance dropped off to only 60% (average) of students attended any given lecture and attendance declined over the semester from 79% to 43%. Imagine running a restaurant, where people pay for the food up front and 40% fail to turn up. You’d surely question the quality of the food.
The bog standard lecture is still the lazy, primary teaching technique in Higher Education. I say lazy because it is based on convention and habit. It is embedded practice as opposed to tased on evidence-based research. Higher Education values research over teaching, yet studiously ignores the research around teaching. Research on attendance is worrying. Research on retention is clear. Research on why researchers don’t make great teachers is also clear. Astin’s longitudinal data on 24,847 students at 309 different institutions found a strongly negative correlation between orientation towards research and teaching. 
It’s an inconvenient truth but researchers are systematic, obsessed by detail and often lack the social skills to be good teachers. Witness the 20 stab-point PowerPoint slides, the often dull delivery, the lack of engagement. Teaching skills demand social skills, communication skills, the ability to hold an audience, keep to the right level, avoid cognitive overload, good pedagogic skills and the ability to deliver constructive feedback. 
If you do lecture then recording is an important stopgap. Suppose that journalists read out pieces once a day in the local square, a novelist reads his book only once and didn’t publish it in book form. That’s unrecorded lectures for you. To deny students second and subsequent bites of the cherry is an act of conceit. Little is learnt on first exposure, most is learnt from subsequent effort. Original lectures are often delivered too fast, especially if it is the learner’s second language. Students can stop, drill down on a point through research, then resume rewind, repeat, watch at any time at any place, take notes second time round, watch after illness,  and move through the course at their own pace.
But the real solution is to largely stop lecturing. In practice, students increasingly learn via Google, YouTube and other online educational tools. At Stanford University this year, I sat with academics and medical students in the same room. Many lecturers were surprised at the range of online tools and apps used intensely by students, fine-tuned to active learning, retrieval, reinforcement, spaced practice, retention and recall. The lecture is increasingly under attack from superior tools, sensitive proven research and how we really learn. We see the rise of smart, AI-driven technology that focuses on the well-researched area of retrieval practice and adaptive learning designed to educate everyone uniquely. Designed to raise attainment and stop drop-out, these tools increase efficacy in learning, yet few academic are even aware of their existence.
Note that I’m not wholly against lectures. Students want to see academics in the flesh. To see a practicising philosopher or physicist, be inspired by that person and subject. This type of inspiring, spot lecture is important. Slabbing lectures out over a term, by researchers who struggle to present, never mind teach, is not just lazy, it is dishonest. Teach me, don’t lecture me.