Tuesday, December 04, 2018

What one, intensively--researched principle in learning is like tossing a grenade into common practice?

Research has given us one principle that is like tossing a grenade into common practice – interleaving. It’s counterintuitive and, if the research is right, basically contradicts almost everything we actually practice in learning.
The breakthrough research was Shea & Morgan (1979), who had students learn in a block or through randomised tasks. Randomised learning appeared to result in better long-term retention. This experiment was repeated by Simon & Bjork (2001), but this time they asked the learners at the end of the activities how they think they’ll perform on day 2. Most thought that the blocked practice would be better for them. They were wrong. Current performance is almost always a poor indicator of later performance.
Interleaving in many contexts
Writing the same letter time after time is not as effective as mixing the letter practice up. HHHHHHIIIIIIIIIIJJJJJJJJJ is not as good as HIJHIJHIJHIJHIJHIJHIJHIJ. This also true in conceptual and verbal skills. Rohrer & Taylor (2007) showed that maths problems are better interleaved. Although it feels as though blocked is better but interleaving was three times better! The result in this paper was so shocking the editors of three major Journals rejected the paper on first reading. The size effect was so great that it was hard to believe, so hard to believe that few teachers even do it.
Interleaved in unrelated topics
Rohrer, Dedrick & Stershic (2015) took this a stage further and took unrelated topics in maths, to compare blocked with interleaved practice. Interleaved produced better performance in both short and long-term (30 days). William Emeny, a teacher in England showed that interleaving is actually done by many teachers but only in run up to exams, but that, he showed was where most of the actual learning was taking place.
Interleaving in inferences
What about learning from examples, learning general skills from exposure to examples, like reading X-rays or inferring a painter’s style by exposure to many paintings by specific painters. Kornell & Bjork (2008) did the painter test, 12 paintings by each of 6 artists, then show learners 48 new paintings. The results showed that interleaving was twice as effective as blocked training. It has been replicated in the identification of butterflies, birds, objects, voices, statistics and other domains. Once again, learners were asked what sort of instruction they thought was best. They got it wrong. In young children, 3 year olds, Vlach et al (2008) showed that learning interleaved with play produced better performance.
So why does interleaving work?
Interleaving works as you are highlighting the ‘differences’ between things. These relationships matter in your own mind. Blocking seems more fluid, where interleaving seems confusing, yet it smooths out comparisons.  Another problem is that learners get years and years of blocking in school. They’re actually taught bad habits and that prevents new, fresh habits from forming or even being tried. 
Conclusion
This is a strange thing. Interleaving, as opposed to blocked learning, feels wrong, feels disjointed, almost chaotic. Yet is it much more effective. It seems to fly in the face of your intuitions. Yet it is significantly more efficient as a learning strategy. Yet how often do we see interleaving in classrooms, homework or online learning? Hardly ever. More worryingly, we’re so obsessed with ‘student’ evaluations and perceptions that we can’t see the wood for the trees. We demand student engagement not learning, encourage the idea that learning is easy when it is not. When it comes to teaching, we’re slow learners.

 Subscribe to RSS

Wednesday, November 28, 2018

Why almost everything we think about online learning may be wrong and what to do about it…

One thing that research in cognitive psychology has gifted to us over the last decade or so, is clear evidence that learners are delusional when it comes to judgements about their own learning. The big name in the field is Bjork, along with many other high quality researchers, who says that learning is “quite misunderstood (by learners)…. we have a flawed model of how we learn and remember”. There’s often a negative correlation between people’s judgements of their learning, what they think they have learnt and how they think they learn best - and what they’ve actually learnt and the way they can actually optimise their learning. In short; our own perceptions of learning are seriously delusional. This is why engagement, fun, learner surveys and happy sheets are such bad measures of what is actually learnt and the enemy of optimal learning strategies.
Desirable difficulty
Most learning is illusory as it is too easy. Learning requires Desirable (accomplishable) and difficult learning that requires real effort for high-retention to take place. This is why so much online learning fails. To simply click on faces and see speech bubbles of text, drag and drop labels, choose true or false, even multiple-choice questions, rarely constitutes desirable difficulty. This is click-through learning.
The solution is to provide effortful, retrieval. This means moving beyond the traditional model of text/graphics punctuated by multiple-choice, towards cognitive effort, namely retrieval through open input. This effortful learning gives significant increases in long-term retention and recall. Online learning needs to adopt these techniques if it is to remain credible.
Retrieval 
Retrieve and recall what you need to know Bjork (1975) results in much higher levels of retention. Rather than read, re-read and underline, look away and try to retrieve and recall what you need to know. Rather than click on True or False or an option in a short list (MCQ), look away, think, generate, recall and come up with the answer. The key point is that research has shown that retrieval is a memory modifier and makes your memory more recallable. Counter-intuitively, retrieval is much more powerful than being presented with the information. In other words it is more powerful than the original ‘teaching’ event.
Take a learning experience that you have probably been through many, many times – the airline safety demonstration. Try to think through what you have to do in the right order – find life jacket, put over head, then what….. not easy is it? Ah yes… inflate it through the blow tube… then there’s the whistle. No. Many choose the ‘inflate’ option but to inflate it inside the aircraft is a BIG no, no. In fact, airlines should set up a spot in the airport, where you actually sit down then have to DO the whole thing. Next time you sit there, watch, then afterwards, close your eyes and retrieve the process, step by step – that also works.
Roediger & Karpicke (2006) researched studying v retrieval testing (without feedback). One week later the retrieval tested group did much better. They also asked them how much you are likely to remember in one week’s time for each method – oddly, the majority of learners got it completely wrong.
Making errors is also a critical component of successful learning. According to Kornell, Hayes & Bjork (2009), generating the wrong thing, then getting it right, leads to stronger learning. The reason is that you are activating the brain’s semantic network. Retrieval testing does better than reading or watching, as it potentiates recall. So are unsuccessful tests better than presentations? The work by Kornell (2009) shows that even unsuccessful testing is better. Retrieval testing gives you better internal feedback and works even when you get few or no correct answers. Testing even before you have access to the material, as a learning experience, also helps learning. Once again, almost bizarrely, Heustler & Metcalfe (2012) asked learners what worked best and they were largely wrong.
From Gates (1917) who compared reading and re-reading with retrieval, to Sptzer (1939) who halted forgetting over 2 months with retrieval in 3000 learners, to Roediger (2011) who got a full grade increases with retrieval techniques and McDaniel (2011) who increased attainment in science, the evidence is clear. For a clear summary of this, and detail on the research, this excellent talk by Bjork is pretty good.
Online learning
In online learning the mechanics of this have also been researched. Duchaster & Nungester (1982) showed that although MCQs help you answer MCQs, they are poor in actual retention and recall. Kang (2007) showed that retrieval is superior to MCQs. At the really practical level, Jacoby (1978) showed that typing in retrieved learning was superior, as did MacDaniel (1986) and Hirsham & Bjork (1988) who showed that even typing in some missing letters sufficed. Richland (2005) did real world experiments that also proved efficacy.
We have the tools in Natural Language Processing and AI to do this, so technology has at last caught up with pedagogy. Let's not plough the same furrow we've plowed for the last 35 years. Time to move on.
Conclusion
I wrote, in a rather tongue in cheek manner (25 ways you which your e-learning sucks), about why I think most current e-learning is click-through and therefore low retention eye candy. This research shows that our methods of online learning are sub-optimal. The problem we face is that immediate success often means long-term failure. More focus should be given to retrieval, NOT presentation, clicking on items and multiple-choice. We need to be presented with desirable difficulties, through partial or complete open input. This is exactly what we’ve spent the last two years building with WildFire.

 Subscribe to RSS

Monday, November 26, 2018

Do we really need all of this 'mentoring' malarkey’?

I’ve never had a mentor. I don’t want a mentor. I don’t much like mentoring. I know this is swimming against the tide of liberal orthodoxy but I value liberal values more than I value fads, groupthink or orthodoxy. Don't mind people doing it but there are many reasons why I’m suspicious of mentoring.
1. Fictional constructs
Mentor was a character in Homer’s The Odyssey and it is often assumed that his role was one of a guiding, experienced guide for his son and family. This is wrong. Mentor was simply an old acquaintance, ill-qualified to play a protective role to his family, and worse, turned out to be a patsy for a hidden force, the God Athena. A similar tale has unfolded in recent times, with mentoring being revived on the back of late 19thcentury psychoanalytic theory, where the original theory has been abandoned but the practice upon which it is based survives.
There is another later work of fiction that resurrected the classical model as a source for the word ‘mentor’ in education, Fenelon’s Les Adventures de Telemaque(1699). This is a tale about limiting the excesses of a king and reinforced the presence of the word ‘mentor’ in both French, then English. Yet Mentor, in this ponderous novel, is prone to didactic speeches about how a king should rule (aided by the aristocracy), hardly the egalitarian text one would expect to spark a revolution in education. Interestingly, it pops up again as one of two books given to Emile in the novel of the same name, by Rousseau.
2. Psychoanalytic veneer
Mentoring came out of the psychoanalytic movement in education, through Freud and Rogers. Nothing survives of Freud’s theories on the mind, education, dreams, humour or anything else for that matter. But Rogers is different. His legacy is more pernicious, as his work has resulted in institutional practice that has hung around many decades after the core theories have been abandoned. We need to learn how to abandon practice when the theories are defunct.
3. Mentoring is a one person trap
As Homer actually showed, one person is not enough. To limit your path in work or life to one person is to be feeble when it comes to probability. Why choose one person (often that person is chosen for you) when there are lots of good people out there? It stands to reason that a range of advice on a range of diverse topics (surely work and life are diverse) needs a range of expertise. Spread your network, speak to a range and variety of people. Don’t get caught in one person’s spider’s web. Mentoring is this sense is a singular trap.
4.  People, social media, books etc. are better
You don’t need a single person, you need advice and expertise. That is also to be found in a range of resources. Sure, a range of people can do the job and the best write books. Books are cheap, so buy some of the best and get reading. You can do it where and when you want and they’re written by the world’s best, not just the person who happens to be chosen in your organisation or a local life coach. And if you yearn for that human face, try video – TED and YouTube – they’re free! I’d take a portion of the training budget and allow people to buy from a wide reading list, rather than institute expensive mentoring programmes. Then there's social media, a rich source of advice and guidance provided daily. This makes people more self-reliant, rather than being infantalised. Twitter also has strong benefits in CPD.
5. Absence of proof
Little (1990) warned us, on mentoring, that, “relative to the amount of pragmatic activity, the volume of empirical enquiry is small [and]... that rhetoric and action have outpaced both conceptual development and empirical warrant.”  This, I fear, is not unusual in the learning world. Where such research is conducted, the results are disappointing. Mentors are often seen as important learning resources in teacher education and in HE teaching development. Empirical research shows, however, that the potential is rarely realized, see Edwards and Protheroe (2003) and Boice (1992). The results often reveal low level "training" that simply instruct novices on the "correct" way to teach Handal and Lauvas (1988), Hart-Landsberg et al., (1992). Indeed, much mentoring has been found to be rather shallow and ineffective Edwards, (1998).
6. Fossilised practice
Practice gets amplified and proliferates through second-rate train the trainer and teacher training courses, pushing orthodoxies long after their sell-by, even retirement, date. Mentoring has sometimes become a lazy option and alternative for hard work, effort, real learning and reflection. By all means strive to acquire knowledge, skills and competences, but don’t imagine that any of this will come through mentoring in any efficient manner.
7. Over-formalised
Mentoring is what parents, grandparents and older members of the community used to do, and well. I’m all for the passing down of learning and wisdom, but when it gets formalized into specific people, with supposedly strong ‘mentoring’ skills, I have my doubts. By all means encourage people to share, especially those with experience but don’t kill the human side of this with an over-formalised process.
Conclusion: get a life, not a coach
I know that many of you will feel uncomforted by these arguments but work and life are not playthings. It is your life and career, so don’t for one minute imagine that the HR department has the solutions you need. Human resources is there to protect organisations from their employees, rarely either human or resourceful. Stay away from this stuff if you really want to remain an independent thinker.
Bibliography
English translation of Les Adventures de Telemaque https://archive.org/stream/adventuresoftele00fene#page/41/mode/thumb
Boice (1992)Lessons learned about mentoring https://onlinelibrary.wiley.com/doi/abs/10.1002/tl.37219925007
Edwards and Protheroe (2003)Learning to See in Classrooms: What are student teachers learning about teaching and learning while learning to teach in schools? British Educational research Journal.
Handal and Lauvas (1988) Promoting Reflective Teaching
Little, J.W. (1990) ‘The Mentor Phenomenon and the Social Organisation of Teaching’, in: Review of Research in Education. Washington D.C: American Educational Research Association.
Warhurst R (2003) Learning to lecture Paper presented at the British Educational Research Association Annual Conference, Heriot-Watt University, Edinburgh.http://www.leeds.ac.uk/educol/documents/00003330.htm

 Subscribe to RSS

Sunday, November 18, 2018

Why is learning so hard? Hyperbolic discounting – what is it and what to do about it

Julie Dirkson knows a thing or two about learning. Well versed in the research, she is especially good at bringing ‘behavioural psychology’ to the foreground. Understand learners and you understand why it is so difficult to get them to learn. So it was a pleasure seeing her speak and speak with her afterwards. 
Her starting point is the metaphor of the elephant and its rider, the rider the conscious, verbal, thinking brain; the elephant the automatic, emotional visceral brain. Academically this is Kahneman’s two systems, fast and slow, which she explains using the elephant and rider metaphor. It works, and is proof that you don’t have to wade through 400 pages of a quite dense book like Thinking Fast and Slow to understand a useful theory. (An alternative is to read the eminently readable story The Undoing Project by Michael Lewis.) 
Hyperbolic discounting
One cognitive bias that is hits learning hard is that of hyperbolic discounting, a well researched feature in behavioural economics. Take two similar rewards, humans prefer the one that arrives sooner rather than later. We are therefore said to discount the value of the later reward and this discount increases with the length of the delay.
If the consequences of our learning are distant, we are likely to take it less seriously. Smokers don’s stop smoking just because you tell them it’s dangerous, and there’s no greater danger than death! In practice, smokers see the consequence as being some time off, so they don’t stop smoking just because you warn them of the consequence. So it is with learning. Rewards feel distant in learning, which is why students tend to leave study and cram just prior to exams, or write essays on the last night. They are not committed when it is likely that they won’t use their newly acquired knowledge and skills for some time, if at all. No one would watch a printer problem video unless they had a printer problem.
So how do we get the learner to be a rider and not be stopped by the elephant?
Get social
Reframe learning into a more social experience, online or offline, so that learners have their peer group to compare with. If you see tat others are doing things on time, you are more likely to follow than be presented with some distant consequence. Future promises of promotion, even money, have less effect that near experiences of being part of a group doing things together or being encouraged, even peer reviewed, as encouragement and feedback engenders action.
Autonomous control
Give people control over their learning as personal agency acts as an accelerant. If I feel that things are not imposed upon me, but that I have chosen to take action, then intrinsic motivation will, on the whole, work better than extrinsic motivation. Giving people the choice over what and when they learn is therefore useful.
Push to engage
Technology allows us to push motivating messages and opportunities to learners. We can nudge them into learning. Nudge theory has been used in everything from insects in urinals to reduce splashes to serious behavioural change. Differ is a learning chatbot that raises learner engagement by nudging and pushing students forward through timely reminders. We know that learners are lazy and leave things to the last minute, so why not nudge them into correcting that behaviour. Woebot is a counselling chatbot that simply pops up in the morning on Facebook Messenger. You can choose to ignore or re-schedule. It has that drip-feed effect and, as the content is good and useful, you get used to doing just a few minutes every morning. LINK
Place in workflow
Just in time training, performance support and workflow are all terms for delivering learning when it is needed. This closes the gap between need and execution, thereby eliminating hyperbolic discounting, as there is no delay. Otto is a chatbot that sits in Slack, Messenger, Microsoft teams of whatever social or workflow system your organisation uses. It provides a natural language interface to learning, when you need it.
Use events as catalysts
A sense of immediacy can be created by events – a merger, reorganisation, new product, new leader. All of these can engender a sense of imminence. Or manufacture your own mini-event. Several companies have implemented ‘phishing’ training by sending fake phishing emails, seeing how people react and delivering the training on the back of that event.
Recommendations
Almost everything you do online – Google, Facebook, Twitter, Instagram, Amazon and Netflix, use recommendation engines to personalise what the system thinks you need next. Yet this is rarely used in learning, except in adaptive systems, where AI acts like a teacher, keeping you, personally on course. These systems are not easy to build but that do exist and is another example of AI in learning.
Visual nudges
Online learning needs to pick up on contemporary UX design and use slight movement, colour changes, positioning and layout to push people into action. In WildFire we use AI to create extra links during the learning experience. These appear as you work through an idea or concept, and are highlighted of the system thinks you didn’t really get it first time. But there’s lots of things you can do to nudge people forward in learning.
Calls to action
A neat combination of events as catalysts, nudge learning and calls to action, used widely in marketing, was a project by Standard Life. They used a merger with another large organisation as the catalyst, short 90 second videos as nudges and challenges(calls to action) to do something in their own teams as calls to action. Use was tracked and produced great results. Calls to action are foundational in marketing, especially online marketing, where you are encouraged to contact, registered, inquire or buy through a call or button. Have a look at Amazon, perhaps the most successful company in the world, built on the idea of calling to action.
Habit
Habitual learning is difficult to embed, but once adopted is a powerful motivator. good learners are in the habit of taking notes, always having a book in their bags, reading before going to sleep and so on. Choose your habit and force yourself to do it until it becomes natural, almost unthinking. In Kahneman language you must make sure that your System 2 has some of the features of what were once System 1. Or your elephant starts to get places on its own without the rider urging it along.
Conclusion
Learning is one thing, getting people to learn is another. Psychologically, we’re hard-wired to delay, procrastinate, not take learning seriously and see the rewards as far too far down the line to matter. We have to fight these traits and do what we can to encourage authentic and effortful learning, Make it seems as though it really does matter through all sorts of nudges; social, autonomy, push, place in workflow, events as catalysts, recommendations, visual nudges, recommendations, calls to action and habits.
Bibliography
Lewis, M. (2017) The Undoing Project. Penguin
Kahneman, D. (2011) Thinking fast and Slow.Penguin
Roediger, H. McDaniel M. (2013) Make It Stick. Harvard University Press

 Subscribe to RSS

Friday, November 02, 2018

World’s first hologram lecture (not really) but are they necessary?

Imperial College, London, claimed to have held the World’s first hologram lecture. What they haven’t mastered is the art of being honest or doing a modicum of research.  There have been hologram lectures before, most notably, by Stephen Hawking. But let’s put to one side the usual hyperbole by ‘Women in Tech’ and look at this critically.

For the Imperial event, 3D figures were beamed in live from the US. They are projected on to a glass screen, with a backdrop giving the illusion of depth of field and interact with other panellists and the audience. Nice idea but will it fly?
Before I start, I’m no fan of slabbing out academic lectures as a method of teaching and learning and would much rather institutions either, recorded lectures, or made sure that the people who deliver them receive some training on teaching and presentation. The number of students who simply don’t turn up is evidence enough that they are a failure. Only 60% turn up, even at Harvard. The very words ‘Lecture’ or ‘Lecturers’ says everything about the appalling state of pedagogy in Higher Education.
One of the saddest learning stories I’ve ever heard was from the actress Tilda Swinton. She was the only student who turned up to a lecture at Oxford by Raymond Williams where he read out his lecture, from notes, from behind the lectern, and neither of them even acknowledged each other. How sad is that? Almost every University has even worse tales of lectures where not one student turned up.

Samuel Johnson saw the folly of it all:
‘Lectures were once useful; but now, when all can read, and books are so numerous, lectures are unnecessary. If your attention fails, and you miss a part of a lecture, it is lost; you cannot go back as you do upon a book... People have nowadays got a strange opinion that everything should be taught by lectures. Now, I cannot see that lectures can do as much good as reading the books from which the lectures are taken. I know nothing that can be best taught by lectures, except where experiments are to be shown. You may teach chymistry by lectures. You might teach making shoes by lectures!’

As David Hume, observed, it is the content, not the person who matters:
‘...as you know there is nothing to be learnt from a Professor, which is not to be met with in books, and there is nothing to be required in order to reap all possible advantages from them, but an order and choice in reading them...I see no reason why we should either go to a University, more than to any other place, or ever trouble ourselves about the learning or capacity of the professor.’

On the other hand I have no problem with talks by experts (not adjuncts and/or PhD students) who command respect and give students a feel for what it is to be a physicist or psychologist. Lectures as inspirational and motivational events I have no problem with, where world class speakers and teachers do their thing – but few have that presence or possess those skills. The problem with a focus on just the technology here, is that a hologram of a bad lecturer won’t solve the problem.
This idea is interesting in terms of getting World Class experts and teachers to talk and teach in institutions around the world. If, as the evidence suggests, it increases presence, that’s great, especially if you feel as though they are really there. But I’m not convinced that hologram lectures are much more than a gimmick. They’re technically difficult to organise, expensive and try too hard to mimic what is, essentially, a flawed pedagogic technique. It perpetuates the traditional lecture format, rather than moving things on. It’s taking something that’s not that good in the real world and mirroring it virtually.

Skype or Zoom
On the other hand, we have to probe this a little? Wouldn’t it be easier and simpler to use video, either Skype or Zoom? These have tools that supplement the experience. For example, Skype translator can translate voice, in real-time, in 10 languages. Its text translator works in 60 languages. For global transmission, this makes sense.

Webinar software
Lectures as webinars make even more sense, as the supplementary tools allow for as much interaction as you wish and it is scalable. It is rather odd that educational institutions don’t make more use of this form of delivery. Questions, chat, polls, links and many other features are available in this type of software. It talks some skill to do this but it is a skill worth learning as it results in better teaching and more importantly, better learning.

VR
Full immersion gives the advantage of full immersion and full attention. One of the most compelling features of VR is the fact that you really are in that environment and the brain finds it difficult to jump out or be distracted. We know that attention is a necessary condition for learning, so this could be the optimal solution. One problem is the difficulty of taking notes, although speech dictation would be possible.

Conclusion
All technologies have their affordances. We need to identify what we want, then use the best technology available. Holograms seem like overreach. If students aren’t even turning up, let’s reflect on that first. If we’re not recording lectures, despite the overwhelming evidence that they are good for students being taught in their second language or those who are absent due to illness. In addition, learners can stop and rewind if they miss something, want to find something out or want to take more detailed notes. But the biggest argument is that they can be used for learning through repeated use and revision.

 Subscribe to RSS

Wednesday, October 31, 2018

Agile training at Jurys Inn – used AI to produce in minutes not months

So you walk into a hotel, you’re in the queue, and it’s taking ages to check-in. Or at breakfast, there’s no milk on the table…. it’s all those little things that matter in the hospitality trade. In the age of Tripadvisor, it matters. Jurys Inn take this seriously by training their staff to a set of defined, exacting standards.

Agile AI solution
This means agile training – relentless attention to training and the behaviour of customer facing staff. By agile they mean the use of AI to create training quickly, that really does deliver high retention training to front-line staff on any device. WildFire, an AI online content creation service, quickly turned a set of 209 standards, into 6 online learning modules on: 
  • General standards
  • Front office
  • Breakfast
  • Restaurant
  • Bar
  • Room service 
Each module focused on essential knowledge - what staff need to know.

Agile learning
Every module has an AI generated audio introduction (text to speech), that explains what you have to do and to relax about getting things wrong, as it is OK to make mistakes while learning. Staff then read the standards and, rather than click on multiple-choice questions, have to bring to mind and type in the behavioural standards. This ‘effortful’ learning was inspired by recent research in learning that shows recall, with open input, is superior for retention to simply recognising answers from a list. 

Agile production
The six modules were literally created in a day, as the AI created the content, by identifying the relevant learning points and automatically creating the learning experiences. This superfast production process meant that quality assurance could be done on the real modules, without the need for design documents and scripts. There was no need for multiple iterations by SMEs, as the original document contained all the necessary knowledge. The look and feel, logo, images, palette numbers for screen features were quickly agreed. Everything from brief to final delivery was done through Skype. Not a single face-to-face meeting was necessary.

Agile data
Finally, the modules were SCORM wrapped for delivery on the LMS – Totara. As SCORM is rather limited, we also embedded extra data gathering capability which WildFire harvests for further analysis. This allows detailed analysis of who did what when and within the training, the specific times taken by each individual. Beyond this. WildFire have been looking at data and correlating it with other data on sales.

Agile preparation
One important lesson in being agile is the ‘Garbage IN: Garbage OUT’ rule. When you use an agile production process, you need agile preparation. An intensive look at the input material pays dividends. Eliminate all of that extraneous material and text, cut until it bleeds and cut again, catch those pesky spelling and punctuation errors, make sure things are consistent. In our Jurys Inn and TUI work the source material was edited down to the essential ‘need to know’ content.

Agile project management
Another essential ingredient is an agile project manager. In both the Jury's Inn and TUI projects, we spoke frequently, but didn’t have a single face-to-face meeting. It was all quick decisions, problems solvedon the spot and process change to get things done. The project managers never saw problems, only issues to be solved – quickly.
It was made easier by the fact that AI was used to produce content in minutes not months. That means the quality control was on real content, not paper documents. And as we use approved documents, PowerPoints and videos, there was no real need for intensive SME input, which is the main brake on agile  production.

Agile production
This is where the real gains lie. AI is now being used to create content and add curated resourced at the click of a button. WildFire will take any document, PowerPoint or video and turn it into high retention online learning, in minutes not months. Want an audio podcast or audio introduction to the course?  It takes seconds using AI to do text to speech. The time savings are enormous, as well as costs. AI is used not only to identify learning points, it also constructs the questions, assesses open input (words or full short answer) and locates external resources for further learning.

Agile promotion
I’ve lost count of the number of times we’ve bust a gasket getting something finished, only to see it languish on a server before release. Agile marketing should also be part of our mindset. This means blogs like this and good internal (and external if required`) marketing. Managers and learners are intrigued by the use of AI in learning – use that to best advantage.

Conclusion
Agile is as much a state of mind as process. Yet Learning and Development still works to an old model of months not minutes. We procure slowly, prepare slowly, produce slowly and deliver slowly. Despite relentless calls to align with the business and respond to business needs, we are too slow. L and D needs things in days, yet we deliver in months. This is what needs to change.
Online learning has traditionally been a rather slow in design and production. We can now use AI in WildFire to create content quickly, to produce agile learning and data that allows us to adapt to new circumstances. I may have used the word 'Agile' too often here but it captures, in a word, what is now necessary. The days of seeing online learning production as some sort of feature film project with matching budgets and timescales – months, should be re-examined. Sure some high-end content may need this approach but much can be automated and done at 10% of the cost, in minutes not months.

 Subscribe to RSS

Monday, October 22, 2018

Agile online learning in 'impossible timescales' saves £453,000 with 36% increase in sales


This Award winning, agile, learning project was groundbreaking (I know - bit of a cliche). We used an AI tool, WildFire, to deliver a project to a large multinational (TUI).  The project delivered 138 modules on the locations for its holidays, flights, airport codes and so on. Recognising that they could never produce content on this scale, as the estimated costs for external development were just under £500,000, and it had to be delivered in weeks not the estimated 8 months, they opted for WildFire. This uses AI to create content in minutes not months, along with supplementary curated content, also selected by AI.

Agile production
As WildFire does most of the design and build, an agile approach to production and project management was intrinsic. No scripts were necessary, as the software built the content so quickly that quality assurance could be done on the actual modules.
As it was ‘agile’, let’s cut to the quick...
  • 95% rated the design and approach as good or very good
  • 62% confirmed they could identify a specific sale based on knowledge gained
Their knowledge of the countries, locations, attractions, currencies, airport codes and so on was reported, time and time again, by front-line staff as having helped them sell holidays and flights. Remember - this is a location-driven business. If you want to sell holidays and cruises, you have to know the destinations and attractions.

ROI
Total savings, compared to traditional online learning production, were calculated as: 
  • “£438,000 with extra savings of £15,000 in salary costs”
  • “Freed up 15% of manager time” to do other things"
  • “With a bit of lateral thinking and a lot of tenacity – seemingly impossible timescales were met”
Further benefits
When the modules were released to the wider business, sales benefits started to emerge:
  • 36% increase in sales has already been recognised in the first few months the training has been available”
Within the company the “wider business now recognises the benefits of being bold with new learning technologies” and sees the project as an “outstanding example of achieving our strategy to invest in and develop our people”.

Agile mindset
This was a ground-breaking project, delivered without a single face-to-face meeting. It shows what can be achieved when a training department is innovative and brave. We must get past the model that says it takes months to produce content at prohibitive costs. Agile production needs agile tools, agile production methods and agility of thought and mindset. Organisations have traditionally moved faster than training delivery. That means we’re often out of phase with the business. An agile mindset and production attitude allows us to transcend that historical gap. Once we become responsive to the business they will respect us more and we are more aligned with their natural speed.

Conclusion
At the learning Technologies awards, the judges said the following....
"To speed up production TUI took the brave step of selecting.... an Artificial Intelligence tool, WildFire. The result saw triple savings; six months knocked off the expected timescales, £15,000 in salaries and £438,000 of the development budget."

If you want to know more about WildFire, agile production and high-retention, online learning, contact us here...







 Subscribe to RSS

Has L&D been hijacked by ‘identity’ politics?

I got asked an interesting question last week on a panel at an Learning and Development conference. Where is L and D? Is it advancing or regressing? Interesting question.
In many ways it has advanced by embracing technology. Few individuals or organisations of any scale ignore online solutions to deliver learning and development. There has also been some movement towards taking research more seriously. I meet far fewer people who believe in learning styles or NLP. There seems to be much more awareness of learning theory and by that I mean cognitive science.
On the other hand, these advances have been matched by a serious reversal - the swallowing, lock, stock and both barrels of ‘identity’ politics. Gone are the days when HR and L and D were at the forefront of personal development. Much ‘training’ is now targeted at protecting the organisation from their own employees. Employees are now subjected to a tsunami of compliance and identity training . Much of what is actually delivered via an LMS is actually protective training, or as almost everyone says, ‘tick-box’ training. We know it and learners know it. We have to be honest about the fact that they are often openly contemptuous about this form of ‘training’. Somewhere along the line our development agenda got hijacked.
Identity and training
Training that establishes difference also implies exclusion. Fixing favoured identity groups, seeing some as oppressed and others oppressors is a destructive force. And when HR aggressively promotes and ‘us’ and ‘them’ culture it does us all a disservice. When we become the managers and administrators of difference, serving up a never-ending diet of identity and diversity training courses, we swirl around in our own echo-chambers. We become the seekers out of ‘wrong-think’, policing ordinary people, as if they were stained by original sins. It results, not in rational consensus within an organisation, but the false exaggeration of difference.
Leadership
One specific identity group that has been put on a pedestal is ‘Leaders’. We have thrown resources at Leaders and Leadership training, as elitist an approach to employee engagement as I’ve seen in my 35 years in this business. In a desperate attempt to appear legitimate we have turned ‘elitism’ into a sort of training cult. No one would seriously call themselves a ‘Leader’ without being ridiculed. This form of single  group identity is a serious distortion of real needs in organisations.
Values
Another facet of identity politics has been the unedifying sight of organisations forcing faux values down the throats of their employees. I have values and have no interest in HR telling me what my values should be. Strangely alliterative – all starting with ‘I’ or “C’ they are abstract nouns that bear no relation to the real world or the workplace. Worse still are those values that fit some acronym, where at least some of the values have been made up to fit that word. They want to shape your identity by imposing their values on you. They are, of course, largely ignored or treated with contempt. Few can recall them, fewer still care.
Diversity
The rot set in long ago, when HR thought it should take the role of therapeutic diagnosis. Myers Briggs, a flawed and crude tool, has been used to determine people’s lives. As if this weren’t enough, a slew of interrogative techniques, from learning styles (a fiction) to NLP and mindfulness, were employed to caricature, categorise and, in some cases, condemn ordinary people. There has been no end to this slicing and dicing, based on dubious diagnostic techniques. The evidence is clear. Diversity training does not work.
Unconscious bias
It has now reached surreal levels of interrogative nonsense, with courses on ‘unconscious bias’. Not satisfied with superficial courses making us consciously compliant, HR suddenly started to probe our unconscious. How did that happen? What on earth gives anyone in HR or L&D the right to even think that my unconscious is open to their investigation, something to be probed by some half-baked questionnaire that has no scientific validity? This is completely out of control. The assumption is that certain out-of-favour groups - white, male, working class - are guilty by virtue of being born, so leaden with bias that they need to be re-educated. It’s a pathological view of human nature, where minds must be interrogated and forced to admit their guilt. This was never our goal. HR and L&D is full of good people but it has been hijacked by a form of policing that has abandoned personal development for personal identity.
Conclusion
All of this is fuelled by so called ‘experts’ who design courses on ‘X’, train others to deliver those courses as ‘practitioners’ who sell their courses for ‘$Y’ so the Ponzi schemes begin. Rather than focus on the real needs of organisations, real knowledge, skills and competences, we have been sucked into a world of exaggerations, abstract concepts and fictions, where people have to be identified, shamed and re-educated, to be ‘correct’ and ‘compliant’. Time and time again I hear pleas for HR and L&D to be more business focussed. What we’ve done, in reality, is turn our back on real business issues, to focus on therapeutic concerns, that invade people’s privacy. So let’s get off the identity bandwagon and get back to business.

 Subscribe to RSS

Friday, October 19, 2018

"On average, humans have one testicle”... that's why most data analytics projects in education are misleading....

AI will have a huge impact in learning. It changes everything… why we learn, what we learn and how we learn. Of course, AI is not one thing. It is a huge array of mathematical and software techniques. Yet looking at the spend in education and training, people have been drawn to one very narrow area – data analytics. This, I think, is a mistake.
Much of this so-called use of AI is like going over top of head with your right hand to scratch your left ear. Complex algorithmic and machine learning approaches are likely to be more expensive and far less reliable and verifiable than simple measures like using a spreadsheet or making what little data you have available, in a visualized, digestible form to faculty or managers. Beyond this, traditional statistics is likely to prove more fruitful. Data analytics has taken on the allure of AI, yet many of it is actually plain, old statistics.
Data problems
Data is actually a huge problem here. They say that data is the new oil, more likely the new snakeoil. It is stored in weird ways and places, often old, useless, messy, embarrassing, personaland even secret. It may need to be anonymised, training sets identified and subject to GDPR. To quote that old malapropism, ‘data is a minefield of information’. It may even be massively misleading, as in the testicle example.
Paucity of data
In the learning world, the data problem is even worse as there is another problem – the paucity of data. Institutions are not gushing wells of data. Universities, for example, don’t even know how many students turn up for lectures. I can tell you that the actual data, when collected, paints a picture of catastrophic absence. Data on students is paltry. The main problem with the use of data in learning, is that we have so little of the stuff. 
SCORM, which has been around for 20 plus years literally stopped the collection of data with its focus on completion. This makes most data analytics projects next to useless. The data can be best handled in a spreadsheet. It is certainly not as large, clean and relevant, as it needs to be to produce genuine insights.
Other data sources are similarly flawed, as there's little in the way of fine-grained data about actual performance. It's small data sets, often messy, poorly structured and not understood.
Data dumps
Data is often not as clean as you think it is, with much of it in:
·      odd data structures
·      odd formats/encrypted
·      different databases
Just getting a hold of the stuff is difficult.
Defunct data
Then there’s the problem of relevance and utility, as much of it is:
·      old
·      useless
·      messy
In fact, much of it could be deleted. We have so much of the stuff because we simply haven’t known what to do with it, don’t clean it and don’t know how to manage it.
Difficult data
There are also problems around data that can be:
·      embarrassing
·      secret
There may be very good reasons for not opening up historic data, such as emails and internal social communications. It may open up a sizeable legal and other HR risks for organisations. Think Wikileaks email dumps. Data is not like a barrel of oil, more like a can of worms. 
Different types of data
Once cleaned, one can see that there are many different types of data. Unlike oil, it has not so much fractions as different categories of data. In learning we can have ‘Personal’ data, provided by the person or actions performed by that person with their full knowledge. This may be gender, age, educational background, needs, stated goals and so on. Then there’s ‘Observed’ data from the actions of the user, their routes, clicks, pauses, choices and answers. You also have ‘Derived’ data inferred from existing data to create new data and higher level ‘Analytic’ data from statistical and probability techniques related to that individual. Data may also be created on the fly.
Just when you thought it was getting clearer. You also have ‘Anonymised’ data, a bit like oil of an unknown origin. It is clean of any attributes that may relate it to specific individuals. This is rather difficult to achieve as there are often techniques to back engineer attribution to individuals.
In AI there’s also ‘Training’ data used for training AI systems and ‘Production’ data which the system actually uses when it is launched in the real world. This is not trivial. Given the problems stated above, it is not easy to get a suitable data set, which is clean and reliable for training. Then, when you launch the service or product, the new data may be subject to all sorts of unforeseen problems not uncovered in the training process. This is a rock on which many AI projects founder.
Data preparation
Before entering these data analytics projects, ask yourself some serious questions about 'data’. Data size by itself, is overrated, but size still matters, whether n = tens, hundreds, thousands, millions, the Law of Small Numbers still matters. Don’t jump until you are clear about how much relevant and useful data you have, where it is, how clean it is and in what databases.
New types of data may be more fruitful than legacy data. In learning this could be dwell time on questions, open input data, wrong answers to questions and so on. More often than not, what you have as data are really proxies. 
Action not analytics
The problem with spending all of your money on diagnosis, especially when the diagnosis is an obvious limited set of possible causes, that were probably already known, is that the money is usually better spent on treatment. Look at improving student support, teaching and learning, not dodgy diagnosis.
In practice, even when those amazing (or not so amazing) insights come through, what do institutions actually do? Do they record lectures because students with English as a foreign language find some lecturers difficult and the psychology of learning screams at us to let students have repeated access to resources? Do they tackle the issue of poor teaching by specific lecturers? Do they question the use of lectures? Do they radically reduce response times on feedback to students? Do they drop the essay as a lazy and monolithic form of assessment? Or do they waffle on about improving the ‘student experience’ where nothing much changes?
Conclusion
I work in AI in learning, have an AI learning company, invest in AI EdTech companies, am on the board of two other AI learning companies, speak on the subject all over the world, write constantly on the subject . You’d expect me to be a big fan of data analytics and recommendation engines - I’m not. Not yet. I’d never say never but so much of this seems like playing around with the problem, rather than facing up to solving the problem. That's not to say you should ignore its uses - just don't get sucked into data analytics projects in learning that promise lots but deliver little. Far better to focus on the use of data in adaptive learning or small scale teaching and learning projects where relatively small amounts of data can be put to good use.
AI is many things and a far better use of AI in learning, is, in my opinion, to improve teaching through engagement, support, personalised, adaptive learning, better feedback, student support, active learning, content creation (WildFire) and assessment. All of these are available right now. They address the REAL problem – teaching and learning. 

 Subscribe to RSS

Tuesday, October 16, 2018

Nudge learning

Things move fast in organisations and when Standard Life merged with Aberdeen Asset Management, an agile learning approach to changing behaviour in the new organisation was implemented, a training intervention that is itself agile and resulted in actual behavioural change. A huge traditional course, whether face-to-face or online, based on a diet of knowledge would have been counterproductive in this fast moving, post-merger commercial environment and be seen as a bit old-school and non-agile. Whereas a series of short, sharp interventions that nudge people into applying agile in their own context and work environment was likely to work better. At least, that's what Peter Yarrow, Head of Learning, thought – and I think he's right. It was his brainchild.
His successful project used the 'nudge' technique. Nudge theory recommends small interventions to push people into changing behaviour. Famous examples include the image of a fly in men’s urinals, to improve aim and reduce cleaning costs! Opting out, rather than into organ donation is another. The psychological theory is laid out in the book 'Nudge: Improving Decisions About Health, Wealth and Happiness’ by Thaler and Sunstein. They could well have added ‘Learning’ to the title.
Nudge solution
In learning, Standard Life Aberdeen sent small, professionally shot videos (on average 1min 30secs long), mainly talking heads from leaders and experts in the organisation, out via email. In addition to the video, there was a ‘challenge’ to apply the lesson in their own working environment. I like this approach, and it is a truly fresh and agile 'nudge'intervention.In their case it was general management techniques but I feel that agile techniques could be applied in response to all sorts of needs. Each starts with a proposition, or problem, followed by a suggested solution and finally, and crucially, a call to action. This is based on techniques also used in web and online design.
Example 1
Video on importance of comms
It’s hard to be a high performing team if colleagues don’t know each other well. Without trust, mutual respect and goodwill, performance will most likely remain middle of the road. Exceptional performance is fuelled by positive working relationships. Take this week’s challenge to get to know your colleagues better.
Example 2
Video on mentoring
Being mentored is a great way to develop and progress. But how do you get started? Begin by identifying someone you trust who has taken a career path you aspire to. Take this week’s challenge to learn more about making mentoring relation ships work.
These videos and challenges were sent out by email and usage tracked. The take-up across the organisation surprised the training department and the feedback was very positive. People felt that it was integrated into their natural workflow (they were not too long and intrusive) and that it was made more relevant by virtue of nudging people towards action by them as individuals in their specific job.
Suggestions for nudge learning
Great start but rather than batch emails, I'd use an algorithm to decide personal needs and, take data from usage and get more precise in timing and targeting. This means harvesting more data, which one can do, even with internal email systems. People get habituated out of responding if they get too many emails.
On the challenges I’d use more of a pure marketing approach, a strong command verb at the start, really concise, with reason and emotional pull. Give your audience a reason why they should take the desired action, maybe a bit of FOMO (Fear Of Missing Out), maybe some compelling numbers. Writing calls to action is both a science and an art and it’s worth being a little creative.
I'd also do slightly more than just state the challenge. I'd get the user to do something there and then to make sure they got the main points in the video (we've done this by grabbing the transcript and getting the user to check they've understood the main points) using AI generated, open-input experiences with WildFire.
None of this is a criticism of Peter’s pioneering project, merely suggestions to make it more potent.
Conclusion
I really liked Peter’s fresh thinking around the ‘nudge’ thing. It has legs and could go in all sorts of directions. It is the combination of proven marketing techniques with learning that make this approach fly. Few in marketing want to slab out hours and hours of content – they think first audience, second channels and third action. Their whole way of thinking is around ‘less is more’. This also happens to be exactly what the psychology of learning tells us about learning experiences. The limits of working memory, cognitive overload, forgetting and the need for transfer mean doing less but doing it better. 

 Subscribe to RSS