MOOC physics students outperform campus students

Prof Dave Pritchard

This guy teaches both a campus and MOOC course on physics. He is no amateur, a world-class researcher in atomic physics, winner of four major research prizes and three students who won Nobel prizes. When he compared their learning gains, he was astonished at the results. “I had hoped that because they (campus cohort) went to our classes, they would learn a lot more”.

Given the many positive announcements from vendors and negative reactions from academics about MOOCs, there is a remarkable lack of research on learning outcomes. Why this lack of interest? Well one feature of academic research is that it rarely looks at itself, for fear of what it will find. The lecture is the stand-out example. Despite many decades of research showing that lectures, and hterefore the professional job title ‘lecturers’, make little pedagogic sense, most academics will dive into the trenches of irrational indignity in their defence. But reson will prevail and on MOOCs we are now seeing some wonderful research by the likes of MIT and the University of Edinburgh.

Standout study

One standout study by MIT compares campus with MOOC students on the same course. It tackles the key issue; do students learn using MOOCs and how does this compare to the much more expensive campus course? If it is true that students learn more or as much on a MOOC, then the cost differential is such that it makes absolute sense to use MOOCs for these courses. This is a solution to the ballooning costs in HE. Even of the students learn less, there is still a strong argument for using MOOCs in terms of costs. Far too many researchers ignore this key issue – the massive reduction in cost.

Enter this study on Physics. It is a introductory, review course on mechanics with an emphasis on strategic problem solving. It starts with straight line motion through to momentum, mechanical energy, rotational motion, angular momentum and harmonic oscillation. The interesting addition is the content on problem solving that requires several different concepts and approaches leading to a single solution.

1000 students who completed the MOOC (Certificates of Completion) were compared with their similarly scored counterparts on the campus course. The relative progress across all groups was roughly equal across all groups. Identical pre- and post-tests were given to assess learning gains.

Results

What surprised the researchers was that the online students did as well as the campus students, and this is the interesting part – regardless of previous academic experience, whether they had a PhD, Masters, Bachelors or high-school diploma. In fact, the MOOC generated “slightly more conceptual learning than a conventionally taught on-campus course”. But there was a further piece of analysis that was even more surprising. When comparing MOOC students with MIT campus students , even though the campus students had received a lot of extra input and instruction, the relative results were the same with “no evidence of positive, weekly relative improvement of our on-campus students compared with our online students”.

Conclusion

This is the only study of its type that I know but would be pleased to hear about more if they focus on learning outcomes. At last we are beginning to see some sensible research that cuts through both the hype and defensive posturing. Good, level-headed academics and institutions are doing what should have been done years ago – researched the learning and cost outcomes. The researchers are now going on to look at what caused the learning. This is good work and long overdue.

Spaced-practice: FREE White Paper & 10 practical ways to make it happen

I've written a White Paper, available here, on spaced practice that includes the theoretical basis, along with informal, formal and technological solutions. Lots of detail, wth full citations but here's some simple techniaues for implementing spaced-practice without too much effort.
Spaced practice, despite being well known since Ebbinghaus first suggested it as a solution to his forgetting curve, in 1885, remains a rarely practiced technique in learning. The reasons are obvious enough. Most education and training delivers isolated doses of learning, lectures, presentations, classroom courses and the learners walk out of the door at the end, job supposedly done. Teachers had no real way of getting to them after the event had finished.

Spaced practice needs to be habitual. John Locke and William James both emphasised the key role that ‘habit’ plays in learning, lessons we’ve ignored.

Good learners, in my experience have developed good learning habits. They always have something to read in their pocket or bag. They tend to be obsessive note takers, often with a long series of filled notebooks. They habitually elaborate what they hear and actively try to remember. They replay and recall in their own minds, through dialogue and re-reading their notes. They also tend to kick start new learning habits, such as blogging, using Evernote and so on.

So how can we make spaced practice habitual?

1. Top & tail

As a teacher, if you deliver a series of lectures, classes, modules, whatever, the simple practice of summarising what was taught in the last lecture, period, class or event and doing the same at the end of the lecture/class/event, gives two reinforcement events for the price of one. There’s a double dividend in that you take advantage of primacy and recency (also discovered by Ebbinhaus), the fact that learners tend to remember the first and last things more than what comes in between.

2. Note taking
As a learner, get into the habit, not only of taking notes, but rereading and rewriting those notes. Wittrick and Alesandrini (1990) found that written summaries increased learning by 30% through summaries and 22% using written analogies, compared to the control group. If you take notes AND review them, you do better on assessments (Kiewra 1989, 1991). Interestingly, Peper and Mayer (1978) found that note taking increased skills transfer and problem solving in computer programming and science(1986). Shrager and Mayer (1989) found similar effects in college students, learning about cameras. It would seem that note taking allows learners to relate knowledge to experience. 

3. Places

Planes, trains and automobiles, as well as airports and any other place where you find yourself hanging around - opportuntiy for a bit of recall. There are other things in life you do regularly, like eat, go to the toilet, leave the house and so on. 

4. Email

If all of your learners use email then this is an easy and efficient way to deliver spaced practice events.  Group emails, set up and timed for release, can get whatever reinforcement event you wish to design straight to your audience. A simple text email, infographic, question, video, even piece of e-learning; anything that makes them rethink, will help fix the learning in long-term memory.

5. Blog
Bloggers regularly report the learning value of writing and crystalising teir thoughts in blog posts. This, in itself, involved deep processing, therefore better encoding, retention and recall. Blogging is, in this sense, a massively effective way to reinforce learning. It’s one of those things that, when it becomes habitual, is massively effective as an aide memoire.
6. Twitter & Facebook

Given the fact that 1.5 billion people are on Facebook there’s a good chance that your learners are easier to reach on Facebook than they are in your institution, library or any other physical space. The notifications system on Facebook is superbly efficient and that little red circle with a number in it is a strong stimulus for attention. Simply message your students with a series of cues from the lecture or course.

7. Sleep

One of the most effective methods of habitually delivering spaced practice is to encourage learners to get into the habit of a little practice and recall just before they go to sleep. This takes discipline but studies show that it is very effective as the brain appears to consolidate memory during sleep.

8. Exercise

If you exercise regularly, that is the chance to recall and reinforce whatever you want to retain. A podcast through your headphones in the gym? Simply record your own lists, notes, reinforcement events and replay on demand. Get into the habit and you’ll get both physical and psychological gains

9. Mobile

Systems, like ENCORE, deliver reinforcement events, spaced between any two times to your learners’ mobiles. You can choose what to deliver when using a variety of media. This system gets to that powerful, personal, portable device in the pocket of every learner.
10. Adaptive spaced-practice
Sytems are emrging with personalised spaced-practice that is delivered according to your needs, paying attention to your preferred channels, intereleaving known and less well known items for practice, avoiding cognitive overload when you miss some and so on.

Conclusion

Spaced practice is arguably the most powerful, yet most overlooked benefit in learning. Implemented properly and it is possible to have huge gains in productivity, namely the retention and recall of whatever has been learnt. I’d go further and say that if you don’t have a spaced practice strategy, you don’t have a properly designed course.

So let’s have that list again….

To & tail

Notes

Places

Email
Blog

Twitter & Facebook

Sleep

Exercises

Mobile
More in this series...

10 ways to make badass INTROs in online learning 

10 bloody good reasons for using much-maligned TEXT in online learning 

10 essential online learning WRITING TIPS in online learning 

10 stupid mistakes in design of MULTIPLE CHOICE questions

10 essential points on use of (recall not recognition) OPEN RESPONSE questions

10 rules on how to create great GRAPHICS in online learning ‪

10 sound pieces of advice on use of AUDIO in onlinelearning 

10 ways based on research to use VIDEO in online learning

10 ideas on use of much maligned TALKING HEAD videos in online learning

Jeff Haywood: stunning analysis of the future of online in HE (MOOCs etc)

When it comes to online learning, the University of Edinburgh, has become the most active and interesting HE institution in the UK. This is something of a surprise, after its disastrous performance in student surveys on teaching, perhaps even, a response to this problem. They do, however, have one great advantage, Jeff Haywood, who has led the charge towards the sort of experimentation and strategic, economic thinking that put the rest to shame.

Haywood keeps an eye on the data emerging from online activity in HE and points towards a steady rise in its use and acceptance by students and faculty (one third have taught an online class, 71% used OER). He points towards an increasing portion of younger people taking MOOCs with a rise towards 1 in 5 students taking online courses. There is a growing recognition that online may even be as good, if not, at times, better that the traditional campus course

.

Students not staff

In his recent ALT talk, Haywood gave a masterful talk on how to think strategically about online learning in HE. I have written about their bold move with six Coursera MOOCs, along with their excellent publication of the resulting data. This is only one strand of a strategy that looks towards 2025. It is a strategy that takes the business case, the economics, seriously.

Where he is way ahead of most commentators in he is that 90% of the tech is used by students not staff. The focus on staff is, to his mind a bit myopic and a bit of an EU obsession. This movement is “student oriented rather than institution oriented”.

MOOCs

“Love or hate them MOOCs have forced open a debate at policy level on digital education”.  Spot on. David Willetts, with Martin Bean, took the bull by the horns and kick started Futurelearn. There is no doubt that this action has stimulated both debate and action in HE. Unfortunately, he has been sacked, and there is no sign that the Shadow Minister for Education Tristram Hunt, is anywhere near as interested.

Unlike many in academe, Haywood speaks with authority based on real and substantial experience, with substantial data, when he makes the important statement that MOOCs “touch learners much more than you might think”. As an aside he also mentions that it has forced Universities to up their game on marketing. We have learnt he says, that “courses can be run at surprisingly large scale” and that “charismatic teachers can reach learners”. Accelerated innovation has also been seen with “a range of technological innovations doing things at scale – peer assessment, comparative judgement online etc.”.

Scale

One word matters above all in this analysis “scale”. Haywood is right to place it at the centre of his analysis. Sure we know that faculty have low digital skills and that there are low rewards for teaching but we must get to scale and move beyond the “bijou and niche”

Where Edinburgh and Haywood have the intellectual upper hand is in their honest appraisal of the economic implications of online learning. In my view the majority of academics are stuck in an analysis that focuses only on quality, ignoring the real issue of cost. Seb Schmoller and others have been arguing for some time, with estimated figures for cost per student, that online, while it may not match the quality, is so much cheaper, that even a weak result makes it worthwhile. I’d argue that the quality issue is fast disappearing, with degree delivery by the likes of IDI. His vision looks to on demand, self-paced, location flexible, relevant to your future, global & local, personalised, affordable, value-added learning; “Without technology this is undoable”.

Walks the walk

At national and international level, policy level discussion is needed with a road map that has clear steps over the next ten years to 2025. This is also true within your institution. If not planned it will not happen. Without investment it will not happen. Without agility it will not happen. This is why “MOOCs and the children of MOOCs are so important”. But that’s just a fraction of the story.  Edinburgh’s MOOCs have racked up huge numbers, with 800k learners on 15 MOOCs and another 15 in the pipeline. Edinburgh has 30k students on campus but also with 50 odd fully online Masters Degrees. (2500 students). It’s a mixed strategy.

Vision by 2020

By 2020 they want 40k students with “all students taking one full online course” and all faculty and teaching staff will have some experience of teaching online. They have the ambition to try to reach 10s of millions of learners through increased online Masters degrees, OER and MOOCs. The means to the end is a series of real funded experiments and pilots, which are all potentially scalable.

Haywood is optimistic and thinks that he is swimming with the tide. The technology has matured, interest risen among learners and policy discussions are far more outcome oriented. One wit in the audience thought that CAVE dwellers (Colleagues Against Virtually Everything), were his biggest problem. Haywood thought that MOOCs had been useful in that those Universities that had taken this leap have found that MOOCs encourage faculty to come forward, as they know they will get support. He added that employers are clearly interested in MOOCs. In Scotland SMEs are interested. For Haywood this is about “opening up the boundaries of space and time – as campus education is limited on both”. He sees nothing wrong with pro bono working education with the secondary aim of recruiting students and charged services coming through.

Diana Laurrilard made a point she often makes and it is pertinent, that Universities have never really understood the cost of teaching. This is true, they don’t even know what is being delivered and to whom. Unfortunately, she has been on the warpath against MOOCs, but only on a straw man basis. She doesn’t believe that MOOCs will entirely replace current HE model. That’s fine, neither do most MOOC providers, including Haywood. Haywood’s response to her question on quality was entirely right. Sure, the tough part is supporting and nurturing students through their personal intellectual development but the answer may be in the middle way. We know that lectures can easily scale so what about the varying degrees of personalised support (something grossly exaggerated in HE). He thinks that technology is already providing solutions, allowing portions of courses to be run on their own, without tutor intervention. Haywood is keen to use intelligent technology at the kind of numbers we run on our MOOCs. He, unlike Laurrilardian sceptics, know a good deal more about the technology, such as adaptive learning, and rightly look for economies of scale, before making rash announcements.

Leaps

The sort of leaps he sees in technology, that allow you to step back and let parts of course run on scale are being looked at, especially in their expanding masters programme. First, there’s online assessment, the Achilles’s heel in HE. The lecture model completely negates sensible formative assessment and the long-form essay, with slow, and often amateurish feedback, seems incredibly dated.

Adaptive learning

He is spot on in looking at learning analytics, especially adaptive learning, for scaling right across the institutions. This is the one technology that already offers hope in tackling the hard to deliver ‘tutor’ functions, pushing courses towards competence-based learning, where learners get personalised learning delivered at their own pace. This is already happening in the US, with considerable investment by the Gates Foundation and a rack of other institutions. Haywood and Edinburgh are the first UK institution to pick up on this and take it seriously. He is being true to his word in retaining some small group pedagogy where you need it but always looking for economies of scale.

Conclusion

Information technology has been extremely consequential in HE over the last 25 years but principally in ‘output enhancing’ ways that do not show up in the usual measures of either productivity or cost per student. Stanford 2012 Tanner Lecture

This is a great quote and recognises that technology has enhanced what we do but the economic are rarely understood. Haywood wants to find ways to use technologies to increase the throughput of students, move through curriculum at their own speed, and automate some parts of the curriculum where you can scale. We need to address the question of increasing productivity without decline in quality. If not we’re just polishing the current system without addressing access on scale.

At last we have a University and academic who is sophisticated enough to think big, be thoroughly strategic, agile, consider the economies of scale with an impressive focus on productivity and costs. One of the great failures in HE planning is a serious attempt to consider actual costs, to be specific, as Lewin and Blefield recommend on detailed cost effectiveness analysis. He also has a vision of the future University, not just being more “open on the boundaries“ but more productive and efficient.