MOOC physics students outperform campus students

Prof Dave Pritchard

This guy teaches both a campus and MOOC course on physics. He is no amateur, a world-class researcher in atomic physics, winner of four major research prizes and three students who won Nobel prizes. When he compared their learning gains, he was astonished at the results. “I had hoped that because they (campus cohort) went to our classes, they would learn a lot more”.

Given the many positive announcements from vendors and negative reactions from academics about MOOCs, there is a remarkable lack of research on learning outcomes. Why this lack of interest? Well one feature of academic research is that it rarely looks at itself, for fear of what it will find. The lecture is the stand-out example. Despite many decades of research showing that lectures, and hterefore the professional job title ‘lecturers’, make little pedagogic sense, most academics will dive into the trenches of irrational indignity in their defence. But reson will prevail and on MOOCs we are now seeing some wonderful research by the likes of MIT and the University of Edinburgh.

Standout study

One standout study by MIT compares campus with MOOC students on the same course. It tackles the key issue; do students learn using MOOCs and how does this compare to the much more expensive campus course? If it is true that students learn more or as much on a MOOC, then the cost differential is such that it makes absolute sense to use MOOCs for these courses. This is a solution to the ballooning costs in HE. Even of the students learn less, there is still a strong argument for using MOOCs in terms of costs. Far too many researchers ignore this key issue – the massive reduction in cost.

Enter this study on Physics. It is a introductory, review course on mechanics with an emphasis on strategic problem solving. It starts with straight line motion through to momentum, mechanical energy, rotational motion, angular momentum and harmonic oscillation. The interesting addition is the content on problem solving that requires several different concepts and approaches leading to a single solution.

1000 students who completed the MOOC (Certificates of Completion) were compared with their similarly scored counterparts on the campus course. The relative progress across all groups was roughly equal across all groups. Identical pre- and post-tests were given to assess learning gains.

Results

What surprised the researchers was that the online students did as well as the campus students, and this is the interesting part – regardless of previous academic experience, whether they had a PhD, Masters, Bachelors or high-school diploma. In fact, the MOOC generated “slightly more conceptual learning than a conventionally taught on-campus course”. But there was a further piece of analysis that was even more surprising. When comparing MOOC students with MIT campus students , even though the campus students had received a lot of extra input and instruction, the relative results were the same with “no evidence of positive, weekly relative improvement of our on-campus students compared with our online students”.

Conclusion

This is the only study of its type that I know but would be pleased to hear about more if they focus on learning outcomes. At last we are beginning to see some sensible research that cuts through both the hype and defensive posturing. Good, level-headed academics and institutions are doing what should have been done years ago – researched the learning and cost outcomes. The researchers are now going on to look at what caused the learning. This is good work and long overdue.

Jeff Haywood: stunning analysis of the future of online in HE (MOOCs etc)

When it comes to online learning, the University of Edinburgh, has become the most active and interesting HE institution in the UK. This is something of a surprise, after its disastrous performance in student surveys on teaching, perhaps even, a response to this problem. They do, however, have one great advantage, Jeff Haywood, who has led the charge towards the sort of experimentation and strategic, economic thinking that put the rest to shame.

Haywood keeps an eye on the data emerging from online activity in HE and points towards a steady rise in its use and acceptance by students and faculty (one third have taught an online class, 71% used OER). He points towards an increasing portion of younger people taking MOOCs with a rise towards 1 in 5 students taking online courses. There is a growing recognition that online may even be as good, if not, at times, better that the traditional campus course

.

Students not staff

In his recent ALT talk, Haywood gave a masterful talk on how to think strategically about online learning in HE. I have written about their bold move with six Coursera MOOCs, along with their excellent publication of the resulting data. This is only one strand of a strategy that looks towards 2025. It is a strategy that takes the business case, the economics, seriously.

Where he is way ahead of most commentators in he is that 90% of the tech is used by students not staff. The focus on staff is, to his mind a bit myopic and a bit of an EU obsession. This movement is “student oriented rather than institution oriented”.

MOOCs

“Love or hate them MOOCs have forced open a debate at policy level on digital education”.  Spot on. David Willetts, with Martin Bean, took the bull by the horns and kick started Futurelearn. There is no doubt that this action has stimulated both debate and action in HE. Unfortunately, he has been sacked, and there is no sign that the Shadow Minister for Education Tristram Hunt, is anywhere near as interested.

Unlike many in academe, Haywood speaks with authority based on real and substantial experience, with substantial data, when he makes the important statement that MOOCs “touch learners much more than you might think”. As an aside he also mentions that it has forced Universities to up their game on marketing. We have learnt he says, that “courses can be run at surprisingly large scale” and that “charismatic teachers can reach learners”. Accelerated innovation has also been seen with “a range of technological innovations doing things at scale – peer assessment, comparative judgement online etc.”.

Scale

One word matters above all in this analysis “scale”. Haywood is right to place it at the centre of his analysis. Sure we know that faculty have low digital skills and that there are low rewards for teaching but we must get to scale and move beyond the “bijou and niche”

Where Edinburgh and Haywood have the intellectual upper hand is in their honest appraisal of the economic implications of online learning. In my view the majority of academics are stuck in an analysis that focuses only on quality, ignoring the real issue of cost. Seb Schmoller and others have been arguing for some time, with estimated figures for cost per student, that online, while it may not match the quality, is so much cheaper, that even a weak result makes it worthwhile. I’d argue that the quality issue is fast disappearing, with degree delivery by the likes of IDI. His vision looks to on demand, self-paced, location flexible, relevant to your future, global & local, personalised, affordable, value-added learning; “Without technology this is undoable”.

Walks the walk

At national and international level, policy level discussion is needed with a road map that has clear steps over the next ten years to 2025. This is also true within your institution. If not planned it will not happen. Without investment it will not happen. Without agility it will not happen. This is why “MOOCs and the children of MOOCs are so important”. But that’s just a fraction of the story.  Edinburgh’s MOOCs have racked up huge numbers, with 800k learners on 15 MOOCs and another 15 in the pipeline. Edinburgh has 30k students on campus but also with 50 odd fully online Masters Degrees. (2500 students). It’s a mixed strategy.

Vision by 2020

By 2020 they want 40k students with “all students taking one full online course” and all faculty and teaching staff will have some experience of teaching online. They have the ambition to try to reach 10s of millions of learners through increased online Masters degrees, OER and MOOCs. The means to the end is a series of real funded experiments and pilots, which are all potentially scalable.

Haywood is optimistic and thinks that he is swimming with the tide. The technology has matured, interest risen among learners and policy discussions are far more outcome oriented. One wit in the audience thought that CAVE dwellers (Colleagues Against Virtually Everything), were his biggest problem. Haywood thought that MOOCs had been useful in that those Universities that had taken this leap have found that MOOCs encourage faculty to come forward, as they know they will get support. He added that employers are clearly interested in MOOCs. In Scotland SMEs are interested. For Haywood this is about “opening up the boundaries of space and time – as campus education is limited on both”. He sees nothing wrong with pro bono working education with the secondary aim of recruiting students and charged services coming through.

Diana Laurrilard made a point she often makes and it is pertinent, that Universities have never really understood the cost of teaching. This is true, they don’t even know what is being delivered and to whom. Unfortunately, she has been on the warpath against MOOCs, but only on a straw man basis. She doesn’t believe that MOOCs will entirely replace current HE model. That’s fine, neither do most MOOC providers, including Haywood. Haywood’s response to her question on quality was entirely right. Sure, the tough part is supporting and nurturing students through their personal intellectual development but the answer may be in the middle way. We know that lectures can easily scale so what about the varying degrees of personalised support (something grossly exaggerated in HE). He thinks that technology is already providing solutions, allowing portions of courses to be run on their own, without tutor intervention. Haywood is keen to use intelligent technology at the kind of numbers we run on our MOOCs. He, unlike Laurrilardian sceptics, know a good deal more about the technology, such as adaptive learning, and rightly look for economies of scale, before making rash announcements.

Leaps

The sort of leaps he sees in technology, that allow you to step back and let parts of course run on scale are being looked at, especially in their expanding masters programme. First, there’s online assessment, the Achilles’s heel in HE. The lecture model completely negates sensible formative assessment and the long-form essay, with slow, and often amateurish feedback, seems incredibly dated.

Adaptive learning

He is spot on in looking at learning analytics, especially adaptive learning, for scaling right across the institutions. This is the one technology that already offers hope in tackling the hard to deliver ‘tutor’ functions, pushing courses towards competence-based learning, where learners get personalised learning delivered at their own pace. This is already happening in the US, with considerable investment by the Gates Foundation and a rack of other institutions. Haywood and Edinburgh are the first UK institution to pick up on this and take it seriously. He is being true to his word in retaining some small group pedagogy where you need it but always looking for economies of scale.

Conclusion

Information technology has been extremely consequential in HE over the last 25 years but principally in ‘output enhancing’ ways that do not show up in the usual measures of either productivity or cost per student. Stanford 2012 Tanner Lecture

This is a great quote and recognises that technology has enhanced what we do but the economic are rarely understood. Haywood wants to find ways to use technologies to increase the throughput of students, move through curriculum at their own speed, and automate some parts of the curriculum where you can scale. We need to address the question of increasing productivity without decline in quality. If not we’re just polishing the current system without addressing access on scale.

At last we have a University and academic who is sophisticated enough to think big, be thoroughly strategic, agile, consider the economies of scale with an impressive focus on productivity and costs. One of the great failures in HE planning is a serious attempt to consider actual costs, to be specific, as Lewin and Blefield recommend on detailed cost effectiveness analysis. He also has a vision of the future University, not just being more “open on the boundaries“ but more productive and efficient.

ResearchED conference: blast of fresh air

I’ve attended and spoken at literally hundreds of conferences around the world, but yesterday, on a Saturday morning, I talked to a grass-roots audience of teachers, in an east-end London school, had a school dinner for lunch (apple crumble and custard) and had an insightful time.

Run on a shoestring, by the affable Tom Bennett, it was packed with young teachers, eager to learn and eager to do better for the kids they teach. It wasn’t the usual quango-driven affair, but a genuine attempt to encourage new thinking and debate in the minds of the people that matter.  I liked the fact that I didn’t get a hideous black canvas bag, that marked me out on the tube as a twat. I liked the fact that the introductory speech was by a dedicated headmaster, talking about the very school we were in. I like the fact that Tom was walking the floor, shaking hands and finding time to speak to everyone. I liked the fact that it was all a bit messy and crazy. I liked the fact that Gove wasn’t there as planned (sacked) and that his replacement was too scared to attend after her first tussle (wings clipped on setting).

For those who attended my session on ‘The Good, the Bad and the downright Ugly: 2500 years of learning theory” thanks. For those who didn’t attend, thanks also, because I spoke to tons of you around the place. If you want to find out more on learning theory (30 mins for 2500 years was a tough call) then look here.

What did I learn? These are a bit random but that’s what I liked about the day…

1.     INSET days are a bad way to deliver CPD

2.     Alternative: more ResearchED days

3.     Alternative: more online sharing (Twitter feed was fab)

4.     Forget quangos, let’s talk

5.     Young teachers are mustard keen to learn

6.     Then again, don’t get too precious about ‘teaching’

7.     Shadow Minister for Education is anodyne

This is the sort of event that moves things on. Most of the people attended in their own time, not on the ‘payroll’. I assume that most of the speakers, if not all, did it for free.

Tristram Hunt: if it looks like a duck...

Downton Abbey meets Labour Party policy. So Labour's education policies come down to demanding that private schools help us poor state folks with some private school noblesse oblige. The last thing state schools need is to be patronised by the private sector. Labour needs to be firm on scrapping their tax breaks and charity privileges. There is no evidence that private school teachers are better than in state schools. When I was a school governor that last thing we would have wanted was some private school types from Brighton College coming into our school to 'tell us how it's done'. Let's be clear here - we have NOTHING to learn from you.
After the Thornberry disaster, labour should be worried, very worried. It's not that their education policies are bad, they're downright dangerous. Hunt is remote, his policy ideas either anodyne or irrelevant. What's clear is that this continues in that grand old tradition among posh folk, that 'what was good for me in education' is damn well good for you. It is so obviously driven by his own private school experience. Like Miliband, he needs to go - now.
Earlier this year I gave a talk on 2500 years of learning theory, the good, the bad and the downright ugly. Tristram Hunt, the Labour shadow secretary of state for education was on after me in another room, so, given the impending 2015 election, I trotted off to see what the future holds.

Bad news folks. The future, if Labour win, in education, is anodyne. Tristram, who looked every inch the public schoolboy dressing down to appear like one of ‘us’, comes across as a bright and affable Oxbridge type, who is largely clueless when it comes to the real world. Son of a Blair created peer, he’s the product of that odd form of social mobility, the socialist who gives their son a posh sounding moniker and sends them off to private school. To be fair he hasn’t become a hideous freak, like Toby Young, but he may get power, which is worse. If asked what party this lad was in from this photo alone, what would you guess?

Gove Malt with a little water

He, like Gove and co, is besotted with OECD and PISA. His opening gambit was to parrot out the PISA results and worry about the looming 2018 tests. He’s already worried about falling off the leaning tower, before he’s even in power. Why? Because the next round of tests are bang in the middle of the next political term. Rather than present a real vision for education that tackles the ideological obsessions of the right, he simply buys into their vision and adds a little water to their expensive and elitist, single, malt whisky.

You’ve been quangoed

His next pronouncement was even worse. He wants to fund a ‘Royal College of Teaching’. Just as we’ve recovered from the implosion of the General Teaching Council, he wants to do that old Labour trick of quangoing a problem. This is a sop to teachers but it we’ve seen it all before. These quangos come and go. They are overfunded, badly managed, populated by previous quango-types and largely ineffective. They now have a half life of maybe 3-4 years. It’s like bonfire night. We spend all that time building a huge tower of babel, then rightly see it as a pile of rubbish and burn it down.

A little less

On vocational learning he will continue with the madness of forcing GCSE Maths down the throats of everyone right up to the age of 18, even for students who want to take a more vocational route in life. How do I know this? I asked him. He refuses to consider a separate functional maths qualification and has no real ideas on the hollowed out vocational heart of our educational system, the real reason for our low international, economic performance. The US is still the world’s most vibrant economy and they perform even worse than us in PISA.

Conclusion

I’ve been a lifelong Labour member and supporter. But there comes a time when you no longer recognise the values of a party, you drift apart. I have nothing in common with the current homogeneous crop of Oxbridge think-tank clones, who have little experience of the real word and little contact with the people they profess to protect. Tristram, Miliband and co look like ducks and if they look like…… They float across the surface, quacking away, in their neat little first suits, their little feet furiously paddling beneath but they are bereft of ideas, policies and even charisma. I had heard from others that Tristram was a Gove-light and this was confirmed yesterday, with a charming little speech that played to the gallery of teachers (tossing them a quango) but in reality was continuing with the policies Gove has introduced, just watered down a little and delivered with with a little less scorn.