10 uncomfortable truths about the con that is University rankings

Six Universities have been hauled up by the Advertising Standards Authority but it could have been them all. Almost all Universities claim to be above the lowly business of commerce but still willingly contribute to the petty hit parades in the University league table season. It demeans the sector. They search for whatever scraps they can find by selecting data from one ranking table or another. They love to claim they are above the competitive, capitalist, corporate game but they are by far the worst when it comes to the dog-eat-dog, institutional competition that are the rankings. Worst of all, for the people that pay, whether its taxpayers, parents, national or international students, the University Rankings are largely a con.

1. Bait and switch

The sector loves to take the high moral ground on keeping managerialism out of education, then use the slimiest form of managerial marketing, ranking tables, to promote their wares. Aimed firmly at parents and students, they bait and switch. The hook is baited with data on research and facilities, then the message switched to make it look like the teaching experience you’ll pay for, when in fact, the rankings are about measures that have little to do with teaching. That is a classic 'bait and switch' con.

2. Teaching ignored

They may SAY they take teaching into account but they don’t. They often claim to have ‘measures’ on teaching, but actually draw their data from proxies, such as staff qualifications and research activity and use nothing but indirect measures to measure teaching. The Times rankings are a case in point. They claim that their ranking scores include teaching. In fact, only 30% is based on teaching but they use NO direct metrics. The proxies include student/staff ratios (which is skewed by how much research is done) and, even more absurdly, the ratio of PhDs to BAs. It is therefore, a self-fulfilling table, where the elite Universities are bound to rise to the top. There is little direct measurement of face-to face time, lecture attendance or student satisfaction. In some cases it’s laughable, as Malcolm Gladwell pointed out, with Faculty salary, levels of degree in Faculty and proportion of faculty who are full time, being taken as proxies for quality of teaching. It’s like having a Premier League table based on the performance of the backroom staff and not the real games and players.

3. False precision

Up one place in the rankings – yippee! Down two places – time to worry. Yet the idea that these rankings are in any way precise is silly. They’re a mish-mash of misleading data, under vague (even misleading) categories and often watered with a heavy dose of opinion (expert panels drawn from top Universities). In any case, they’re always changing the criteria for ranking, so year-on-year comparisons are useless. This shows itself in the huge disparities between the different ranking systems. The LSE is 3^rd in The Sunday Times rankings but 328th in the US News and World Report Rankings, 71st in the QS Rankings and 34th in the THE Rankings). Other universities like Manchester and KCL do badly in British rankings but well in international tables. This gives ample room for cherry picking but is poof enough that the way the rankings are calculated is seriously flawed. If the rankings were research they'd be rejected by even the lowliest of Journals.

4. Apples and oranges

They don’t compare like with like. In Edinburgh, where I come from, we have four Universities; Napier, Heriot-Watt, Edinburgh and Queen Margaret. You couldn’t get four more diverse institutions in terms of what they teach and their history. In 2012 Edinburgh were in top five for research but came stone-cold last in the teaching survey. That same year, Heriot Watt came top in Scotland and 4^th in UK on Student experience but way, way down in the rankings. In that same year, more than a third of the Russell Group Universities found themselves in the bottom 40 of 125 institutions (2012) on teaching. These comparisons are truly odious.

5. Skews spending

What is sad, even morally wrong, is they they really do influence strategy and spending. Ranking status is often stated explicitly in their goals. In effect, as teaching doesn’t really get measured, except through false proxies, it leads to spending on everything but good teaching – physical facilities, research and so on. This direct causal effect on behaviour also leads to overspending, as it’s a runaway train, where everyone tries to outdo everyone else. There is no incentive to save money and become more efficient, only to spend more. Weirdly, there’s rarely any accounting for students costs in calculating the rankings. Shouldn’t a University that costs a lot less get ranked higher than one that does not? It would appear that prejudice trumps economics. This is a topsy-turvy world, where being more expensive is an intrinsic good.

6. Gaming the system

It’s not just spending that’s skewed by rankings, they also skew behaviour and priorities. Universities are far from being free from the rat race, they just have some very smart rats. In practice, this means that they are good at gaming the system. What are the criteria and weightings for ranking? OK, those are this year’s targets. More facilities, let’s get them built.

7. Self-fulfilling prophecy

The more you spend, the higher your ranking. So the rich get richer, the poor get poorer. The separation, in terms of research grants between the handful at the top and the rest is huge. Naturally, this leads to a separation of the so-called cream from the so-called milk. In that sense it’s a deterministic system, where the top remain at the top and the rest scrabble around for the scraps.

8. Agendas

What’s more, the different tables often have uncomfortable relationships with newspapers. And let’s not imagine that, given the nature of newspaper ownership in this country, they don’t have agendas. The Complete University Guide has had relationships with The Telegraph, Times and Independent. They keep falling out. The Sunday Times has its Good University Guide. The Guardian has yet another. These tables sell newspapers to middle class parents, that’s the real driver.

9. Old boys club

Reputation scores feature in lots of the rankings. You go out and ask people what they think; academics, publishers, employers etc. Of course, given that most of the people asked are from the highly ranked Universities, there’s an obvious  skew in the data. That's shameful, qualitative nonsense.

10. Status anxiety

What is their real effect on parents and students? Nothing but an irrational race. They induce ridiculous amounts of status anxiety. Parents and kids are being encouraged to play a game which is already gamed and get stressed over data that encourages distasteful behaviour.

Conclusion

I haven’t even begun to tackle the issue of cheating, being economical with the truth or fiddling around with the submissions. There are examples of straight up cheating, and as there’s no real quality control, it’s likely to be far more common than reported. In truth, no one really knows what the ideal criteria for ranking should be, as it’s a set of competing ideological choices – accessibility, teaching, research, graduation rates? And with what weightings? That’s why the different rankings have these huge disparities. We need, like Reed University in the US, to refuse to hand in the assessments. If the game is being gamed, don’t play the game.

10 x 10 lists on common mistakes in online learning

To ten tips in top ten topics in online learning:

10 ways to make badass INTROs in online learning 

10 bloody good reasons for using much-maligned TEXT in online learning 

10 essential online learning WRITING TIPS in online learning 

10 stupid mistakes in design of MULTIPLE CHOICE questions

10 essential points on use of (recall not recognition) OPEN RESPONSE questions

10 rules on how to create great GRAPHICS in online learning ‪

10 sound pieces of advice on use of AUDIO in onlinelearning 

10 ways based on research to use VIDEO in online learning

10 ideas on use of much maligned TALKING HEAD videos in online learning

10 unusual ways to make SPACED PRACTICE happen in online learning 

This started with a simple observation that I'm seeing, over and over again, the same mistakes being make on screen, with online learning. I hope you find them useful.

10 Corbyn education policies that actually make good sense

I’m no Corbyn fan but his BIG idea in education (NES) is way beyond the pale, insipid Tristram Hunt policies that typified the last Labour leadership's educational policy. I was highly critical of Hunt's approach. Actually, it wasn’t a policy at all. They decided to keep the issue off the political canvas, as they largely agreed with the Conservatives. No difference, don’t mention it. Thankfully the hapless Hunt has resigned. But what does our Jeremy really promise in education? He outlined his ideas here but I've speculated a little, based on past pronouncements and alliances.

1. National Education Service

In his acceptance speech he thanked the "Socialist Education Association". Their ideas have informed his policy. It was they who want to  ‘develop a single, broad and inclusive framework for the curriculum from early years to adult education’. The emphasis on a universal (comprehensive) system, with equality of opportunity at its heart, is exactly what this organisation recommends.
So what is the National Education Service? Corbyn equates it with the National Health Service - free at the point of delivery. So far, so good. This is a fine idea. The world of learning is like the separate horizontal layers of an old, stale cake. Pre-school, primary, secondary, sixth form, FE, HE, adult learning. Yet individual learners go on a vertical journey and have to smash through each of these horizontal layers in turn, often failing, getting disillusioned and the net result is a system where nobody’s happy – learners, teachers or parents. We need to see learning as a lifelong experience, like health.
It may also align education with health in another fashion, with a focus on evidence-based teaching and learning. We have long compared education with medicine, showing that one has advanced while the other has remained largely static in terms of both delivery and outcomes. Here's an opportunity to take the research and professionalisation agenda seriously. The problem he faces is that teachers want professionalisation but then go all woozy when it comes to professional standards and a research and evidence-based approach to teaching and learning. You want to be like the NHS, then make the whole thing student-centred, in the same way that health is patient-centred.
The NES needs to be fleshed out. How will it be funded, run and organised? He has to reconcile his belief that Local Authorities should build and run schools v a NES. You can do what is currently done with the NHS - directly funded by Government, lots of control, appoint a CEO, free at point of delivery but with professional bodies, such as NICE, recommending and publishing guidelines (Ben Goldacre has written smartly about the need for a NICE model in education). A second model is the BBC-type model, where you set up a separate Trust, at arms-length from Government. This is unlikely as control is what they want and probably need, to get things done. A third model is a highly devolved model back to Local Authorities but this is dangerous as many could be hostile. Unfortunately, that's the chosen model, bringing all free schools and academies under local authority or Mayoral control.
But remember also, that it is not possible to have a ‘National’ Education Service. What he means is an 'English' Education Service. Scotland, Wales and NI, long disgusted by the political shenanigans in England, have long gone. Nevertheless there may be room for more alignment, as this is the model they sort of have elsewhere.

2. More vocational

It also unifies the funding. I’m in favour of unifying educational funding as it oils the wheels for more rational decisions, especially the balance between academic and vocational. HE has had its own way for too long. It’s bloated and over-funded. We need to rebalance HE with a stronger approach to vocational. Curiously, Corbyn is more aligned with the Conservatives and their 3 million apprenticeship promise at the last election. But his view of apprenticeships is not one of being employer-led. It would be accredited by FE (that's weird) and employers. This is a bit fuzzy but there is a clear need for a properly defined and funded apprenticeship system, which to be fair, the Conservatives have structured through a levy. It has cross-party appeal.  The good news is that he really does value ‘skills’ and wants to stop the deep erosion of FE and the adult skills budgets. It's anti-Blairite and it's right. One sad footnote - he wants to pay the minimum wage to apprentices - this is not necessary. To create a viable system we neeed to recognise that overloading it with costs is unwise.

3. Scrapping University fees

He wants to scrap fees but also reintroduce grants. This is par for the socialist course but it has consequences, not least the subsidising of the rich by the poor. The majority of young people do NOT go to University and there's growing evidence that the Blair policy didn't work as it crushed alternatives to the expensive HE path. Yet this is the only Corbyn policy that gets any attention in education, as the middle class have sharper elbows - that's a distorted shame.

4. Online education?

He’s a fan of the Open University, saw it as a socialist triumph, and talks about it fondly and explicitly as a great Labour achievement. Good on him. I agree, and hope that he will expand on this online approach to education. Tom Watson's a digitally sophisticated politician and really does get this stuff.

5. Get rid of charitable status for private schools?

Keiza Dugdale, Corbyn’s emissary in Scotland, has hung her hat on this policy in education and I’m sure Corbyn agrees and will attempt to do this south of the border. This is long overdue. The Conservatives and, unfortunately, the hapless Tristram Hunt, was in thrall to this elitist system. It's unlikely they will call for its abolition but we do have to see them as the businesses they are and recognise that they are a major force in creating inequality.
6. Scrap Grammar Schools and 11+
He want to scrap all Grammar Schools and the 11+. Indeed, this was romoured to be the cause of his divorce. This has long been a stupid anomaly in England and causes no end of chest beating in the Conservative Party. David Willets, one of the smartest people in the Conservative government, was sidelined and eventually sacked, just because he held this belief, so it is not a mad, loony-left policy but a mainstream belief. Good policy, let's get it done.
7. Fewer tests
Fewer tests! Thanks God. This taps into the widespread view among teachers and parents that this has got out of control. However, the hard-left have a habit of sticking with centralised state-control in education , so don't hold your breath. The SNP, north of the border have just introduced another raft of testing.
8. No league tables for schools
Yipee. Again this taps into the zeitgeist about education being a right, not a competitive market. We don;t have league tables for hospitals, nether should we for schools. Education is far too important to turn into a competitive sports spectacle.

9. Single examination boards?

My guess is that he’d also unify examination boards, A Gove idea but a good one nevertheless. It’s what they have in Scotland and makes a NES that much easier to implement. the current boards can barely handle the quality control necessary for an efficient system and every year we have unanswerable or stupid question items. Schools also cherry-pick (on what basis I wonder?).

10. Corporation tax

On costs he wants to add 2% to corporation tax. This is reasonable, and matches the ‘levy’ the Conservatives want to load on to employers for apprenticeships. In many ways this is easier to implement and redistributes profits into training. The problem here is that this raises only £3 billion. This nowhere near covers what Corbyn is proposing. So, as usual, the policies are not costed.
Ministers
One worry I have is that, in education, Corbyn has a pretty dismal personal record. He was pampered through fee-paying prep and boarding schools but only got two E grades at A-level. (You get an E for turning up.). He then dropped out of his University course for disagreeing with his tutors. Wow. 
Also, Lucy Powell, appointed Shadow Education post, is an apparatchik politician - school, Oxford, party HQ, assistant jobs, MP. Far from stellar, she's a bit of a Labour clone. On the upside Angela Eagle, who should have this portfolio, is a formidable and capable politician. BIS has always been badly run with lacklustre civil servants. It needs a shake-up.

Conclusion

At least it’s a bold idea, roughly in line with what many want. But to pull this off you have to centralise funding agencies (a good idea) and save costs. Yet, the idea also brings in its wake the usual quango-building. Corbyn is unlikely to go for a merger approach, so we’ll likely end up with a profusion of bodies with a large administrative centre. That’s worrying. Bureaucracy may be its downfall. One last point. Why is almost everyone in education ignoring these policies, other than student fees? Is the teaching profession sweeping actual policies and reforms under hte carpet with a focus on the one polocy that affects their kids - University fees? A conundrum.

10 ways to make badass INTROs in online learning (& ditch dull learning objectives)

So many online learning programmes don’t start well. They’re often dull, overlong or, worse, a boring list of learning objectives. We have to get over the idea that we’re putting textbooks on screen. This is the web folks and the rule is – you have 2 seconds to impress. Attention is a necessary condition for learning, so your job is to raise attention and curiosity, not bore them into submission.

1. First impressions matter

First impressions matter, so they say, but in online learning they really do matter. Ebbinghaus showed us, back in 1885, that memory has a tendency towards ‘primacy and recency’, a bias in which the first and last things are retained and recalled better than what is presented in-between. So pay attention to the intro. It is the door to the learning experience and they should want to push it open. Make it relevant and memorable.

2. Titles

Too many courses are have titles that seem designed to turn you off before you’ve even started. A great title will catch attention, intrigue, give an idea of the content and even set the tone or voice of the leaning experience. Write a list of titles, one word titles, two word titles, three word titles, Why..., How to... Is there a concrete image that can be used?  How about a play on words, rather than Use of Gamification in mobile learning' try 'Game of Phones'. Pick a title that excites. That’s what movie makers do and it’s a good practice. Rather than ‘Learning technologies 101’ try “From Gutenberg to Zuckerberg’. Be more imaginative. Question? Did the title to this article, or word 'badass' get you here in the first place? One can also do a little A/B testing to get this right - this is what tech and ad companies do as a matter of course.

3. Ditch learning objectives

Straight out of behaviourism, this practice lingers on and on in courses. In online courses, avoid this edu-speak and focus on an effort that gets the learners attention. Attention is important in learning and it is counter-productive to bore them with a list of dull objectives. For more on this see 7 reasons to kill up-front learning objectives

4. Avoid padding

Subject matter experts, perhaps because they’re used to writing textbooks or manuals, have the unerring habit of writing over-long pieces called ‘Introduction to…’ or ‘The history of…’ or ‘Background to…’ at the start of courses. This is rarely either necessary or desirable. Attention is your currency, don’t devalue it by turning your subject into a snoozefest. Look at how great books and movies grab you with their opening lines or scenes.

5. Focus on just one thing

Nothing raises attention and curiosity more than a suprise. Most great movie openings do this. They start with being wide open then bang, focus on a close-up or one great scene. Great courses start with these surprises – a great quote, shocking statistic, compelling image, poignant question, conundrum. Think long and hard about your singular intro, as it sets the scene for the whole course. Great movies have great opening sequences. Check out this opening sequence in one of The Hangover films.... 

6. Keep it short

There’s nothing worse than an interminable legal warning, disclaimer, video, overlong animation or boring text introduction to a course. Your learners may have come to the course with high expectations, even low expectations. It is your job to grab and excite them. That is rarely achieved with long opening sequences. Make it count but cut to the chase. If you do have to have this stuff, make it optional, like terms and conditions, from a button.

7. Interactive

Online learning is interactive, so don’t be afraid to start with participation. Try a question, a common misconception, something that wakens the learner up, raises attention. A good ‘hinge’ question can work well. On the other hand, whatever you do, don’t start with a long learning styles quiz (because they don't exist and it will be a waste of time) or some fatuous Myers-Briggs nonsense. More on the Ponzi scheme that is Myers-Briggs here. Be bold.

8. Humour

Doesn’t always work but when it does, it can do exactly what you want, raise a smile and, if relevant, make a great opening point. You can do a lot worse than raise a smile at the start of a learning experience. I made a programme for maintenance engineers once, where I deliberately made the screen go blank. Every engineer in the world leans forward to check the power supply then the lead connections. I then switched the programme back on and said, "That's what customers feel like when their service gos down...." It raised a laugh or two.

9. Skip on return

It can be annoying to see the same intro time after time. If the user returns to a course or module across many sessions, allow them to skip the intro or remove it altogether.

10. Movies and TV

Watch the openings to Movies and TV, then ignore the fact that you have to have credits. But do pay attention to the way they use smart titles, pose questions, make you think about what you’re about to see, show a fascinating clip that you’ll see later. They want to grab you before you switch to another channel. Plagiarism is a form of flattery. Here's a list of the Top 25 film openings.

Conclusion

To get off to a good start, attention should be your aim, not showboating with overlong sequences or dull objectives. There’s no silver bullet here, as each course needs its own unique introduction. Hopefully, these ten ideas provide some sort of stimulus when you’re faced with that blank piece of paper.

Other related pieces…..

10 bloody good reasons for using much-maligned text in online learning ‪http://bit.ly/1KnJB2c 

10 essential online learning writing tips & psychology behind them ‪http://bit.ly/1JnUo6J 

10 stupid mistakes in design of Multiple Choice question ‪http://bit.ly/1JvMNCf 

10 essential points on use of (recall not recognition) open-response questions ‪http://bit.ly/1PPjIXb 

10 sound pieces of advice on use of AUDIO in online learning ‪http://bit.ly/1MccsXJ 

10 rules on how to create great GRAPHICS in online learning ‪http://bit.ly/1iguKL4 

Hattie: Visible learning - the naked teacher - a primer

Almost every educational intervention has a positive effect, to a degree, but what matters is to select those evidence-based interventions that work well and can make a real difference. John Hattie set out to examine, over 15 years, a synthesis of 800 meta-analyses of over 50,000 pieces of evidence, categorise, then assess their impact, namely students achievement. If, he thought, we could determine effect sizes of interventions on student achievement, we can recommend good and better teaching practice. The end result was a table of interventions that could be used to guide policy and practice in schools. Hattie sums his central idea of bringing teaching and learning to the surface, through research, as Visible Learning, the title of his first book.

Visible Learning

Visible learning is the idea that teachers and, importantly, students, should make things ‘visible’.  The goal is to enhance the role of teachers through the evaluation of their own teaching. Hattie thinks that Visible Learning and Teaching occurs when teachers see learning through the eyes of students and when students, to a degree, become their own teachers. He is part of that tradition that wants a more evidence-based approach to both the profession and process. All of this is expressed in his three books; Visible Learning, Visible Learning for teachers and Visible Learning into action.

He attempts to identify and rank teaching and learning interventions by impact and abhors the current culture of teaching, as, “We have a profession where everything goes…where we close the door and don’t let anyone discuss what we do behind that door”. He backs this up with evidence about what teachers actually talk about in staff rooms and training sessions. What they talk about, when recorded and measured, are kids, the curriculum and a culture of complaint around the politics of the school. “It’s a profession that doesn’t talk about teaching” he claims. The evidence? When measured, only, “One minute a month they talked about teaching!”

Effect sizes

In Visible Learning Hattie hung his hat on ‘effect size’. Every teacher intervention has an effect but not all effects are equal in terms of impact Some interventions may have small effect sizes but you’ve lost the opportunity cost of applying other more successful interventions. He recommends interventions with an effect size greater than 0.4. These are:

This is a fascinating list and what’s just as interesting are the things Hattie regards as low impact and by this he means wasteful. These lie at the bottom and include many of the sacred cows Hattie sees as distractions in the education debate; school leaders, class sizes, homework (in primary schools), extra-curricular activities, gender, ability grouping, open learning spaces. summer holidays, welfare policies and television.

Teacher training

He is highly critical of teacher training, claiming that it is gets bogged down in largely irrelevant debates and is largely led by opinion, not research. This is important, as he sees a renewed focus on teacher practice as the best way to improve student attainment. It’s not that teachers need more time, they need to do things differently. In short, he thinks that teacher training and CPD debate is upside down, with its focus on things at the bottom of his table, that have relatively small effects, especially structural issues in schools. He wants teachers to change in response to what has been shown to work best. He is highly critical of allowing teachers to be largely ‘autonomous’, closing the door on their classrooms, often trying out untested and personal ‘bandwagon’ techniques. His recommends far more collaboration and sharing.

Criticism

Black and Wiliam criticized the work for failing to recognize that effect size is influenced by the range of achievement in the population. He has also been widely criticised or presenting probabilities as being negative, a mathematical impossibility. Hattie defends his use of the first but admits he made an error in the calculation of the second. Hattie also excluded student background and social context from his research, yet many assert that these trump many of the effects Hattie puts forward as good interventions. On the whole the criticism centres around the idea that the issues are more complex than Hattie asserts and that effect sizes are not nearly enough, in terms of evidence, to lead to the sort of policy decisions that are put forward on the basis of this one book. There is another issue around the sensitivity to instruction. None of the meta-analyses control for the effects of differences in the sensitivity to instruction of the different outcome measures. It has long been known that this is a significant effect, and that it is difficult to assess.

Conclusion

Hattie attempts to raise the bar for the teaching profession in seeing teaching practice as something that needs to be informed by research and evidence. His point is that continuous improvement must be sought, based on what we can show works, rather than autonomy, traditional or existing practice. We can criticise his effect sizes and data, but the recommended principle is sound and remains intact.

Bibliography

Hattie, John A. (2008). Visible Learning: A Synthesis of Over 800 Meta-Analyses Relating to Achievement.

Hattie, John A. (2011). Visible Learning for Teachers: Maximizing Impact on Learning.

Hattie, John A. (2011). Visible Learning for Teachers: Maximizing Impact on Learning.

Hattie, John A. Masters, D, Birch, K. (2015) Visible Learning into Action: International Case Studies of Impact.