Like the real Leaning Tower of PISA, the OECD PISA results are built on flimsy foundations and are seriously skewed. Nevertheless, they have become a major international attraction for educators, and have sparked off an annual educational ‘international arms’ race’
In the UK, education policy is rooted in, and firmly targeted at, the PISA results. The English Baccalaureate has PISA written all over it. PISA is also used as a political football by D-list celebrities and wannabes, like Toby Young and Katherine Birbalsingh to beat the state system over the head. They stare at the learning tower and swear blind that it’s straight.
Both left and right now use the ‘sputnik’ myth, translated into the ‘Chinese competitiveness’ myth, to chase their own agendas – more state funding or more privatisation. This is a shame, as the last thing we need is yet another dysfunctional, deficit debate in education. Nations have different approaches to education, different demographic and social mixes and different agendas.
The problems in the data are extreme as PISA compares apples and oranges. In fact it compares huge watermelons with tiny seeds. PISA is seriously flawed because of the huge differences in demographics, socio-economic ranges and linguistic diversity within the tested nations. Here’s seven skews as a starter:
1. Size skew
Shanghai topped the table this year but why was one city from an entire nation singled out? Could it be that Shanghai is China’s flagship city? To compare a set of students from the richest city in China (on average 3 times the national income), that has attracted a high proportion of high-achievers, with a similar sample from across the whole of the US is odd. It smacks of political manipulation by the Chinese. There was also evidence, presented in the New York Times, of student priming. Imagine if the cohort was drawn from rural China? This comparative method will fail as there will be lots of outliers in the data. It is not surprising that small homogeneous cities and states pop up at both the top and bottom of the table. The incentive to 'cheat' is huge.
2. Tuition skew
One huge variable that is NOT accounted for is external tuition. This is rife in some of the high perfroming countries, especially China, Singapore and Hong Kong. Many would argue that this one variable alone kills of the statistical significance. We have a choice, ht house or bring up blanced, rounded kids. I know what side I'm on.
3. Immigrant skew
Different tested cohorts have different immigrant ratios. The difficulties that immigrants have with language, social adjustment, school and poverty, is a serious pollutant to the data. As one would expect, Finland and Shanghai have very small numbers of immigrants in their tested cohorts. It is bonkers to compare cohorts with radically different numbers of immigrant children.
4. Selective immigration skew
There is another odd skew associated with immigration, namely that for some countries, immigration is controlled, so that only wealthy or smart kids get through. So, for example, there’s only one country in the PISA results where the immigrant students outscore the natives and that’s Australia, where immigration is highly selective. There’s a huge difference between refugee immigration and cherry-picked immigration.
5. English skew
Associated with immigration, is a curious linguistic skew – the tendency for smart immigrants to migrate towards English speaking nations. This could mean that English speaking nations benefit in the long term from such immigration but show poor short-term results due to high first generation immigration with associated language problems at school.
6. Linguistic skews
On reading, languages with regular structures are likely to do better than languages which are more irregular. The tests may favour languages with simplified spelling structures such as Finnish. Reading data may also be skewed by reading habits as PISA doesn’t recognise reading on screens. It’s big on books.
7. Subjects skew
PISA measures academic subjects only, namely maths, reading and science. To be fair PISA have recognised this flaw and are now embarking on a correction process. But is it right to judge education in these subjects only? One need only focus the curriculum heavily on these subjects to do well, which is exactly what many counties do. Dump sport, music, the arts and humanities and you can produce stellar results.
8. Subject focus skew
Simple differences in taught curricula can also affect the results. In maths, for example, if you have taught ‘series’ theory you will do well in the 2009 results, as a major set of questions focused on extrapolating series in the test. If this is not part of your curriculum, you will score badly.
9. Data fatally flawed?
Sven de Kreiner Danish statistician says PISA is not reliable at all. In the reading tests 28 questions were supposed to be equally difficult in every country. PISA has failed here as differential item functioning - items with different degrees of difficulty in different countries - are common. In fact he couldn't find any that worked without bias. Items are more difficult in some countries. He used his analysis to show that the UK moves up to 8 or down to 36. PISA assumes that DIF has been eliminated but not one single item is the same across the 56 countries.
10. Differences not that great
OFQUAL published a Progress Report (International Comparisons in Senior Secondary Assessment ) in February 2011 making similar points.
They listed several major criticisms:
- differences between countries’ performance are not that large…usually statistically insignificant
- whether or not a country has moved up or down the league tables is not that meaningful partly because the absolute differences in scores between countries are not that great
- the constituent group of comparators changes from study to study and from year to year
They point to three major but dangerous assumptions that:
- items tested for are somehow an objective measure of what is best
- learners undertaking the study are a balanced representation of all learners at that stage of education
- learners sampled in each country are equally motivated to perform well in the tests
Additionally, these snapshot studies do not isolate variables and may well be skewed by “factors in the past that no longer apply”, such as “learner performance in an examination may be the result of curriculum developments undertaken” or “investment in education infrastructure some time in the past”. In other words, using the data to praise or blame the current system is unwise.
Gove’s skew
The great danger is that the world skews its curriculum to fit the PISA expectations, just as PISA draw away from their own curriculum tested areas. This has already happened in the UK with Gove’s Ebac. Gove has specified A*-C passes in five subject areas: English; maths; two sciences; ancient or modern history or geography; and a modern or ancient language. It has all the hallmarks of a PISA-led curriculum. First it’s far too academic and restrictive. Second, it excludes too many sensible options. But his greatest crime is to have moved the goalposts after goals have been scored. If you change the goalposts so dramatically and quickly, you simply condemn 85% of students as failures (only 15% currently meet the Ebac standard). What’s worse, Gove is applying the measure retrospectively. This is like moving the goalposts at the end of the game and disallowing goals scored. It’s madness. You can have schools with high achievement in Maths and English plummet down the new league tables from near the top to near the bottom, as they haven’t focused on humanities or languages. The consequences of this error could be disastrous as the staff pressures will also be enormous, with thousands of teachers in vocational subjects being rendered useless in favour of history, geography and language teachers. One weird consequence is that a student who does Latin and Ancient History will be judged above those who do Business Studies, Engineering, psychology, a third science and lots of other subjects. It’s worse than bad , it’s perverse. I’m glad my kids are leaving secondary education, as it descends into this backward looking nonsense.
Devil’s in the detail
Politicians and activists distort PISA to meet their own ends. They cherry pick results and recommendations, ignoring the detail. Finland is famously quoted by the right as a high performing PISA country. Yet, it is a small, homogeneous country with no streaming, high levels of vocational education, no substantial class divisions and no private schools. Facts curiously ignored by PISA supporters.
Conclusion
One could quibble with the details of my analysis, but I’m convinced the PISA comparisons are riddled with skews and errors, many more than indicated above. The great danger, and it is already happening, is that people read causality into the data. It’s crap schools, crap teachers, money spent doesn’t matter etc. The scope for false causality is enormous and exploited by politicians for their own ends.
PS
"In the last 10 years we've plummeted in the PISA rankings" heard this before from Michael Gove - he lied. UK results were excluded in 2000 (low response rate) and 2003 as data was dodgy. Only gathered in 2006 and 2009. PISA tests not that important but National tests have gone up - what's happening? OECD does not compare over the 10 years. Performance has not fallen, if anything it's flat.
6 comments:
As always with Gove, I wonder: did he consult anyone about this? Did he think it through? Or did he just come up with the idea off the top off his head? The problem for schools is that now they're all having to run around trying to find extra history teachers, extra French teachers, extra physics teachers to cope with the surge in demand. Will there even be enough teachers to meet demand? And presumably at the same time schools will have to sack teachers in other subjects.
At the same time Gove keeps saying he's going to give teachers more autonomy, and he's going to reduce the number of compulsory subjects in the national curriculum. But what's the point of doing that if you then say that this particular set of subjects represents some kind of standard that everyone has to aspire to? Does he *know* that he's contradicting himself?
He doesn't care about due process or evidence or 'abuses of power'. He's purely ideological. Don't imagine for one minute that they have thought this through. But what on earth are the Libdems doing supporting this nonsense?
Donald, we are in danger of "doing a Gove" - taking PISA at headline value. If you read the recommendations for policy nothing could be further than Gove. Gove is the problem not PISA.
PISA suggests the following:
# To become effective learners, students need to be able to figure out what they need to learn and how to achieve their learning goals. They also need to master a wide repertoire of cognitive and meta-cognitive information-processing strategies to be able to develop efficient ways of learning. At the same time, fostering effective ways of learning, including goal setting, strategy selection and controlling and evaluating the learning process, should not come at the expense of students’ enjoyment of reading and learning, since proficiency is the result of sustained practice and dedication, both of which go hand-in-hand with high levels of motivation to read and learn.
# PISA assesses an innovative concept of literacy, which refers to the capacity of students to apply knowledge and skills in key subject areas and to analyse, reason and communicate effectively as they pose, interpret and solve problems in a variety of situations.
# In short, many successful school systems share some common features: low levels of student differentiation; high levels of school autonomy in formulating curricula and using assessments with low levels of school competition; and spending in education that prioritises teachers’ salaries over smaller classes.
# Allow schools to observe the inequities among schools, which could be considered the first step towards redressing them.
# PISA suggest that schools and countries where students work in a climate characterised by expectations of high performance and a readiness to invest effort, good teacher-student relations, and high teacher morale tend to achieve better results.
# Research suggests that creating conditions that promote reading practice, and letting students read what they want to read, could be beneficial.
Don't doubt PISAs intentions but the unintended consequence is an 'arms race' with politicians failing to pick up on the detail. My piece was mostly about the lack of integrity in the headline data and leagues. You can't make good inferences from poor data. Indeed, skewed data gives skewed conclusions.
PISA is like colonialism in education. Living in the Peruvian Andes, PISA tests are meaningless because native communities's (quechua and aymara)education is more focused on Andean cosmovision.
And of course, you can't compare human beings, cultures and education with a simple test.
I have Finland friends and they are not proud of their education or about Finland's PISA grade. They explained me that was just a coincidence: what PISA evaluated coincided with their current system education.
Good pts on PISA. I think there are two main reasons for Gove's English Bac. One is his narrow academic view of what education should be about.
But the key reason is, i think, more sneaky. Currently only 15% get the right combination of subjects for the English Bac. By making schools focus their choices on it, that % will at least double, without any real increase in achievement. Gove will claim to have transformed education, based on figures that are smoke and mirrors.
As a governor I feel in a very difficult position. When the focus was on 5 GCSEs (including english and maths), the needs of the school and the students were aligned. Pushing those borderline kids helped them and the school. But pushing kids to do the English Bac, because the school needs a higher %, may not be in an individual student's interest at all.
Post a Comment