Educational research largely useless - no costs, no bite

A question that is often asked is why education seems to never change in relation to to technology, compared, for example, to health? One answer is that there's no real consequences of failure for deliverery in education, there is in health. A more extreme example is the universal use of very expensive flight simulators for pilots. Why? Pilots go down with the plane! But another problem is the lack of a culture of cost-effectiveness analysis (CEA).

A wonderful paper by Levin and Blefield ‘Guiding the development and use ofcost-effectiveness analysis in education’ (2013) shows how education, unlike other areas of massive expenditure fails to apply Cost-Effective Analysis (CEA) to research projects. In practice, this leaves a massive vacuum at the centre of arguments around educational interventions and explains why progress is glacial.

Given the massive costs to society for education this is a puzzle. Surely, say the authors,“alternatives that show greater productivity relative to their costs, i.e., that are more cost-effective and more efficient in the use of social resources, should be preferred for adoption and implemented more intensively.” But how often do you see costs mentioned and if they are, they are of such low quality as to be meaningless.

The problem, as the authors of the report claim, is endemic. There’s little understanding of some basic economic concepts among educational researchers. No real distinction is made between Cost Benefit Analysis (CBA) and Cost Effectiveness Analysis (CEA). There is rarely any conceptual understanding of Opportunity Costs or realistic Sensitivity Analysis. This leaves research incomplete and hanging. A good CEA can even shift marginal and even negative research results into positive territory, as one delivery method may result in marginal or even negative learning effectiveness, but at a much lower cost, allowing other interventions to take place, making the overall system better.

Examples - MOOCs, Blended, Tablets

Let me give you an example. In debates around MOOCs, many claim that MOOCs are no substitute for face-to-face campus courses. But that is not the point. If they result in the same measurable academic outcomes, or even lower educational outcomes, the real win is in the fact that MOOCs can deliver a scalable solution at a tiny fraction of the 'cost per learner' of traditional campus courses. We’re not talking about shaving percentages off the costs but coming down to a tiny fraction of the original costs. This, in turn frees up resources to do better teaching and even more research.

One more example is blended learning, where effective alternatives are rarely, truly calculated in terms of the costs components in relation to effective interventions in a blended solution. Blended learning often turns out to be just variations on blended ‘teaching’, with no real appreciation of the true costs of delivery.

Yet another example is the purchase of tablets in schools. I doubt of there is a single exhaustive cost-effective analysis on any of these large-scale purchases in the UK. In fact, disasters due to inadequate procurement have already been reported from the US.

Rarity of true costs

An additional problem is resource-based costing. In practice, estimates of costs based on resources are often hopelessly inadequate, as the institutions often doesn’t actually know the true costs of delivery. The actual costs of personnel and the all important issue use of buildings and accompanying resources is rarely calculable. Yet we know that many educational institutions, schools, colleges and especially Universities have woeful occupancy rates, making this a substantial figure. This is not easy as ‘rental’ rates are not applicable, as they are in many other areas of the economy but with a little effort one can calculate the costs of the building and its amortisation and maintenance costs. There may also be other hidden funding costs from Government or other sources. It is important to be exhaustive here. The true costs of any control group must be clear if cost-effectiveness is to be measured.

Once you have the true costs you really can determine a ‘cost per learner’, a metric I like. The next step is to also include the marginal cost per learner as further participants in online courses are normally at marginal costs. You can go further and look at the distribution of these costs but few get as far as this first metric.

Conclusion

The truth may be hard to bear here, but much educational research is meaningless in the sense of having no real chance of impact and change, as it does not carry through to Cost Effectiveness Analysis (CEA). The anti-corporate attitudes in our Universities is one problem, the lack of actual fiscal skills among researchers is another. But the main problem is the lack of commissioned research that demand rigour on costs. Without a truly rigorous Cost Effectiveness Analysis in education we will continue to spend huge amounts of money on fruitless research. The lesson is clear link effectiveness to costs. If you don’t the research will fall into the category of ‘inconclusive’. That means no evidence-based change will happen.