Philip Dixon challenges Gerry Holtham’s case for measuring school performance to drive up standards

June 28th, 2011

When you are censured by one of the great and the good of Welsh society it’s difficult to know if you should feel flattered or flustered. As Chair of the eponymous commission, Professor Holtham deserves the respect and gratitude of all in Wales for pointing out how we are short changed by the current Barnett formula. But even the great and the good can get it wrong – our present celebrity culture is only the worst manifestation of the fallacy that if you’re outstanding in one area then you must be outstanding in all others.

IWA and Cardiff University Lecture

6pm Wednesday 29 June, Julian Hodge Lecture Theatre, Cardiff Business School

Education Minister Leighton Andrews AM

Improving School Standards

In February the Permanent Secretary predicted that the Welsh Government’s robust response to the poor PISA education attainment published last December will prove a transformational moment in the first 20 years of devolution. In this keynote lecture for education policy in the fourth Assembly, the new Welsh Government’s Education Minister Leighton Andrews will set out in detail how he intends to make this happen.

To attend the lecture register here.

In an article in last week’s Click on Wales Professor Holtham made a plea for League Tables. Not crude ones it must be confessed. To be precise Holtham called for Value-Added ones, but they are League Tables nonetheless. He accused those who were less enthusiastic about such a proposal of  being ‘pious’, without argument, and, in the case of myself and the leader of the NUT in Wales, just plain self-interested – concerned to protect our members from scrutiny. A Value Added League Table would be simple, provide parents with clear information about the health or otherwise of their school, and thus, it was implied, provide the mechanism for driving up standards. All so simple and clear it seems – but so wrong. There are those who would defend the Barnett formula along the same lines. It’s simple and it’s clear – Wales gets its percentage and we can know quite quickly what the Welsh Government will get as a consequential. But as Professor Holtham showed, it’s wrong. The same is the case with League Tables.

The simplest and clearest League Table of all would be one based on crude results – the scores on the doors for GCSEs. It’s the model that operates in England, and it’s clear and simple. But it doesn’t really tell us much more beyond where the rich kids live, and we knew that anyway.

Professor Holtham doesn’t believe in this level of simplicity. What he wants is a table that’s based on value added. As he notes, children come to schools with all sorts of advantages and disadvantages. What needs to be measured is the effectiveness of the school – the value it adds. It sounds fairly simple and fairly clear – until you delve further. How do you define value added? What do you want to build in? Free School meals are a good indicator of deprivation – until you look at why free school meals are not taken up. What about those pupils whose first language isn’t English – shouldn’t that go into the mix too? What are the baselines? Progression from key stage to key stage? Seems clear and simple until one delves underneath and finds that Key Stage 2 data, when children move from primary to secondary school, is notoriously unreliable. But as Professor Holtham presciently notes, these are simply technical difficulties which could be ironed out.

So now we present parents with some comparatively complex data. Not crude scores but a more nuanced account which shows crude exam outcomes balanced against value added,  which itself is the product of some not so simple gearing of other factors. In Professor Holtham’s ideal world, parents would sit down of an evening and compare the relative achievements of the schools in their catchment area, and weigh up crude results against value added. And then they would choose a school. Well they could if they live in certain parts of Wales, but in others kids go to the local school because the logistics in travelling to the next nearest are a nightmare. Many believe that between 30 and 40 per cent of children are affected by limitations of geography, and yet more will be constricted by reasons of poor transport or poverty.

But all this aside, the real Achilles heel of Professor Holtham’s argument lies in the unstated and, I suspect, unquestioned assumption that a market based on choice will drive up standards. Such causality is by no means assured. The information provided by the league tables may be but partially understood. Those with less understanding will be at more of a disadvantage than those with more. The numbers who can choose school X are constrained by its physical limitations. And choice is always exercised by the relatively powerful. In this imperfect market schools can cascade down the spiral still further and produce worse results for their pupils, as has been the case in some parts of England.

Instead of raising standards across the board league tables can increase the gap between high and low performance. This coupled with the facts of geography, which mean that for many children there is just one monopoly supplier, surely challenges us to find a different model. I am presuming, of course, that we want to improve the chances of all pupils and not just the few.

So shall we then continue on our complacent way in Wales? I agree completely with Professor Holtham – we can’t afford to. He is right to insist that we need more accountability, and I am grateful to him for smoking me out on the need for metrics. To be explicit, we need metrics which measure performance. But it’s how that’s done that’s important. The data has to be interpreted effectively.

We might want all statistics to be simple and clear, but they rarely are. And so we do have to be careful how the data is put in the public domain. This is not some patrician disdain for the hoi polloi that Professor Holtham seems to discern – being the son of a postman would hardly qualify me to take such a position. Instead, it is borne of a deep seated concern that the data is used effectively by those in the system to drive up standards.

Professor Holtham takes exception to my use of the word ‘shame’. Well, again for the record, I don’t think that shame is the sort of motivator I’d want to encourage and I can’t believe it would be high in the arsenal of Professor Holtham’s effective management tools. And as I’ve said, exactly how the publication of the data in the form of a league table would be helpful in driving up standards is something that must be proved not assumed.

Most of the unions are supportive of an agenda that seeks to enhance and strengthen the accountability of schools. The PISA results were a wake up call to all those engaged in education. We are cooperating with the Welsh Government in developing new proposals for the banding of schools, which will challenge them all whatever their circumstances to become the best they can be. There will be no hiding place for the complacent. That’s why the phrase ‘intelligent accountability’ is more than just a sound bite. It means a use of data that is sophisticated, accurate and challenging rather than a simplistic ranking which might shame but won’t motivate. Leighton Andrews is right to insist that we get our eye back on the ball and create a much better education system. But as for league tables – keep them for football.

Tags: , , , , , , ,

Philip Dixon is Director of ATL Cymru.

5 Responses to:“League tables should be kept for football”

  1. i b jones says:

    It’s good to see that Mr Dixon is in favour of the use of data that is sophisticated, accurate and challenging. With this in mind could I refer him to a paper published in 2010 by Simon Burgess, Deborah Wilson and Jack Worth from the University of Bristol which tests the hypothesis that the publication of school performance tables raises school effectiveness. They compared outcomes in England and Wales before and after the abolition of school tables and following a rigorous analysis of the data they summarised their conclusions thus:

    ‘We find significant and robust evidence that this reform (ie abolition of school tables) markedly reduced school effectiveness in Wales’.

    We have all been shaken by the PISA results. Let’s hope that Gerry Holtham, the University of Bristol, Ciaran Jenkins of the BBC and others will continue to hold the Welsh government and its partners to account.

    (Report comment)

  2. Billy Pilgrim says:

    The paper cited above refers to two identical education systems operating in England and Wales during these years with the exception of the publication of league tables.

    I can think of a few differences that ocurred during this period – a massive school building programme in England and a £600 per pupil funding gap between Welsh and English pupils. It is just as logical to claim that it is this that is accountable for performance differences.

    It is the epitomy of irony that this paper was produced in Bristol where choice has distorted the schooling system so badly that there are more independent than state secondaries.

    (Report comment)

  3. Jeff Jones says:

    This proves once again that the last person anyone who is interested in public service reform should listen to is a representative of the producers. Gerry Holtham in his original article was spot on. Too many Welsh schools are sadly for both the Welsh economy and the children of Wales are coasting as the valued added tables clearly showed. It is a scandal that one of the comprehensives in my own local authority has just 29% of its pupils obtaining 5 GCSEs including Maths and English at grade C and above. Basically 81% of the pupils by the age of 16 are being condemned to a life of either unemployment or if they do work then it will be in an occupation which is low skilled and therefore low waged. Frankly as a taxpayer I have the right to know how my local school is performing.

    (Report comment)

  4. gerald holtham says:

    It seems that Philip Dixon and I have lot of views in common as well as one or two where we apparently disagree. But I want to clear up one misconception. I would not argue that value-added tables should help parents to shop around for a school. I accept the point that many parents have little choice for geographical or other reasons. Anyway as I pointed out the shoppers-around will go for crude results not value-added every time. There are two classic responses for members of an organisation who are disatisfied:- voice and exit. The alternative to leaving is staying and pressing for improvement. Parents are in a better position to do the latter, use their voice, if they have comparative information to add to their own experience. I also accept that the present value-added measures may well need refinement. But I still don’t understand why such refined measures should not be available to parents. We talk of “challenge” but publication would be likely to make challenge a reality and increase pressure for improvement. And I write not as an expert, which I am not in this area, but as a citizen with grandchildren going through the system.

    (Report comment)

  5. i b jones says:

    In respect of Billy Pilgrim’s comments, the Bristol study was able to isolate the impact of league tables by comparing England and Wales over two periods ie before and after the abolition of school tables in Wales. Indeed, the researchers were able to conclude that different resource levels did not explain the variances in performance. This is a serious piece of work and one can only conclude upon reading it that the Welsh government made a critical error in abolishing rather than refining league tables. A generation of school students have, as a result, seen their examination grades diluted. Thankfully, this era of information famine is behind us because, whatever the Minister announces tomorrow, the Welsh media (BBC Wales in particular through freedom of information requests) have now latched on to this debacle and will, quite rightly, publish whatever they can.

    (Report comment)

Have your say

Please let us know in your message if you do not want the IWA to contact you in future or related IWA activity.