How to game Progress 8 and keep your professional self-respect

It’s a rule that any performance indicator with any consequence will eventually become corrupted, its human nature to find shortcuts and loopholes. Previous blogs here highlight how many commenters expect Progress 8 may be gamed, by putting every less academic child into History and Geography in the hope of scraping a grade that counts towards Ebacc. However, overall the new key performance indicator for schools Progress 8, seems likely to be much harder for schools to fiddle than the current 5ACEM.

This is because Progress 8 is based on an average and there are only two ways to move an average:

1. Improve a minority of results so much that their improvement could be redistributed to everyone and still show an improvement overall (a skewed distribution)

2. Improve everyone’s results

But there is another way likely to produce better Progress 8 results than other schools.

Progress 8 is all about getting better progress between KS2 and KS4 than achieved nationally. So, schools have to find ways of getting pupils to exceed the trajectory they seemed to be on at the end of Primary. A major problem with doing this is that KS2 is an extraordinarily powerful predictor of KS4 performance, meaning that we pretty much know with a high degree of certainty what a pupil is likely to get at KS4 because it is so closely related to what they get at primary…mostly….for most kids.

Mostly, and this is the rub, the loophole. It is a massively under recognised phenomena that how well you can predict KS4 outcomes goes up with KS2 performance. In other words, children who performed well at KS2 are more predictable than children who performed poorly.

A working paper from Prof Simon Burgess at the Centre for Market and Public Organisation at the University of Bristol, showed how well different levels of achievement at KS2 predicted KS4 performance, predictability goes up with performance (variability in outcomes goes down) (best explained in the graph).

http://cmpo.wordpress.com/2014/01/21/the-new-school-accountability-regime-in-england-fairness-incentives-and-aggregation/

This means that whatever it takes to do well at KS4 is less well related to low KS2 performance than high. So, another way of looking at it is to say high KS2 has a good chance of identifying high KS4 performers but low KS2 has much worse chances of identifying low KS4 performers.
One of the main things likely to determine KS2 performance is underlying ability, so we might say that high KS2 performance is a good indication of high ability (at least in ability to do national curriculum tests) and this seems to be borne out by the evidence as high KS2 seems to be a very powerful predictor of high KS4. However, low KS2 performance does not seem to be nearly as good a predictor of low KS4, indeed the evidence from Burgess indicates many low KS2 performers do very well at KS4.

This sort of situation is very common in research and is known as a Type I Error, where we think we have identified a true result but we have not, in our case where we think bad KS2 performance indicates kids are thick or disengaged when many are not.

Overall, the unpredictability of low KS2 children tells us something very important, that there are many reasons why children might not do well at KS2 and a sizable chunk of those reasons are not to do with the stuff it takes to do well at KS4.

And this gives us, in my view a hugely heartening way schools can get ahead of the curve in Progress 8. If schools put a lot of resources behind achieving good progress with their low KS2 attainers the national stats say they are more likely to have a larger number of bigger successes exceeding national expectation than if they put similar efforts into driving progress of their high KS2 performers.

One group of low KS2 attainers seems particularly likely to exceed their KS2 expectations, those with English as an Additional Language as they quite frequently make large progress at secondary after languishing in primary because for some reason their language ability suddenly seems to catch up at Secondary.

Of course, this strategy is highly questionable, indeed perhaps morally suspect as it involves restricting resources for some to give to others, not in their interest but for the purposes of improving schools stats. However, in the real world shortcuts will be found and if it is going to go on anyway schools focusing on their lower KS2 performers seems likely to only do good if it is not at too extreme a disadvantage of high attainers. Moreover, such a strategy will be held in check by schools also being held accountable by Attainment 8 which is best raised by some raw high attainment in turn best achieved by support of your highest KS2 performers.

There also seems something basically positive about the strategy of looking for wildcards among the low KS2 performers as it implies hope that early disadvantage does not determine outcomes.

Successfully getting low KS2 performers to beat their KS2 expectation seems likely to involve an ability to distinguish between those low KS2 performers secondary is unlikely to be able to reach and those ‘false positives’ those whose low KS2 incorrectly identifies their potential at KS4. Those schools good at spotting diamonds in the rough are going to find Progress 8 easy, those with prejudices about pupils with low KS2 achievements are going to struggle to compete.

Advertisements

One thought on “How to game Progress 8 and keep your professional self-respect

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s