RAISE in your allowance

It’s a weird thing how much people’s attention is draw to anything highlighted in red. OFSTED/RAISE have just introduced colour coding to perhaps the most important statistic in school education today, the gap between Pupil Premium and others.

Highlighting anything in red in performance reports ought to be handled with a great deal of care because such symbolism can have a lot of influence on judgements. They may even affect thinking more than the numbers.

Imagine if your inspector turns on RAISE just before visiting and sees lots of red, it seems likely this will have a big effect on their attitude to your school.

So, we would expect that the rules governing when something shows up red would be very carefully thought through and developed from detailed professional advice. It appears that this has not happened.

The rules for colouring your RAISE red are:

KS4 – Percentage of disadvantaged represented by 3 pupils

If you have 100 disadvantaged pupils your figures will go red if the gap in percentage achieving expected progress between Pupil Premium and others is 3% or more than the national percentage achieving expected progress.

If you have 10 disadvantaged pupils it will go red if the gap is 33% or more than the national.

(If you want to calculate for yourself the gap you can have before your figures goes red, divide 3 by the number of Pupil Premium children you have at any given prior attainment level, then times that by 100 e.g. 3 / 100 (made up number of Pupil Premium at Level 1) = .03, X 100 = 3%)

KS2 – Percentage of disadvantaged represented by 1 pupil

If you have 100 disadvantaged your figures will go red if the gap in percentage achieving expected progress is 1% or more than the national. With 10 disadvantaged, you go red if the gap is 10% or more than the national.

So, we can see your “allowance” gets bigger the fewer disadvantaged pupils you have.

This is done because the results for a small numbers of kids can vary a lot, one child among a small group doing very well or very badly will completely change the figures for the group. The results for one pupil will not be a fair reflection of schooling overall.

If you only had 3 disadvantaged kids, failure to get one to achieve expected progress would drop your success rate from ‘perfect’ 100% to 66% which looks really poor.

Peter Nye and another statistics character James Pembroke have already commented both saying there is huge problem here for small schools and those with small numbers of disadvantaged kids.

“… potential to create unjustifiably negative impressions…” Peter Nye

Peter Nye has a well worked out example of the problem

http://peterjnye.wordpress.com/2014/10/31/raiseonline-closing-the-gaps-and-the-introduction-of-shading/

James Pembroke has an enjoyable rant

http://sigplus.blogspot.co.uk/2014/11/mind-gap-devilish-detail-in-raise.html

Peter Nye has pointed out most primaries will have small numbers of disadvantaged in the test year 2 and 6 so the potential for confusion will be common.

So why have we been lumbered with this additional complexity, which seems to me and other statos to be difficult to get our heads around and very likely to be misleading for small schools.

This appears to have happened because a body of opinion in education has succeeded in persuading the powers-that-be, that education is somehow different from all other fields of human activity that use numbers to figure out whats going on. They appear to have persuaded the establishment that the method all other professions use for circumstances like these is not appropriate for education. The normal way quantitative professionals in all fields deal with the problem of interpreting small numbers of figures is statistical significance.

Educational significance vs statistical significance

A main problem educationalists seem to have with statistical significance is that it can’t handle (or rather won’t) situations where there are very small numbers of children. This is because results for small numbers of kids are likely to vary a lot and significance just refuses to work properly when data is messy and it hasn’t got enough info to go on. However, these small numbers of children are still regarded as of ‘educational significance’ because each child’s education is important.
Moreover, it just happens that a lot of primaries are small and the numbers of disadvantaged in Year 2 and 6 will often be tiny. So OFSTED have a real problem as they want us to focus on narrowing the gap but very many of the places were the narrowing is supposed to take place have too few disadvantaged kids to identify trends.

When I studied statistics I was always told “never interpret any data on less than 30 people”. 30 was treated like magic number. Through some mystical force of some cosmic meaning trends don’t tend to appear with any reliability without gathering at least this much info.

No doubt, education cannot refuse to interpret what’s happening with groups less than 30 because as Peter Nye points out that would mean we could not draw conclusions about the teaching of disadvantage in most small primaries.

I feel a practicable step would be to say “ ignore any RED statistic about a group of less than 10”. This achieves both goals of a) encouraging us to look for trends in teaching of key groups like low prior attaining Pupil Premium whilst b) applying the appropriate professional discipline to refuse to talk about trends when we don’t have enough info.

I would say that you can accept the RAISE gap going RED for less than 10 kids only if all or the vast majority achieved less progress than the national. Likewise you can only demand credit if all or the vast majority of less than 10 kids achieve more progress than the national.

I think we should tell OFSTED to go take a running jump on this one. If you get into a conversation with inspectors about why the gap between the 4 kids who started on Level 1 and the national is RED, push back and say it’s meaningless, there is not enough info to support conclusions about the teaching. Instead pull out your transition matrices for your disadvantaged kids and talk about each child on an individual basis. If you can give an individualised account of the story for each Pupil Premium child with low prior attainment in Year 2 and 6, which you should be able to do with less than 10, you are showing that your mind is firmly focused on narrowing the gap.
Statistics can’t help us with individuals and they should not be forced into trying, its down to professional judgment case by case.