The power of Random in Randomised Control Trials

In all the hubbub about education research and Randomised Control Trials I have seen few signs many fully grasp the real power of this technique – randomisation of who gets the program and who does not. I think closer consideration will excite educationalists at its potential to cut through the fog to give us a glimpse of the truth.

One of the main criticisms of education research is that education is so complex and affected by so many things that most are sceptical any research technique could ever hope to ensure that the responses of one group of pupils are informative of the likely responses of others.

How can the circumstances for one group possibly be the same as the circumstances for another, there are so many variables at play? How can we generalise the results of a specific group to others when every kid is different?

It seems especially baffling how anyone can claim that they can ensure a studied group and comparison group experience a similar level of things that are are unknown or unknowable e.g. home circumstances, hormones, brain chemistry, attitude to teachers, being bullied at lunch time etc.
If we can’t claim the group being studied experiences the same stuff as the comparison group in all ways likely to effect results, then differences between their outcomes might always be caused by these unknown differences between the groups.

But this is precisely what randomisation allows us to do.

The best way I can think of putting the case of how randomisation can do this is from a time when we knew as much about medicine as we currently do about learning.

Once upon a time a doctor in London was concerned about the high number of people dying from an unknown disease that seemed to affect bigger cities especially as they grew. The prevailing view at the time was that this disease was transmitted through bad air called ‘miasma’. However, this did not satisfy the doctor as it seemed to afflict specific neighbourhoods and bad air seemed unlikely to be restricted to streets. So he investigated things in more detail and spoke to the people affected and by so doing got to hear of many more of the cases not reported. To see if the outbreak was localised he plotted the home addresses of the afflicted on a map and they all seemed to cluster around a central point ‘Broad Street’, in Soho. At Broad Street there was a water pump which made the doctor suspect the disease came from the water. He looked very carefully at the water through a microscope but could not identify anything that could cause the deaths. He presented his evidence to the powers that be, of maps showing incidents centred on the water supply but his theory was pooh-poohed by the authorities and his peers.

Up to this point the doctor was playing by the rules and tried to persuade with indirect evidence that supported his theory, but it did not work. So he pulled the handle of the water pump off; after which there were no more cases of the disease we now call …cholera.

It’s easy to scoff at theories after they have been disproven. Microscopes were not powerful enough at the time to see the cause the disease, bacteria, so the culprit was effectively invisible. It was 30 years before the little blighter was identified.

It seems we are in a comparable territory in education research where there are competing theories with little definitive evidence either way or no explanation at all and we are frequently being asked to believe weird left field explanations for which there are only circumstantial evidence to back them up.
I suggest it would be useful to try put yourself in the shoes of the naysayers at the time of the doctor’s depositions. Ask yourself, would you have been persuaded by bunch of clever statistics suggesting some invisible mysterious beasty, never seen before, was in the water or would you have been persuaded by the fact the disease stopped when the water was cut off.

The power of what the doctor did was by removing the pump handle he made it plain to see that the water had something do with it. This was a form of Randomised Control Trial, where everyone else in London was the control group, that is, the rest of London were exactly the same as those receiving treatment in every respect apart from one thing, drinking water from Broad Street. And this is the power of randomised control, but I suggest there is no way around it, we have to use reasoning to see it; there was no difference between the residents of Broad Street and the residents of other areas.

May I suggest it’s also a valuable exercise if you try to think up some reasons why the people living around Board Street might be different from others or might be especially susceptible to a disease, you might speculate it has something to do with it being in Soho, an area of theatres or a particularly impoverished part of London. I hope you notice how easy it is to come up with a number of plausible reasons why this neighbourhood might be more sickly than others. However, apart from the deprivations of poverty making residents less able to fight off bacteria we now know all those plausible explanations were wrong. Cholera is caused by ingestion of the bacterium vibrio cholerae that thrives in tiny plankton feeding off human sewage. The Broad Street plumbing had been badly made and poorly maintained allowing sewage to leak into the drinking water.

The doctor (John Snow) had not proved that this disease was caused by bacteria, the idea that disease-causing micro-organisms existed was not around until many years later, he had not proved the theory of bad air was wrong, indeed many diseases can be transmitted through the air. But he had dramatically narrowed down the possibilities for this one disease. Whatever it was, the cause was probably in the water.
He was searching for causes at a time before an understanding of the mechanisms was possible. And this is where we find ourselves in education research. So little is known it’s very unlikely that full explanations will become available in the foreseeable future, the best we can hope for is the narrow down the possibilities.

And this is the real power of randomised control studies.

It works best when we have little or no clue what’s causing anything. The technique gets us round this huge problem by fact that if we study people selected at random there can be no selection biases and any and all factors influencing results of the population we drew them from will be included in the group we study. This is because the chances of any conditions, factors or individual differences being present in the group being studied is exactly equal to the chance of them occurring the group you draw the study group from.

More technically, randomisation exploits one of the most powerful and frequently occurring patterns observed in nature, central limit theorem, which states that if you a pick, at random, a large number of examples/measurements of things (people) influenced by many different things, they will always end up with scores that spread out around the average in regular pattern. The proportion of scores close the average will always be the same and the proportion of scores a long way from the average will always be the same, known as the normal distribution or bell curve. This means that if you pick a large number of people out of population at random their measurements will have the same average score and vary from the average the same as the population from which you drew them from.

And this the beauty and power of randomised control studies, you don’t need to know everything about those you study and you don’t need to worry about the things you don’t know because they are going to be equally present in your study group as in your comparison group so the differences will cancel each other out.

So I put it to you that if you select a largish group Year 11s at random from a state school with average attainment, average proportion of pupil eligible for free school meals, ethnic groups and pupils with English as an additional Language, those pupils will respond in a way fairly comparable with most Year 11 pupils in the UK.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s