The Problem With Research

Teaching is such a “particularistic endeavor” that guiding teaching practice by one-size-fits-all ‘best-evidence’ research does us no favours.

In Transformative Assessment, W. James Popham (2008) urges caution when it comes to getting fussy over ‘research’:

Teaching, in short, is such a particularistic endeavor that, with remarkably few exceptions, the best we can ever derive from educational research investigations is probability-based instructional guidance.

We are currently obsessed with the word ‘research’ in education and now it has become an industry for conferences and consultants who love to tell us what to do and what the ‘research says’. Research can make anything sound fancy and ‘prove’ all manner of things, including whether the facial hair of a headteacher influences a school’s effectiveness.

But as those of us who have actually taught in a variety of schools know, what may work in one educational context can fail in another.

I was part of a research study that looked into the use of puppets to promote engagement and talk in science.

This was a brilliant project to be involved in and as a teacher with a passion for science it was a real privilege. Did it work? The research provided evidence of an increase in discourse focused on argument and reasoning, and a positive impact on children’s engagement and motivation. But not in all cases.

The puppets were a magnificent success in two of the schools I worked in. They were far less successful in two other schools I moved to. I was convinced they would work anywhere but they didn’t and for various reasons related to those particular contexts. The rich and varied knowledge base in one school was not something that easily translated to another.

So when we ask the question “What works in education?” we can’t ever be sure. Something will probably work somewhere. You could look at the What Works Centre for Education, the Education Endowment Foundation (EEF), or the What Works Clearinghouse (WWC) for guidance but that’s all it is, guidance. They may make recommendations, based on their reading and interpretation of multiple research studies but it is always possible to challenge every single piece of it. There will always be some place where the recommendations don’t gel.

But what happens when someone says the research is “valid and reliable”? Then I’d want to know more. Do teachers realise that validity is not binary and that reliability has three different forms? I doubt it. Reliability and validity are conditional on the specific samples investigated.

We need to stop getting precious about the research and equating it to some sort of medical model. Education is not a science where we find cures. Research can inform, nothing more. Evidence-based schools are really confusing teachers because they have the language wrong.

Some professionals use the terms evidence-based practice and evidence-informed practice interchangeably without thinking much about what they mean.

Evidence-based research is conducted through validated scientific processes. Evidence-informed means using research that is already available and has been tested, tried, and true. This evidence is then combined with the experiences and expertise of an  organisation to best fit the population served.

Paul Kirschner and Mirjam Neelen offer the clearest explanation of why we should stop saying ‘evidence-based’ when talking about all things education:

“Evidence based is a term and a method that comes from medicine and pharmacy. This approach has a number of characteristics which allows people (practitioners, hospital boards,…) to make what is known as evidence-based decisions with an incredible degree of certainty. For example, whether a drug works or not doesn’t depend upon who administers the pill, whether a patient is happy or sad, and/or whether or not (s)he is wide awake or dead tired when it is taken, and so forth. Learning and instruction/training is a whole different thing. Whether an approach to instruction or an or intervention works and to what degree it works differs from teacher to teacher, from context to context, from content-domain to content-domain, and so forth. It’s effect also varies depending on the characteristics of the learner and his or her cognitive, physical or emotional state, the class/learning group and the environment outside of that unit, and so further. For this reason we chose to use the term evidence-informed. We want to help you make evidence-informed instructional or training decisions by informing you in this blog about what works, what doesn’t work, when and for what groups this is the case, etcetera.”

Research can prove anything and it often does. Look at all the zombie ideas education has fallen for over the decades because we fell for ‘the research’ even though that research was paper thin.