The Napkin Versus The Envelope

(And what it means for education research)

Of the many magnificent things humans do, amalgamation is one of the most important. We amalgamate observations into scientific laws. We amalgamate strangers into companies and nations. Both the how and what of how we amalgamate matters: The US President would be different, for example, were votes amalgamated according to numerical majority rather than electoral college.

However, this post is making a different point about amalgamation: that even the simplest act of amalgamation can be interesting, vexing, and perhaps even paradoxical. And that this matters.

To find out why, think about the following story.

Imagine you are a maths teacher, and your Head asks you whether the school should adopt a three-part lesson structure called 1–2–3 Math. Happily, you are well-connected on #edutwitter, and reach out to three mathematics education experts, Professors X, Y and Z for their opinion.

Professor X says no, don’t do it. (2–3 is fine, but 1 is a waste of time, they add.) Professor Y also says no, don’t do it. It’s looking bad for 1–2–3 Math, and whilst Y endorses 1–3, she rubbishes 2. But Z is no more positive, producing evidence that 3 has no place in your school’s practice.

No, no, and no, vote X, Y and Z.

“So they voted *against* 1–2–3 Math comprehensively, then,” summarises your Head with suave authority.

“Quite the opposite,” you reply, armed with a little knowledge from amalgamation studies and lucid contrarian instincts. “They voted collectively *for* 1–2–3 Math.”

“How can that be?” replies your Head, offering you this on the back of a napkin:

“How many experts voted for 1?” you ask.

“Two to one experts voted for 1…A majority.”

“And the other parts of 1–2–3 Math,” you say, offering this on the back of an envelope:

Head scratching follows. Amalgamate the votes one way, and 1–2–3 Math is a slam dunk. A majority of experts voted for 1, a majority for 2, a majority for 3. 3–0 in favour. Amalgamate the other way, 1–2–3 is a non-starter. They all advised against it. A 0–3 defeat. Horizontal voting or vertical? Napkin or envelope?

Clearly, then, evidence-basing institutions that amalgamate vast swathes of data need to know how best to do it, and ought make their methodology clear. The Education Endowment Foundation (EEF) is one such institution. The EEF even amalgamates education meta-studies with education meta-studies – that is, amalgamates amalgamations. It is a prime example of an institution that ought to be explicit and expert at all this – that is, explicit and expert at the basing part of their evidence-basing.

And yet, it seems this is far from the case. Teacher, blogger and tweeter extraordinaire Andrew Old (@oldandrewuk) has revealed that, on the contrary, the EEF seems to be amalgamating badly. So badly, they are not even always counting numbers correctly. He has posted multiple times about this. As a result, the EEF have in fact changed some of their workings – but doubts still remain. A good place to start for the details is this blog:

So.

Napkin or envelope?

How about neither?

Amalgamation requires disciplined and explicit expertise. I hope I’ve persuaded you this is an interesting and worthwhile thing to think about …and persuaded any one concerned with education that we are all owed better by the sector’s leadership institutions.

The 123 XYZ example is loosely based on this fascinating 2010 Philosophy Bites podcast with Philip Petit: http://philosophybites.com/2010/12/philip-pettit-on-group-agency.html

Related posts:

From the West covering the Victorian era to travelling towards the East voicing the Gandhian era, marriages have been regarded as a pious institution of an idealistic society. Culturally diverse and…

You know how they always say, “Go big or go home?” When I was going to graduate, my family’s idea of going big was to become a doctor. High grades were correlated with remarkable intelligence, and…

Uganda faces Human Resources for Health (HRH) challenges that undermine its ability to achieve universal access to health care for all of its citizens. Nationwide, only 73% of the available health…