Classroom Research and Cargo Cults
Hirsch, E. D., Jr., Policy Review
"We really ought to look into theories that don't work, and science that isn't science. I think the educational ... studies I mentioned are examples of what I would like to call cargo cult science. In the South Seas there is a cargo cult of people. During the war they saw airplanes with lots of good materials, and they want the same thing to happen now. So they've arranged to make things like runways, to put fires along the sides of the runways, to make a wooden hut for a man to sit in, with two wooden pieces on his head for headphones and bars of bamboo sticking out like antennas -- he's the controller -- and they wait for the airplanes to land. They're doing everything right. The form is perfect. It looks exactly the way it looked before. But it doesn't work. No airplanes land. So I call these things cargo cult science, because they follow all the apparent precepts and forms of scientific investigation, but they're missing something essential, because the planes don't land."
Richard P. Feynman, "Cargo Cult Science," Surely You're Joking, Mr Feynman!: Adventures of a Curious Character (Norton, 1985).
AFTER MANY YEARS of educational research, it is disconcerting -- and also deeply significant -- that we have little dependable research guidance for school policy. We have useful statistics in the form of test scores that indicate the achievement level of children, schools, and districts. But we do not have causal analyses of these data that could reliably lead to significant improvement. Richard Feynman, in his comment on "cargo cult science," identifies part of the reason for this shortcoming -- that while educational research sometimes adopts the outward form of science, it does not burrow to its essence. For Feynman, the essence of good science is doing whatever is necessary to get to reliable and usable knowledge -- a goal not necessarily achieved by merely following the external forms of a "method."
The statistical methods of educational research have become highly sophisticated. But the quality of the statistical analysis is much higher than its practical utility. Despite the high claims being made for statistical techniques like regression analysis, or experimental techniques like random assignment of students into experimental and control groups, classroom-based research (as contrasted with laboratory research) has not been able to rid itself of uncontrolled influences called "noise" that have made it impossible to tease out the relative contributions of the various factors that have led to "statistically significant" results. This is a chief reason for the unreliability and fruitlessness of current classroom research. An uncertainty principle subsists at its heart. As a consequence, every partisan in the education wars is able to utter the words "research has shown" in support of almost any position. Thus "research" is invoked as a rhetorical weapon -- its main current use.
In this essay I shall outline some fundamental reasons why educational research has not provided dependable guidance for policy, and suggest how to repair what it lacks. On a positive note, there already exists some reliable research on which educational policy could and should be based, found mainly (though not exclusively) in cognitive psychology. In the end, both naturalistic research and laboratory research in education have a duty to accompany their findings with plausible accounts of their actual implications for policy -- as regards both the relative cost of the policy in money and time and the relative gain that may be expected from it in comparison with rival policies. Including this neglected dimension might wonderfully concentrate the research mind, and lead to better science in the high sense defined by Feynman.
A tale of two studies
THE NOVEMBER 2001 issue of Scientific American includes an article called "Does Class Size Matter? …