In its idealized form, science resembles a championship boxing match. Theories square off, each vying for the gold belt engraved with "Truth." Under the stern eyes of a host of referees, one theory triumphs by best explaining available evidence--at least until the next bout.
But in the real world, science often works more like a fashion show. Researchers clothe plausible explanations of experimental findings in glittery statistical suits and gowns. These gussied-up hypotheses charm journal editors and attract media coverage with carefully orchestrated runway struts, never having to battle competitors.
Then there's psychology. Even more than other social scientists--and certainly more than physical scientists--psychologists tend to overlook or dismiss hypotheses that might topple their own, says Klaus Fiedler of the University of Heidelberg in Germany. They explain experimental findings with ambiguous terms that make no testable predictions at all; they build careers on theories that have never bested a competitor in a fair scientific fight. In many cases, no one knows or bothers to check how much common ground one theory shares with others that address the same topic. Problems like these, Fiedler and his colleagues contended last November in Perspectives in Psychological Science, afflict sets of related theories about such psychological phenomena as memory and decision making. In the end, that affects how well these phenomena are understood.
Fiedler's critique comes at a time when psychologists are making a well-publicized effort to clean up their research procedures, as described in several reports published alongside his paper. In fact, researchers generally concede that many published psychology studies have been conducted in ways that conceal their statistical frailty--and thus the validity of their conclusions. But Fiedler suspects the new push to sanitize psychology's statistical house won't make much difference in the long run. Findings published in big-time journals draw enough media coverage to bring the scrutiny of other researchers, who eventually expose bogus and overblown effects. "Advances in psychology will depend more on open-minded theoretical thinking than on better monitoring of statistical practices," he says.
When Fiedler gives talks to groups of psychologists, he tries to identify open-minded theoretical thinkers by posing a couple of questions.
First, he asks audience members to name a published study in which investigators uncovered an interesting, statistically significant effect that vanished in later reports. In a seminar conducted last year by Fiedler at a major Dutch university, 38 research psychologists had no problem citing flash-in-the-pan findings. Many remembered a well-known but now contested report that college students react to subtle reminders of old age by walking more slowly, allegedly because healthy young people unconsciously act out prompted stereotypes of the elderly (SN: 5/19/12, p. 26).
In that experiment, student volunteers were timed walking down a corridor after unscrambling sentences that, for one group, contained senior citizen-related words such as wrinkle and Florida. Researchers who conducted the investigation concluded that students weren't aware of having registered the stereotypical words, but still acted out an elderly stereotype by slowing their pace shortly after the reading exercise.
But researchers did not consider the possibility that their facial expressions or body language might subtly have encouraged the student volunteers to walk more slowly. They didn't ask themselves whether some students noticed elder-related words while unscrambling sentences and supposed that experimenters wanted them to mimic seniors. They did not explore whether some students quickly drew conclusions about what was expected of them and how to …