The Dec. 13 issue of The New Yorker has a provocative article about scientific “truth” by Jonah Lehrer, “The truth wears off: is there something wrong with the scientific method?” (You’ll need a subscription to read the whole thing.) It’s about what Lehrer calls the “decline effect”: the fact that an initial demonstration of something in science tends to be weakened or even disappear when later workers try to replicate it.
Lehrer gives lots of examples—including the bogus demonstrations of ESP by J.B. Rhine at Duke—but concentrates on more recent studies. One is from evolutionary biology: the work of Anders Møller on fluctuating asymmetry (FA) in barn swallows. FA is the phenomenon in which a trait of an individual is asymmetrical in random ways (e.g. the right side may be larger or smaller than the left). In the case of barn swallows, FA estimates the difference in length of the long feathers in their forked tails. In 1991, Møller showed that females prefer to mate with males having more symmetrical tails (less FA), presumably because FA was an index of the genetic quality of a male (more FA, worse genes).
Initial studies of other species showed a similar negative relationship between FA and fitness, but then the effect began to decline: by 1998, fewer studies of FA showed positive effects, and the effects that were demonstrated became smaller.
Lehrer cites another study of many results in ecology and evolution:
In 2001, Michael Jennions, a biologist at the Australian National University, set out to analyze “temporal trends” across a wide range of subjects in ecology and evoutionary biology [see reference below]. He looked at hundreds of papers and forty-four meta-analyses (that is, statistical synthesis of related studies), and discovered a consistent decline effect over time as many of the theories seemed to fade into irrelevance. . . . Jennions admits that his findings are troubling, but expresses a reluctance to talk about them publicly. “This is a very sensitive issue for scientists,” he says. “You know, we’re supposed to be dealing with hard facts, the stuff that’s supposed to stand the test of time. But when you see these trends you become a little more skeptical of things.”
What causes these “declines”? Lehrer suggests that it’s a combination of two things. The first is publication bias: an initial enthusiasm for publishing only positive results. (I’d add that this bias could swing toward publishing negative results after an initial discovery becomes something of a dogma: researchers are then motivated to disprove it.) The second is “selective reporting”: not overt fraud, but what Lehrer characterizes as “one of subtle omissions and unconscious misperceptions, as researchers struggle to make sense of their results. Steven Jay Gould referred to this as the ‘shoehorning’ process.’”
This, then, is not (as Lehrer’s title implies) an indictment of the scientific method of testing and replication per se, but of scientists and the culture of science. Lehrer winds up criticizing the “slipperiness of empiricism,” suggesting that much of what we “know”—even about things like the strength of gravity and the weak coupling ratio of neutrons—may simply be wrong. His ending is provocative:
The decline effect is troubling because it reminds us how difficult it is to prove anything. We like to pretend that our experiments define the truth for us. But that’s often not the case. Just because an idea is true doesn’t mean it can be proved. And just because an idea can be proved doesn’t mean it’s true. When the experiments are done, we still have to choose what to believe.
I tend to agree with Lehrer about studies in my own field of evolutionary biology. Almost no findings are replicated, there’s a premium on publishing positive results, and, unlike some other areas, findings in evolutionary biology don’t necessarily build on each other: workers usually don’t have to repeat other people’s work as a basis for their own. (I’m speaking here mostly of experimental work, not things like studies of transitional fossils.) Ditto for ecology. Yet that doesn’t mean that everything is arbitrary. I’m pretty sure, for instance, that the reason why male interspecific hybrids in Drosophila are sterile while females aren’t (“Haldane’s rule”) reflects genes whose effects on hybrid sterility are recessive. That’s been demonstrated by several workers. And I’m even more sure that humans are more closely related to chimps than to orangutans. Nevertheless, when a single new finding appears, I often find myself wondering if it would stand up if somebody repeated the study, or did it in another species.
But let’s not throw out the baby with the bathwater. In many fields, especially physics, chemistry, and molecular biology, workers regularly repeat the results of others, since progress in their own work demands it. The material basis of heredity, for example, is DNA, a double helix whose sequence of nucleotide bases codes (in a triplet code) for proteins. We’re beginning to learn the intricate ways that genes are regulated in organisms. The material basis of heredity and development is not something we “choose” to believe: it’s something that’s been forced on us by repeated findings of many scientists. This is true for physics and chemistry as well, despite Lehrer’s suggestion that “the law of gravity hasn’t always been perfect at predicting real-world phenomena.”
Lehrer, like Gould in his book The Mismeasure of Man, has done a service by pointing out that scientists are humans after all, and that their drive for reputation—and other nonscientific issues—can affect what they produce or perceive as “truth.” But it’s a mistake to imply that all scientific truth is simply a choice among explanations that aren’t very well supported. We must remember that scientific “truth” means “the best provisional explanation, but one so compelling that you’d have to be a fool not to accept it.” Truth, then, while always provisional, is not necessarily evanescent. To the degree that Lehrer implies otherwise, his article is deeply damaging to science.
UPDATES: Over at Wired, Lehrer explains his thesis a bit more and responds to readers’ questions.
I believe the reference to the “2001″ work by Jennions is actually this two-authored paper published in 2002: Jennions, M. D. and A. P. Møller. 2002. Relationships fade with time: a meta-analysis of temporal trends in publication in ecology and evolution. Proc. Roy. Soc. B. 269:43-48.