Tests of the supernatural fail again: new study can’t replicate findings of precognition

Who says that you can’t test the supernatural?  Intercessory prayer, near-death experiences, and ESP—all have been tested (and refuted) using science; all are classical “supernatural” phenomena whose mechanisms, if they existed, would seem to defy the laws of physics (I’m not going to get into arguments about the definition of “supernatural” here).  And now there’s a new paper in PLoS ONE by Ritchie et al. (free download at link, reference below) that refutes a recent paper presenting evidence for precognition: the idea that somehow one could have intimations in the present about stuff that hasn’t yet happened.

The original paper, published in 2011 in the Journal of Personality and Social Psychology by Daryl Bem of Cornell University (download the paper here, and see my post on it here), gave statistically significant evidence for precognition in several experiments.  In brief, experimental subjects who were asked to memorize a list of words, and then type as many as they could remember onto a computer, did better at remembering those words to which they were subsequently exposed when presented with random selections of the initial word list and irrelevant “control” words. This implied that seeing the words later increased one’s ability to remember them in the past.

The paper, appealing as it did to many people’s love of psychic stuff, got a lot of attention; it was, I believe, a subject on my radio interview with woo-meister Alex Tsakiris at Skeptiko. (Alex loved it of course.)

Bem’s experiment was criticised by other scientists, and I think there are still some attempts to replicate it in the works; my own judgment was that the results couldn’t be replicated by others.  That seems to be the lesson of the paper by Ritchie et al., who took Bem’s most significant experiment and replicated it three times in three different laboratories: The University of London, The University of Edinburgh, and the University of Hertfordshire.

The results are simple: none of the three replications achieved anything near statistical significance. The respective probability values (the values that results as extreme as those seen could be due solely to chance) were 46%, 94%, and 61%; the overall probability was 83%.  For “one-tailed” tests like these, results are considered significant only if the probability of attaining them by chance is 5% or less; and the replication results didn’t even come near that threshold.  Conclusion: Bem’s results are severely in question.

What happened in Bem’s study if his results really were wrong? Ritchie et al. have several theories:

  • There were statistical and methodological “artifacts” outlined by several critics (see references 2, 3, 5, 6 and 7 in their paper)
  • Other variables, not recorded by Bem (subjects’ use of self-hypnosis or meditation, anxiety level, etc.) could have been responsible for the results. I don’t really understand this criticism because it seems that the “supernatural” character of precognition would be unaffected by those variables
  • the effect might be genuine but is hard to replicate. Ritchie et al. note that this is a common claim by psi advocates when results aren’t replicated. It’s like theologians who say, “God cannot be tested.”

The authors favor the hypothesis that Bem’s original result was due to “experimental artifacts.” They also note that there is at least one other published report of a failure to replicate Bem’s response: the paper by Robinson (2011) cited below.  The PLoS paper ends with a cute conclusion:

At the end of his paper Bem urges psychologists to be more open towards the concept of psychic ability, noting how, in Alice in Wonderland, the White Queen famously stated, ‘Why, sometimes I’ve believed as many as six impossible things before breakfast’. We advise them to take a more levelheaded approach to the topic, and not to venture too far down the rabbit hole just yet.

Bem has published a response to Ritchie et al.’s piece: it’s basically a non-response, calling for more work and floating the possibility that the negative attitudes of Ritchie et al. could have had an effect on their results (that, too, would be a paranomal result). As Bem said, “Ritchie, Wiseman, and French are well known as psi skeptics, whereas I and the investigators of the two successful replications are at least neutral with respect to the existence of psi.” That’s a pretty lame defense. Why would you re-test someone’s results if you weren’t a skeptic? On Thursday Ritchie et al. published a response to Bem’s critique.

An interesting side note: Chris French, one of the authors of the Ritchie et al. paper, wrote a piece in the Guardian, “Precognition study and the curse of the failed replications,” giving his take on Bem’s study and describing their own difficulties in getting their failure of replication published. It was rejected by three journals, including the original journal—the Journal of Personality and Social Psychology—before it was finally accepted in PLoS ONE!  The unwillingness of the original journal’s editor to even send Ritchie et al.’s paper out for review is reprehensible, particularly in light of the splash made by Bem’s paper.  Extraordinary results deserve extraordinary scrutiny.  As French notes:

This whole saga raises important questions. Although we are always being told that “replication is the cornerstone of science”, the truth is that the “top” journals are simply not interested in straight replications – especially failed replications. They only want to report findings that are new and positive.

Most scientists are aware of this bias and will rarely bother with straight replications. But straight replication attempts are often exactly what is required, especially when dealing with controversial claims. For example, parapsychologists are typically happy to accept the findings of a new study if it replicates a previously reported paranormal effect. However, if it fails to do so, they are likely to blame any deviation from the original procedure, no matter how minor. It was for this reason that we chose to follow Bem’s procedure as closely as possible (apart from a minor methodological improvement).

Given the high cost of paper publications and the high submission rejection rate of “top” journals, it might be argued that rejecting replication studies was defensible in the pre-internet era. But what would prevent such journals from adopting a policy of sending reports of replications, failed or otherwise, for full peer review and, if accepted, publishing the abstract of the paper in the journal and the full version online? Otherwise, publication bias looks set to remain a major problem in psychology and science in general.

Doubt and replication are the sine qua non of science, and journals must always send out failed attempts to replicate for peer review, and find a way to publish them if they’re sound.

h/t: Diane G.

__________

Bem, D. J. 2011. Feeling the future: experimental evidence for anomalous retroactive influences on cognition and affect. J. Personality Social Psych. doi: 10.1037/a0021524:DOI: 10.1037/a0021524.

Ritchie, S. J., R. Wiseman, and C. C. French. 2012. Failing the future: three unsuccessful attempts to replicate Bem’s ‘Retroactive facilitation of recall’ effect. PLOS ONE 7(3): e33423. doi:10.1371/journal.pone.0033423

Robinson E. 2011. Not feeling the future: A failed replication of retroactive facilitation of memory recall. J. Soc. Psychical Research 75:142-147.

46 Comments

  1. Matthew Cobb
    Posted March 18, 2012 at 5:59 am | Permalink

    “The unwillingness of the original journal to even send Ritchie et al.’s paper out for review is reprehensible”

    +1 – But be careful what you wish for. Rosie Redfield’s attempted replication of the arsenic life study was submitted to Science (where the original article was published), and they correctly sent it out for review.

    Redfield has posted the reviews on her website, and as one of the acerbic but positive reviewers puts it:
    “It seems a pity that one can sell newspapers by both committing and reporting a crime”

    http://www.zoology.ubc.ca/%7Eredfield/ScienceReviews.pdf

    PS Wow that wordpress login thing is complicated!

    • Torbjörn Larsson
      Posted March 18, 2012 at 10:44 am | Permalink

      Thanks, I enjoyed that!

    • Torbjörn Larsson, OM
      Posted March 18, 2012 at 10:51 am | Permalink

      It is not complicated, it is baring me. When I log in on my WP account here, this site reply:

      “Error Accessing Dashboard

      You are logged in as “torbjornlarsson” and do not not have the necessary privileges to access the dashboard for “Why Evolution Is True”. Please ask an administrator of the site to invite you.”

      See, simple, is it “not not”!?

      The complicated question is whether to hound Jerry with (one among many?) invitations as “follower” (I assume). What is WEIT’s new policy?

      • Torbjörn Larsson, OM
        Posted March 18, 2012 at 10:53 am | Permalink

        Eh? And now I _am_ privileged. Truly.

        Very well, maybe this will work anyway. Plz continue as you were!

      • Posted March 18, 2012 at 7:09 pm | Permalink

        I just click the “Like” button below the article and enter my credentials into the overlay. It doesn’t even take me away from the article.

        • Diane G.
          Posted March 18, 2012 at 9:01 pm | Permalink

          I wonder if that works for those of us who are not also bloggers? (I’m not having problems yet–knock on wood–so can’t test it…)

  2. Posted March 18, 2012 at 6:07 am | Permalink

    Precognition, if it existed, would be the one of the most important discoveries of the last several thousand years. I think part of the problem with experiments like this is that psychologists don’t recognize just how extraordinary such an effect would be, given what we know today about the physical world. This was apparent in the Skeptiko interview and following comments, in which people thought even materialization and space-time teleportation of macroscopic objects from the past were no big deal for physics.

    It is shocking that the original journal would not publish at least an abstract or note about the failed replications. The only explanation I can think of is that the editors simply do not realize how mind-boggling the original result would have been for physics. This is far more shocking than faster-than-light neutrinos. Look how the physics community handled that claim….tons of papers on all sides of the issue came out quickly in all possible venues, and failed replications made world headlines. That is the way to do real science.

    • Posted March 18, 2012 at 6:11 am | Permalink

      The big problem with parapsychology as a field is that science is all of a piece. Thus, physics is consistent with chemistry, biology and so on. So the question is not “what knowledge can we derive on the assumption that we know nothing?” but “what knowledge can we derive given what we know already?”

      Basic physics leaves it not looking good for parapsychology as a field in any way. Sean M. Carroll points out that both human brains and the spoons they try to bend are made, like all matter, of quarks and leptons; everything else they do is emergent properties of the behaviour of quarks and leptons. And the quarks and leptons interact through the four forces: strong, weak, electromagnetic and gravitational. Thus either it’s one of the four known forces or it’s a new force, and any new force with range over 1 millimetre must be at most a billionth the strength of gravity or it will have been captured in experiments already done. So either it’s electromagnetism, gravity or something weaker than gravity.

      This leaves no force that could possibly account for telekinesis, for example. Telepathy would require a new force much weaker than gravity that is not subject to the inverse square law, and also a detector in the brain evolved to use it for signaling. Precognition, the receipt of information transmitted back in time, would violate quantum field theory. (Barring theoretical wacky tricks close to a black hole.)

      What this means is that these ideas have pretty much no chance of being right even before we test them directly.

      • Posted March 18, 2012 at 7:00 am | Permalink

        Yes, that is why extraordinary evidence is required. Still, I don’t think it should be ruled out a priori. Do the experiments, but with controls that rise to the seriousness of the claim.

        • Posted March 18, 2012 at 7:07 am | Permalink

          In the sense that Jehovah creating the universe and all life in 4004 BC shouldn’t be ruled out a priori.

          • Posted March 18, 2012 at 8:02 am | Permalink

            No, in the sense that faster-than-light neutrinos shouldn’t be ruled out a priori.

          • Naked Bunny with a Whip
            Posted March 18, 2012 at 5:49 pm | Permalink

            That isn’t ruled out a priori. It’s been ruled out by evidence.

        • Posted March 18, 2012 at 7:09 am | Permalink

          There’s a suggestion that parapsychology can serve as the control group for science itself: a field using the methods of science but where the null hypothesis is always true. If they come up with positive results (as they occasionally do), this shows where the methods of science need improving. (e.g. assuming a 5% significance means more than a 1 in 20 chance.)

          This does have the philosophical problem that it would require dismissing out of hand any positive results, rather than properly evaluating them as merely ridiculously unlikely. Fortunately, this is unlikely to be a practical problem while well-designed tests show no positive results, and the only tests showing any positive results tend to exhibit the experimental design skills of Daryl Bem.

      • Torbjörn Larsson
        Posted March 18, 2012 at 10:40 am | Permalink

        If a long range interaction isn’t subject to the inverse square law, it isn’t a force in the regular sense (able to accelerate masses). Gravity is a pseudo-force that do more (affects non-mass objects by way of curving spacetime) but still obeys the inverse square law.

        Entanglement and decoherence are processes that results in complicated effects on masses and are not grouped with forces. The same can be said for vacuum energy/dark energy, that either have local effects which are grouped with forces, or indirected long range effects by way of affecting spacetime which are not grouped with forces.

        I assume parapsychologists doesn’t care whether their presumed effects are “forces” or not. But it makes it harder for the involved physics. What process can’t affect particles directly, and still be amenable to detection by a particle system (a biological entity)?

  3. Posted March 18, 2012 at 6:09 am | Permalink

    This XKCD sums up Bem’s entire method. Run an experiment enough times, get a run shosing significance, publish.

    • Ken Pidcock
      Posted March 18, 2012 at 7:08 am | Permalink

      And this one is relevant as well.

  4. Posted March 18, 2012 at 6:20 am | Permalink

    “Ritchie, Wiseman, and French are well known as psi skeptics, whereas I and the investigators of the two successful replications are at least neutral with respect to the existence of psi.” So that seems tantamount to saying they were exuding ‘negative energy’ which might be a valid concept for a hippy but not in physics!

    Richard Wiseman is well worth reading –
    http://richardwiseman.wordpress.com/

    • Posted March 18, 2012 at 8:32 am | Permalink

      Actually it’s not entirely an invalid concept in physics. As I wrote four years ago in Negative Energy Research, it could be tested. You would start with a psi tests such as  Bem’s precognition test, but you would run it in a room in front of a one way mirror. Behind the mirror, either there is a skeptic watching the experiment or there isn’t, but double-blind controls insure none of the experimenters or subjects know which. The objective is to determine whether or not the skeptic’s presence (and therefore negative energy) influences the results of the experiment.

      I’m available.

      • Scote
        Posted March 18, 2012 at 9:41 am | Permalink

        No, Alex, you are not really proposing an experiment so much as proposing moving the goal posts. And if the experiment showed that their was no observer effect you’d just move the goal posts again and say that the psychic energy “knew” it was being tested and decided to play along. No need to waste time on an experiment just so you can play your games.

        • Gregory Kusnick
          Posted March 18, 2012 at 10:12 am | Permalink

          Scote, you’re confusing the real skeptico-with-a-C with the phony, non-skeptical Skeptiko-with-a-K hosted by Alex Tsakiris. Click through to skeptico’s blog to see the difference.

          • Scote
            Posted March 18, 2012 at 11:20 am | Permalink

            Doh!!! I thought I’d checked the spelling. I didn’t think to click on the link.

            Either way, my comment was too presumptuous. However, if I object to the woo meister suggesting the test then I must also object to the actual skeptic. The test will not stop woo meisters from claiming a skeptic effect on science.

            While I should have been more careful Ii also think the real skeptic Skeptico needs to add a tag to his name to help us tell the difference. Alex has been too successful with his pretend skeptic name to ignore the problem.

  5. andrewD
    Posted March 18, 2012 at 6:40 am | Permalink

    I could see this coming

    • Torbjörn Larsson
      Posted March 18, 2012 at 10:20 am | Permalink

      That is presumably what Bem should have done.

  6. JamesM
    Posted March 18, 2012 at 7:49 am | Permalink

    But…but unfalsifiability? Karl Popper? Won’t anyone think of the philosophers?

  7. moochava
    Posted March 18, 2012 at 8:11 am | Permalink

    “As Bem said, ‘Ritchie, Wiseman, and French are well known as psi skeptics, whereas I and the investigators of the two successful replications are at least neutral with respect to the existence of psi.’ That’s a pretty lame defense. Why would you re-test someone’s results if you weren’t a skeptic?”

    There’s an idea in the paranormal community that skeptics are actually powerful psychics with the psychic power to nullify other paranormal phenomena. In horror and urban fantasy fiction, it’s a great way of explaining how wizards, ghosts, and demons stay hidden from a curious species. In the real world, it betrays such a fundamental misunderstanding of how science works–turning the whole thing into some kind of big willpower-fight–that anyone who seriously raises the idea deserves only giggles.

    • Mark Joseph
      Posted March 18, 2012 at 8:41 am | Permalink

      Isn’t that (i.e., “idea in the paranormal community” to which you refer) also similar to the conspiracy theorists, who say that the lack of evidence for this or that conspiracy is evidence as to how well it’s being covered up?

    • Torbjörn Larsson
      Posted March 18, 2012 at 10:24 am | Permalink

      Not only pushing magic into imagined gaps, but withdrawing from testing.

      Besides being pathological science it is how religion works.

  8. dunstar
    Posted March 18, 2012 at 8:40 am | Permalink

    lolz. Sounds like much of Michael Persinger’s work on being able to increase ESP using “precise” magnetic fields. lolz.

    Persinger is the guy that also made the “God” Helmet. lolz.

  9. Mark Joseph
    Posted March 18, 2012 at 8:44 am | Permalink

    If you look back at Dr. Coyne’s post in response to Bem’s original study (hyperlinked above), you’ll see that the first comment was by Stuart Ritchie: “Worry not; some colleagues and I at the University of Edinburgh are arranging a replication attempt right now!”

  10. Greg Francis
    Posted March 18, 2012 at 9:30 am | Permalink

    It is possible to test the supposition that Bem used some kind of faulty experimental design or suppressed null findings. Given the magnitude of the effects he reports, he rejects the null hypothesis too frequently to be believed.

    Details are in

    Francis, G. (in press). Too good to be true: Publication bias in two prominent studies from experimental psychology. Psychonomic Bulletin & Review, DOI 10.3758/s13423-012-0227-9. http://www.springerlink.com/content/73400564u414n301/

    A corrolary of this finding is that there is no need to run replications of the Bem experiments. What Bem reported are essentially anecdotes, and science does not have to explain anecdotes.

  11. Nom de Plume
    Posted March 18, 2012 at 10:06 am | Permalink

    floating the possibility that the negative attitudes of Ritchie et al. could have had an effect on their results

    James Randi has written about this rationalization countless times–the whole idea that if you don’t get positive results, it’s all your fault for not clapping loud enough. This is an astonishingly weak response, even for a “woo-meister”.

  12. Greg Esres
    Posted March 18, 2012 at 11:00 am | Permalink

    “new study can’t replicate findings of precognition”

    Wow, who would have expected that!!!

    • Pete UK
      Posted March 19, 2012 at 4:49 am | Permalink

      I think we all already knew the precognition trial wouldn’t produce anything….

  13. Adam Fetterman
    Posted March 18, 2012 at 11:05 am | Permalink

    It’s good to see this coming out. We (as a field) have been pretty embarrassed by it. This, then the Stapel scandal. We can’t catch a break. At our SPSP conference every year, there is always a poster presented that is pretty light-hearted. This year, it was about this Bem thing and…..it was not so light-hearted. It sucks that JPSP won’t publish these refutations, but unfortunately, they have a policy about null-findings and comment/reply articles.

    As a sidenote, my lab runs studies like these a lot (not for “psi” reasons). Every so often we will get one of these pre-cog findings and chalk it up to artifacts or design. However, we do find that extraverts are able to predict random stimuli more accurately, nearly everytime. We chalk this up to our randomization procedures. We assume it is because extraverts are subconsciously better at keeping track of the # of trials in which a particular type of occurence happens. Of course, if we used randomization with replacement, we would assume that these findings would disappear. Perhaps this is the type of thing that criticism #2 is referring to.

  14. MadScientist
    Posted March 18, 2012 at 2:14 pm | Permalink

    Wiseman’s association with James Randi seems to have imbued him with a magical anti-psi auroa. I wonder why Bem is acting such a fool and repeating ancient discredited excuses for psi.

  15. Diane G.
    Posted March 18, 2012 at 10:16 pm | Permalink

    “…whereas I and the investigators of the two successful replications are at least neutral with respect to the existence of psi.”

    IMO that would read better–more like science–if he’d left off the “at least.” With it, it sounds as if he’s eager to add an “if not:”

    “…are at least neutral, if not guardedly optimistic, with respect to the existence of psi.”

  16. Marella
    Posted March 19, 2012 at 9:02 pm | Permalink

    Precognition if it existed at all, would be so extraordinarily useful for any animal to have that it would spread and strengthen until everyone had it, and no-one could ever be surprised by anything. It would be a skill more useful than sight. If it existed at all it would be everywhere, not hiding in the couple of percentage over the odds that these researchers manage to conjure up.

    • Posted March 21, 2012 at 6:07 pm | Permalink

      That’s like saying that intelligence and complex languages are so useful that every animal should possess them.

      • rickdoogie
        Posted March 22, 2012 at 6:12 am | Permalink

        I don’t understand whether that was a tongue-in-cheek reply or not. Don’t we find intelligence and complex communication everywhere in the animal kingdom?
        No one would argue that all animals ought to have human intelligence and human language. Of course, human psychic powers would be different than animal psychic powers, if those powers existed.
        Marella is not saying that psychic powers would be always and everywhere the same. We would see lots of variations in psychic powers, as we see a wide variety of communication methods. But, we wouldn’t be surprised or spooked by the basic ability, because it would be an everyday experience for all animals.

        • Posted March 22, 2012 at 7:34 am | Permalink

          I was actually serious. I was just saying that some abilities may require a certain degree of physical complexity before they could evolve. Since humans are at or near the top in terms of cerebral complexity, it is at least logically possible that there could be new things arising in us that were not present in less complex animals. I agree it is unlikely, I am not arguing that this is in fact the case, but it does seem logically possible.

          • rickdoogie
            Posted March 22, 2012 at 8:19 am | Permalink

            Thanks for the level-headed reply, Lou. I see what you’re saying. I just have to be skeptical about things that hit us as “spooky”. That spooky feeling signals a contradiction in either logic or experience.
            That said, I agree with what you say about cerebral complexity. There are definitely points where evolution takes a “jump” to a new level. The whole psychic thing is too big of a jump for me to swallow, though.
            I’m very open-minded, but it all seems like wishful thinking to me. At first glance, psychic powers seem pretty cool, but there are lots of negative consequences. The psychic noise level in our brains would be a maelstrom. Like much wishful thinking, the wish for psychic powers is short-sighted. IMHO

  17. rickdoogie
    Posted March 20, 2012 at 9:55 am | Permalink

    The human brain and body do all kinds of amazing things. These amazing abilities are commonly known among everyday people. They are not shrouded in spooky mystery because we see them every day. When we measure extraordinary individual abilities, we aren’t observing previously unknown abilities. They are variations or enhancements of abilities we already take for granted.

    If it is shown that we can subconsciously hear, see, or feel far beyond what is commonly thought to be our sensory limit, that isn’t such a stretch of credulity. There would be few standing up to argue one way or another about a discovery that revealed new higher limits to our already well-known abilities.

    If precognition or telekinesis were real human abilities, there wouldn’t be a need for all of this argument. We don’t argue whether a brain can sense degrees and patterns of light, sound, temperature, and so on. We don’t argue whether humans can manipulate their environment.

    Certain individuals might be extra good at sensing and manipulating certain objects and motions in our environment. These are all questions of degree and quantity. We don’t argue about the quality of human perception until we get into the land of pseudo-science.

    I’ve always argued that if telekinesis was a real possibility, we would see glaring proof of it in our everyday lives. A home football game where 5,000 people are willing the ball to go through some game-winning motions – we would see balls change course in midair. Even if it was argued that only a fraction of the crowd had above average telekinetic powers, we would still see some strange movements of the game ball that only telekinesis could explain. It would be a common phenomenon, and no one would be spooked by it.

    Same with precognition. If it was a true ability of human brains, we would see obvious and testable examples in our everyday life. It would not seem spooky to anyone. It only seems spooky because it comes from the realm of fantasy.

  18. rickdoogie
    Posted March 20, 2012 at 9:57 am | Permalink

    Sorry, I guess my long ramble was just repeating what Marella said much more succinctly.


One Trackback/Pingback

  1. […] Tests of the supernatural fail again: new study can’t replicate findings of precognition « Why Ev…. Share this:TwitterFacebookLike this:LikeBe the first to like this post. […]

%d bloggers like this: