What, if anything, is wrong with the Pew poll on evolution acceptance?

by Greg Mayer

I posted on Wednesday about the new Pew poll on evolution acceptance, focusing on the divergence between the Pew results and those from 2012’s Gallup poll of the same issue. In both polls it is possible to divide respondents into three classes that can be thought of as those that accept “naturalistic evolution”, those that accept “theistic evolution”, and those that accept “creationism”. The Pew poll shows a greater preponderance of the first two (32% and 24%, respectively, with 32% creationist) than does the Gallup poll (15% and 32%, respectively, with 46% creationist). I considered how differences in wording between the two polls might have affected their results, but could not come to any convincing explanation for the disparity.

The Pew poll was also noticed in the general media (e.g. MSNBC, Reuters, NPR, Christian Science Monitor), and most of these have emphasized in their stories the fact that creationism is much more popular among Republicans than Democrats and independents (a plurality, 48%, of Republicans are creationist, while 43% accept evolution), and that the popularity of creationism among Republicans has increased notably since the last time Pew polled this question in 2009 (at that time, a majority of Republicans, 54%, accepted some form of evolution, while 39% were creationist). The headline from the Christian Science Monitor, “Percentage of Republicans who believe in evolution is shrinking”, is representative.

A number of commentators– for example Andrew Sullivan, David Graham at the Atlantic, Zack Beauchamp at Think Progress, Allapundit at Hot Air, and Francis X. Clines and Paul Krugman at the New York Times– have also taken note of the Pew poll, and offered various suggestions as to what has happened to increase Republican support for creationism since 2009. (Only Allahpundit took up the question I found most interesting– why did the Pew results diverge from those of Gallup; he also noted differences between Pew and recent Yougov and Harris polls as well.) The two logical possibilities are that Republicans have shifted their views, becoming more creationist; or that creationists have shifted their party allegiance, more of them becoming Republicans, while those who accept evolution have become independents or Democrats.

Both of these type of shifts, of Republicans to creationism and of party allegiance, could be happening, and both have been suggested by one or another of the commentators. Beauchamp, for example, suggests that out-of-power Republicans are rallying to the “team” position, while also noting that white evangelical Protestants have shifted their party allegiance toward the Republicans. Graham pointed to some evidence that scientists have also shifted, and are less likely to be Republicans now than in the past.

While most commentators have offered interpretations or explanations of the Pew results, Dan Kahan of the Cultural Cognition Project at Yale Law School has questioned the results themselves, and complained about the media’s headlining of the Republican embrace of creationism. His complaints received increased attention because they were tweeted by noted science journalist Carl Zimmer, who wrote that Kahan had “pick[ed the Pew poll] apart”. Jerry, with a nod to Zimmer, has taken note of Kahan’s piece in an addendum to my post.

So does Kahan land any blows on the Pew poll? Well, yes and no. He criticizes Pew for not releasing the full crosstabs on their poll, and on this point I share his frustration. In the Pew press release, there’s a link for the “Full Report“, but this leads to a pdf consisting of the press release plus a subset of the exact questions with answers. It would have been nice to know how party affiliation correlated with naturalistic vs. theistic evolution, for example.

But Kahan then goes on to make two further complaints, neither of which stand up. First, he says that if we knew what percent of Democratic respondents could be classed as accepting naturalistic vs. theistic evolution, this would change the entire cast of the results, from a “ha ha ha!” at Republicans’ expense, to…. ?; it’s not entirely clear what, but it apparently would be “complicated and interesting”. But knowing this about Democrats would not change the main news about party allegiance: 48% of Republicans are creationists. No matter how the Democrats divide on naturalistic vs. theistic evolution, it’s not going to stop Republicans from looking really bad.

The only way to really alter the import of the Pew results on evolution acceptance and party allegiance is if the numbers are actually wrong, not just misinterpreted, and that’s the complaint that Kahan takes up next. Now, both Allahpundit and I considered the possibility that Pew’s numbers are wrong in the sense that, given other polling data, Pew’s numbers may not be good estimates of true public opinion. But Kahan apparently believes Pew’s numbers don’t add up on their own terms, that there is some “logical inconsistency” in the numbers. He arrives at this conclusion by arguing that since the percentage of Democrats and independents accepting or rejecting evolution has not changed, but the percentage of Republicans who are creationists has gone up, then it should be the case that there should have been an overall decline in acceptance of evolution:

And logically, in that case, the % of the U.S. public overall who now say they are “creationists” would have had to gone up–especially insfoar as the proportion of the population identifying as Republican has increased a lot since 2009[note: assuming we include "lean Republican" "independents" in the totals, as we should if we are trying to give an accurate senes of partisan identification]. [Kahan's brackets-- GCM]

But according to Pew there has been no change in overall acceptance, leading Kahan to conclude:

So, something does not compute.

At a minimum, Pew has some ‘splainin to do, if in fact it is trying to edify people rather than feed the apptetite of those who make a living exciting fractious group rivalries among culturally diverse citizens.

But Kahan is wrong about three things here. First, the proportion of the population that identifies as Republican has not increased a lot since 2009. In Pew’s own surveys, the proportion of Republicans since 2008 has varied from 24% to 25% (these exact numbers are shown in Kahan’s post).  According to Gallup, Republican numbers are tending down, not up, over the last few years. There is thus no reason to think that the proportion of Republicans has gone up. In Pew’s 2009 survey of evolution acceptance, the unweighted proportion of Republicans was 25%, right in line with its other estimates, so there’s no sign that Pew’s evolution acceptance polls underestimate this proportion.

Second, Kahan says independents who “lean Republican” should be counted as Republicans, but that’s not what Pew did, and Kahan’s wishes as to how he would want them counted does not affect Pew’s numbers.  Kahan is right that a fuller release of data would be interesting, but no inference of logical inconsistency in the Pew data can be based on this.

And third, Kahan has equated the statistical concept of “the same” with mathematical equality. In algebra, if x + y = z, and you increase y while holding x constant, then the sum, z, must increase. This is in outline form the argument made by Kahan (where x and y represent political subdivisions, and z the overall number). But if x, y, and z are statistically estimated quantities with errors of measurement, then it is possible for x to show no significant change and for y to increase significantly, yet z still show no significant change. This would be especially so if the magnitude of y is small relative to z (and the proportion of Republicans is the smallest of the three main political respondent classes). Thus, there is no contradiction between Democrats and independents being statistically unchanged, Republicans showing a significant change, yet the overall result is also statistically unchanged.

The inference that Kahan tries to make is further undermined by the fact that the overall acceptance of evolution is not the weighted average of just Democrats, independents, and Republicans (where the weights are their estimated proportions in the population), but the weighted average of Democrats, Republicans, independents, other parties, no preference, and refused. There are too many unknowns to be solved for; it’s not x + y = z, but t + u + v + w + x + y = z, and thus  knowing just how Democrats (or Democrats+independents) and Republicans have responded is not sufficient to infer the overall response. (I tried figuring out some of these numbers for the 2009 survey from the summarized results by making assumptions to reduce the number of variables, but when I checked my resulting approximations against the actual numbers in the full Pew data set, I was noticeably off.)

So, Kahan is right that it would be useful to have the full data set, but wrong i) that having that full data set would change the apparent embrace of creationism by Republicans; or ii) that it is possible to infer logical inconsistency in the released data. There could be errors in Pew’s calculations of proportions and tests of statistical significance, but that cannot be inferred from the given data.

But is there something nefarious in Pew’s failure to release some of the data, rather than all of it? Kahan clearly thinks there is. He writes:

…this sort of deliberate selectivity (make no mistake, it was deliberate: Pew made the decision to include the partisan breakdown for only half of the bifurcated evolution-belief item) subsidizes the predictable “ha ha ha!” response on the part of the culturally partisan commentators who will see the survey as a chance to stigmatize Republicans as being distinctively “anti-science.” …

Pew lulled those who are making the ["ha ha ha ha ha!"] response into being this unreflective by deliberately (again, they had to decide to report only a portion of the evolution-survey item by political affiliation) failing to report what % of Democrats who indicated that they believe in “naturalistic” evolution. [Democratic results were not selectively withheld; no political breakdown was provided for this question.]  …

Right away when I heard about the Pew poll, I turned to the results to see what the explanation was for the interesting — truly! — “shift” in Republican view: Were Republicans changing their positions on creationism or creationists changing their party allegiance?

And right away I ran into this logical inconsistency.

Surely, someone will clear this up, I thought.

But no.

Just the same predictable, boring “ha ha ha ha!” reaction.

Why let something as silly as logic get in the way of an opportunity to pound one’s tribal chest & join in a unifying, polarizing group howl?  [All brackets mine-- GCM)]

But Kahan’s inference of logical inconsistency cannot be sustained, and thus his speculations as to motive are merely expressions of his own prejudices.

So why didn’t Pew release all the data? Pew’s policy is to release their full data sets a few months after issuing its reports: that’s how I was able to get the full details for the 2009 survey. There may be all sorts of reasons why Pew doesn’t release all its data (most perhaps having to do with the fact that gathering and analyzing such data is most of what they do, and they want first crack at and the chance to publicize their own data before everyone else does), and we won’t be able to check for ourselves to see if Pew made any errors for a few months. But there’s not the slightest hint of error in the released data, and though I don’t know what “tribe” Kahan imagines himself to be a member, if there’s any pounding and howling going on here, it certainly isn’t being done by Pew.

h/t Matthew Cobb

31 Comments

  1. Diana MacPherson
    Posted January 3, 2014 at 10:23 am | Permalink

    sub

    • francis
      Posted January 3, 2014 at 11:53 am | Permalink

      //

      • Jesper Both Pedersen
        Posted January 3, 2014 at 1:13 pm | Permalink

        tres

  2. NewEnglandBob
    Posted January 3, 2014 at 10:53 am | Permalink

    In the last dozen years or so, polls have become unreliable. I also think that many Americans change their mind depending on which news organization and which article they saw last.

    • Posted January 3, 2014 at 12:36 pm | Permalink

      While it might superficially seem so, Nate Silver of 538 fame demonstrated that the various biases (intentional and otherwise) are themselves so reliable that, in aggregate, polling is a near-perfect proxy for election results.

      So, while any single given poll from any single given pollster may well be not very reliable (and generally isn’t anywhere near as reliable as the pollster would have you believe), once you put enough of them together you can get a good idea of what reality actually is.

      Thanks to what Nate is doing, we can reasonably expect polling to improve dramatically and rapidly in the coming years. Not only is he providing the pollsters with an incentive to improve their results (since he ranks them), but he’s providing them with tools to help them do so — again, the ability to independently verify and analyze results.

      Cheers,

      b&

      • gbjames
        Posted January 3, 2014 at 1:57 pm | Permalink

        Nate Silver’s work is great which is why a lot of political pundits hate him.

        • Posted January 3, 2014 at 2:04 pm | Permalink

          Oh, yeah — especially the part of the punditocracy that specializes in hyping the horse races.

          This, though, I think will work out for the best. If people stop paying attention to the horse race predictions, the nattering nabobs will have to turn to actual analysis of political positions and what they might mean for the country’s citizens.

          …I hope….

          b&

          • moarscienceplz
            Posted January 3, 2014 at 3:50 pm | Permalink

            …or, they could become professional psychics. Cool thing about that is that most people stop paying attention before your prediction is proved wrong. It worked great for Sylvia Browne.

  3. eric
    Posted January 3, 2014 at 11:01 am | Permalink

    Pew lulled those who are making the ["ha ha ha ha ha!"] response into being this unreflective by deliberately (again, they had to decide to report only a portion of the evolution-survey item by political affiliation) failing to report what % of Democrats who indicated that they believe in “naturalistic” evolution.

    Given that they have only released the summary findings and a press release, I can think of any number of utterly mundane reasons why they would have done so. They only had room for highlights. The dem breakdown was no differerent from last year. The dem data was hard to analyze. The dem breakdown analysis results are confusing and hard to describe via soundbyte. The ‘by party’ results were not as important to Pew as the ‘by religion’ results, so they haven’t completed the ‘by party’ analysis yet. And so on, and so on, and so on.

    Lastly, as you point out Jerry, it’s the 48% creationism that is stigmatizing to the GOP*, not the percent of Dems who are theistic evolutinists.

    *From an intellectual’s point of view. I’m sure there are a lot of GOPers who will wear that increase as a a badge of pride, and whose biggest question is ‘how do we root out the RINOs who are keeping it from being 100%?’

  4. Posted January 3, 2014 at 11:04 am | Permalink

    Quick poll: how many folks here have actually participated in a poll?

    Actually probably not a good idea to answer that question, as it would result in an avalanche. But I’ll answer it… looking at Pew’s methodology, they seem very solid. Random digit dialing of both cells and land lines. I think there’s some mailings and face-to-face for the 2% that are unreachable by either means.

    I am, however, extremely meticulous about screening my calls – esp. those from cell phone, and for what it’s worth, so are my friends and family. Most people I know have given up their land lines, because the abuses from corporate and political America have gotten so ridiculous, it’s just not worth it. I’ve begun receiving brochures from the phone company begging people to keep their land lines.

    How the Pew group (and others) can maintain that they are getting a representative sample is beyond me, at this point. Seems like it would be skewed somehow – probably to more rural people.

    So perhaps they oversample… geography is a big component of their methodology — but I’d also wager there’s also bias to the less educated in urban environments.

    Now some of the best samplers / methodologists / survey experts I know are the group out of UChicago – people at NORC (e.g. Tom Smith w/ the General Social Survey – GSS). Perhaps they have more of a clue as to how trustworthy Pew and Gallup and such are anymore. I just don’t know. Any thoughts?

    • Posted January 3, 2014 at 11:07 am | Permalink

      My first question reminded me of a Bill Hicks routine… “by a show of hands, how many of you have NEVER participated in a poll? Hmmmm that’s about 80%.”

    • NewEnglandBob
      Posted January 3, 2014 at 11:20 am | Permalink

      I refuse every phone call about polling. My uninformed guess is that the more educated people will do likewise.

      Just because they call random phone numbers, doesn’t mean the sample is a correct representation of the whole.

      • eric
        Posted January 3, 2014 at 11:33 am | Permalink

        If it was Pew or Gallup and it *wasn’t* around election time, I would probably participate if called. But not if it’s some group I’ve never heard of, and not any poll around election time, for the same reason: I am not listening to your advertising-via-push-poll.

        • Achrachno
          Posted January 3, 2014 at 8:11 pm | Permalink

          My attitude exactly. I don’t think I’ve ever been called by a major/legitimate pollster. Push-polls are common, but should be a criminal offense. Two years for the first conviction sounds about right for corrupting democracy.

      • Posted January 3, 2014 at 7:02 pm | Permalink

        As a relic of my days working in an electron microscope lab where distractions could wreck samples, I generally let most of my calls go into voice mail. A significant number of callers hang up when they here the (relatively) polite request to “leave a message…”. Perhaps that explains why I’ve never been a subject in any poll!

    • gbjames
      Posted January 3, 2014 at 11:30 am | Permalink

      I’ve found it interesting to engage the poll-caller with questions about who is doing the poll, who is sponsoring the poll, etc. Most “polls”, at least in election season, are push-polls done by Republican operatives who try to conceal that fact. When I detect such polling I have enjoyed answering in odd-ball or screw-them-up ways.

      Cheap entertainment.

      • eric
        Posted January 3, 2014 at 11:35 am | Permalink

        It doesn’t really screw them up. Heck they probably don’t even bother tabulating the results. Their primary goal is to communicate their message to you. Collecting informatino back is secondary, and they probably only do it at all in order to help them make the next iteration of their push-poll more effective.

        • gbjames
          Posted January 3, 2014 at 11:48 am | Permalink

          It wastes their time, at the least, and pisses them off, depending on how the conversation ends (usually based on my own limited willingness to continue).

          • eric
            Posted January 3, 2014 at 12:35 pm | Permalink

            That, indeed may be worth your entertainment value.

            AIUI a lot of corporate help lines have moved to a policy where a service person will never voluntarily hang up on you. They always wait until you hang up. If you don’t believe me, try waiting for them to hang up the next time you call your phone, internet, or TV service provider. I bet after an uncomfortable 10-30 second silence, they’ll tell you something like “please hang up, sir, I am not allowed to.” Anyway, I bring that up because if you really want to waste their time, you might be able to get away with just putting the phone down on the table and going off to do something else.

            • Posted January 3, 2014 at 12:44 pm | Permalink

              That would be a perfect time to impersonate a dizzy teenager in love… “no… (giggle)… YOU hang up first…”, rinse, repeat…

              • gbjames
                Posted January 3, 2014 at 1:58 pm | Permalink

                LOL

    • Latverian Diplomat
      Posted January 3, 2014 at 1:01 pm | Permalink

      A good poll samples very carefully, checking against biases in geography, income, education level, etc.

      That makes quality polling very expensive.

      Political silly season polling doesn’t usually have the time or money to do that properly, but I would expect a poll like this to put that kind of effort in. That’s my expectation I can’t vouch for them.

      • Posted January 3, 2014 at 11:05 pm | Permalink

        Yep. VERY expensive. That’s why the UChicago folks are the go-to people in my book. They’re funded to the gills (and have been asking the questions consistently over time, to boot). The Pew (and probably Gallup result, if they had any) wold seem to be in contradiction to over all trends, as reported on WEIT earlier.

        Anyway, it’s not a perfect fit or anything, just makes me doubt — and it seems like a bit of an anomalous result, to me.

        • Posted January 3, 2014 at 11:09 pm | Permalink

          …unless this rise in anti-evolution business is more confined to the religiously-affiliated — i.e. there is little cross-talk with the rise of the “nones” – i.e. so-called “mavericks” who would entertain that evolution is not sufficiently supported, etc., etc.

          It just smells fishy to me. I’d be interested to see what corroborative evidence comes out of NORC, if any, in the months/years up ahead.

  5. Latverian Diplomat
    Posted January 3, 2014 at 11:38 am | Permalink

    Kahan’s concerns seem largely driven by the fear that these results add to the perception of Republicans as a bunch of superstitious bigots.

    A big part of his work (much of which is interesting) is that the differences between political groups stem from underlying values (as captured on the group-grid axes) and that the reasoning that goes from values to policies is of equal validity (or lack thereof).

    IMHO, the modern Repbulican party is making this an increasingly difficult position to defend. That is, however, just my opinion.

    • Achrachno
      Posted January 3, 2014 at 8:13 pm | Permalink

      You’re right though.

  6. Posted January 3, 2014 at 12:48 pm | Permalink

    Reblogged this on A man and his brain.

  7. ploubere
    Posted January 3, 2014 at 1:20 pm | Permalink

    My guess would be that the increasingly public debate about evolution is driving a lot of republicans into a defensive reaction on a topic they hadn’t thought much about before.

  8. Paul Spence
    Posted January 3, 2014 at 2:20 pm | Permalink

    Sorry if I didn’t read all the comments so might have missed a similar comment. The big issue, as I see it, is that a large number of “evolutionists” think god played some sort of role in evolution. These people are not evolutionists they are creationists. We have a long way to go in bringing non-magic thought to people (and I mean the whole world).

  9. Achrachno
    Posted January 3, 2014 at 11:23 pm | Permalink

    The only thing wrong is that they need to sample a more sensible population.

  10. vjtorley
    Posted January 4, 2014 at 7:26 am | Permalink

    Hi. For anyone who’s interested, I wrote a post on the contradictions between different opinion polls on evolution: http://www.uncommondescent.com/intelligent-design/can-we-trust-opinion-polls-on-evolution/


Follow

Get every new post delivered to your Inbox.

Join 27,754 other followers

%d bloggers like this: