Doubts cast on recent survey of college students’ attitudes toward free speech

Four days ago I wrote about the results of a poll of American college students’ attitudes towards free speech. That poll was conducted by UCLA professor and Brookings senior fellow John Villasenor, and was supported by the Charles Koch Foundation. The results were scary, with 44% of all students thinking that hate speech is not protected by the First Amendment, 51% of students finding it acceptable to shout over a speaker to squelch their presentation, and 19% of students saying that it was okay to use violence to disrupt a speaker whose words were “offensive and hurtful.”

I now feel obliged to report that, according to a piece in Friday’s Guardian, the results of this poll have been heavily questioned by some experts. Since the poll’s methods hadn’t been published at the time (the author felt it important to release the data before they were peer reviewed), there may be some serious problems. As the Guardian notes,

The way the survey results have been presented are “malpractice” and “junk science” and “it should never have appeared in the press”, according to Cliff Zukin, a former president of the American Association of Public Opinion Polling, which sets ethical and transparency standards for polling.

John Villasenor, a professor of electrical engineering at the University of California Los Angeles, defended his survey as an important window into what he had called a troubling atmosphere on American campuses in which “freedom of expression is deeply imperiled”. Villasenor, a cybersecurity expert, said this was the first public opinion survey he had conducted.

However, his survey was not administered to a randomly selected group of college students nationwide, what statisticians call a “probability sample”. Instead, it was given to an opt-in online panel of people who identified as current college students.

“If it’s not a probability sample, it’s not a sample of anyone, it’s just 1,500 college students who happen to respond,” Zukin said, calling it “junk science”.

“It’s an interesting piece of data,” Michael Traugott, a polling expert at the University of Michigan’s Center for Political Studies, said. “Whether it represents the proportion of all college students who believe this is unknown.”

. . . [Villasenor] secured funding from the conservative Charles Koch Foundation to survey students this August about their views on free speech. Rather than write an academic paper, he posted some of his results online this week, arguing that given “the timeliness of the topic, I believe it is important to get some of the key results out into the public sphere immediately”.

. . . Villasenor’s results had gone through no peer review process. The methodology section of his online post was vague, prompting several polling experts to question how reliable the survey’s conclusions might be.

Villasenor wrote in an email that he was reluctant to give a yes or no “sound bite” answer to the question of whether the students he surveyed were nationally representative of college students or not.

By some measures, Villasenor wrote, the 1,500 respondents to his survey had seemed to reflect the rough demographic makeup of American college students. By others, they might not.

Villasenor had calculated a margin of error for his survey results and included it in the public writeup of his report, even though the sample of students he had surveyed was not random. Public polling experts said this was inappropriate and a basic error. Zukin called it “very misleading” and “malpractice”.

By including a margin of error, the author appears to be “trying to overstate the quality of his survey”, said Chris Jackson, the vice-president of Ipsos Public Affairs, a public opinion firm.

Timothy Johnson, the current president of the American Association for Public Opinion Research, called it “really not appropriate”.

Jackson also notes that asking this question right after the Charlottesville marches and killing might have conditioned students to be more disapproving of offensive speech than normal.

The lack of a random sample is indeed disturbing (I may be guilty of not catching that), and we’ll see if this thing gets published. My guess is still that students will still be shown to be remarkably ignorant of the First Amendment, and likely to take a more punitive attitude toward “hate speech” than mandated by U.S. courts, but we’ll see.



  1. Ken Phelps
    Posted September 24, 2017 at 1:18 pm | Permalink

    Examining your own writing? Correcting mistakes?

    Good Lord, man, what country are you living in?

  2. GBJames
    Posted September 24, 2017 at 1:25 pm | Permalink

    First, alternative facts. Then alternative science.

  3. Peter
    Posted September 24, 2017 at 1:36 pm | Permalink

    Maybe of interest: Report from Pew Research Center, 2016:
    Evaluating Online Nonprobability Surveys

  4. Derek Freyberg
    Posted September 24, 2017 at 1:39 pm | Permalink

    I went back and skimmed Villasenor’s original article, to which the OP links, and did not see any indication that this was an “opt-in” poll rather than a random poll. To that extent, Villasenor failed to provide sufficient context. Also, I don’t see why he couldn’t have provided the questions themselves as an attachment to his article. Neither of these is good.
    But it’s not “junk science” as Zukin described it – people run “opt-in” polls all the time; it’s just not good science. Traugott’s criticism seems much more measured to me.
    The subject deserves a better poll; but I would not be surprised to find that there was significant sentiment on college campuses for the suppression of right-wing views. After all, one can look at what has happened at several universities to see how students have been quick to condemn a lack of what they see as political correctness (Christakis at Yale, Weinstein at Evergreen).

    • GBJames
      Posted September 24, 2017 at 1:55 pm | Permalink

      How is “not good science” substantively different from “junk science”?

    • Posted September 24, 2017 at 2:00 pm | Permalink

      If it’s not junk science, then we should be able to articulate what these results tell us. What question do they answer? That depends entirely on the sampling, the context and timing of the survey, and the question wording. But the sample distribution seems pretty much unknown. So what it tells us is also pretty much unknown. Drawing conclusions from this survey might be no better than rolling dice.

      • Derek Freyberg
        Posted September 24, 2017 at 5:06 pm | Permalink

        As the Pew article mentioned by Peter suggests, nonrandom polls are not automatically bad: it really depends on who is doing the polling.
        In any event, the answers are apparently those given by a group of people self-identified as college students and scattered about the US. “Here is some more detailed information regarding the survey: This web survey of 1,500 undergraduate students at U.S. four-year colleges and universities was conducted between August 17 and August 31, 2017. … . I designed the survey questions and then requested that UCLA contract with a vendor for the data collection.” He says more – read it for yourself and decide how meaningful the sample is and the questions are.
        Villasenor’s “sin”, if you want to call it that, is writing as if it were a random poll, not in conducting the poll he did.

        • Posted September 24, 2017 at 5:23 pm | Permalink

          It may be possible to rehabilitate the data in some way, but based on the information that we have, it seems that the survey did not really measure what it’s author intended, was both misdescribed and under-described by the author, and bypassed peer review in order to make a splash in popular media. Those are some red flags for junk science. And I still don’t know what it means other than it found this many university students who believed those particular things.

  5. jay
    Posted September 24, 2017 at 1:50 pm | Permalink

    Have you seen this:

    Antifa thugs take over the press conference and the college officials are chased from the roomm.

    • Randy schenck
      Posted September 24, 2017 at 2:49 pm | Permalink

      Pretty bad. I don’t really need a survey to figure this one out. Even Comey, the fellow fired from the FBI, was hired by Howard University and the other day he was shouted down by people at the University. Lovely bunch of kids at these colleges.

    • Saul Sorrell-Till
      Posted September 25, 2017 at 8:04 am | Permalink

      What’s that got to do with the quality of this particular study?

  6. Gregory Kusnick
    Posted September 24, 2017 at 3:53 pm | Permalink

    Perhaps this warrants an update to your original post warning readers that the numbers may not be trustworthy.

  7. Posted September 24, 2017 at 6:13 pm | Permalink


    Villasenor’s survey asked students if it was more important for colleges to create an “open learning environment where students are exposed to all types of speech and viewpoints, even if it means allowing speech that is offensive or biased against certain groups of people” or “a positive learning environment for all students by prohibiting certain speech or expression of viewpoints that are offensive or biased against certain groups of people”.

    I think this party itself destroys the credibility of the survey. The characterisation of the censored environment as “positive” is clearly loading the question.

    • Posted September 24, 2017 at 6:15 pm | Permalink

      “…this part by itself…” not “this party itself”.

      • Saul Sorrell-Till
        Posted September 25, 2017 at 8:01 am | Permalink

        Yes, agreed.

        • Posted September 25, 2017 at 5:56 pm | Permalink

          Agreed that the correction is right or that the corrected original post is right?

  8. Posted September 25, 2017 at 9:11 am | Permalink

    Selection bias is one of the cornerstones of social science research. The Guardian has no problem making headlines out of studies with shoddy methodology, so long as the findings match its regressive left ideology.

    • GBJames
      Posted September 25, 2017 at 9:15 am | Permalink

      Uh…. the subject at hand is junk science supported by Charles Koch, about as far away from the left as you can get, regressive or otherwise. The Guardian has nothing to do with this, whatever their biases might be.

      • Posted September 25, 2017 at 11:45 am | Permalink

        The Guardian is guilty of It’s Okay When We Do It™

        • GBJames
          Posted September 25, 2017 at 11:46 am | Permalink

          Nice whataboutery.

          • Posted September 25, 2017 at 2:48 pm | Permalink

            I’m calling out hypocrisy. Why are you so defensive of the Grauniad?

            • GBJames
              Posted September 25, 2017 at 3:37 pm | Permalink

              I’m not defending the Grauniad. Or even The Guardian.

              I’m pointing out that whattaboutery is not a legitimate response to fake science generated from the right.

              • Posted September 27, 2017 at 8:28 pm | Permalink

                Gosh, sorry, I didn’t get the memo on the official “legitimate response”.

                You’re accusing me of a tu quoque, but I’m not defending the study in question. I’m merely observing that heavily biased newspapers in glass houses shouldn’t throw stones.

%d bloggers like this: