July 26, 2013

Government Surveillance: A Question Wording Experiment

Court Approval a Big Factor in Public Support

Overview

In the wake of leaked information about the government’s telephone and digital surveillance programs last month, public opinion surveys reported a wide range of reactions. For example, a Pew Research Center/Washington Post survey conducted immediately after the revelations found broad support for the program, while a Gallup survey conducted just days later found more disapproval than approval. These, along with a number of other surveys during that period, all made an effort to describe the program as accurately and neutrally as possible, yet different question wording clearly produced different responses.

Four Experimental Treatments of Government SurveillanceTo better understand how the manner in which the government’s surveillance program is described affects public evaluations, the Pew Research Center conducted a question wording experiment in a national telephone survey fielded between July 11 and 21, 2013 among 2,002 adults. The survey respondents were asked whether they would favor or oppose a government data collection program, but the wording of four elements of the program were described differently to different groups of respondents. These are: whether metadata or content is being collected; whether phone calls or emails are being monitored; whether the program has court approval; and whether the program is part of anti-terrorism efforts.

Mentioning the role of courts and describing the program as part of anti-terrorism efforts each had a substantial effect on public sentiment. Among the roughly 1,000 respondents who heard the government surveillance program described as occurring “with court approval,” support was 12 points higher than among the other 1,000 who heard no mention of courts. This is consistent with the findings of a separate Pew Research Center survey, which found that people’s impressions of whether or not there is adequate court oversight of the program are more strongly linked to overall support an opposition than are other perceptions.

2 Mentioning Court Approval, Terrorism Increases NSA SupportMentioning the goal of terrorism also affects the level of public support. When the surveillance was described as “part of anti-terrorism efforts” it garnered 9% more support than when this goal was not mentioned.

Describing the government as collecting metadata, such as the date, time, phone numbers and email addresses, drew more approval than when the program was described as collecting the actual recordings of phone calls or the text of emails.

By contrast, the distinction between a program targeting phone communications and email communications appears to have no effect. There was almost the same reaction when a program was described as monitoring phone calls as when it was described as monitoring email communication.

The combination of these four wording tests produces a total of 16 possible program descriptions1, which can be ranked in terms of public support. For simplicity’s sake, since the telephone/email distinction did not elicit different responses, we have removed it here to narrow the range to eight possible descriptions, with roughly 250 respondents (one-eighth of the overall sample) in each group.

3 Views of Government Data CollectionCombined in this way, respondents who heard the program described as collecting only “data such as the date, time phone numbers and e-mails… with court approval as part of anti-terrorism efforts” were the most supportive: 41% said they would favor this kind of program. By contrast, only 16% favored a program they heard described as collecting recordings of phone calls or the text of emails with no mention of either courts or the goal of fighting terrorism – fully 25-points lower than support when these other considerations are mentioned.

Under every condition in this experiment more respondents oppose than favor the program. This stands in contrast to our new survey in which slightly more approve (50%) than disapprove (44%) of “the government’s collection of telephone and internet data as part of anti-terrorism efforts.” This disparity picks up on another element of question wording that can affect people’s evaluations: whether they are being asked to evaluate a program that is already in place or a program that might be put in place. Because this experiment included varying descriptions of the program that may or may not apply to the existing NSA program, we asked the question in the conditional verb tense: “Would you favor or oppose the government collecting data…” The other survey, which was intended to test support for the actual program, uses the present verb tense: “Do you approve or disapprove of the government’s collection of data…” While this distinction may seem minor, it raises a fundamentally different consideration in people’s minds: the difference between something the government might do as opposed to something it is doing.

The experiment found little difference in how partisan and age groups reacted to the different programs descriptions. For example, the distinction between monitoring phone vs. email communications had no effect on any age group, while mentioning court approval raised support by similar levels across all age groups.

Similarly, the wording experiments drew similar reactions across partisan lines. Mentioning anti-terrorism efforts received comparably greater support for the program from both Republicans and Democrats alike.

  1.  There were four concepts tested each with two different descriptions, combining to form 16 possible combinations: (2 x 2 x 2 x 2 = 16)