Sampling
The typical Pew national survey selects a random digit sample of telephone numbers from telephone exchanges in the continental United States. The random digit aspect of the sample is used to avoid "listing" bias and provides representation of both listed and unlisted numbers (including new numbers not yet listed). The design of the sample ensures this representation by random generation of the last two digits of telephone numbers selected on the basis of their area code, telephone exchange, and bank number. A bank is defined as 100 contiguous telephone numbers, for example 800-555-1200 to 800-555-1299.
The telephone exchanges are selected with probabilities proportional to their size. The first eight digits of the sampled telephone numbers (area code, telephone exchange, bank number) are selected to be proportionally stratified by county and by telephone exchange within county. That is, the number of telephone numbers randomly sampled from within a given county is proportional to that county's share of telephone numbers in the U.S. Only banks of telephone numbers containing three or more working residential numbers are selected.
The sample is released for interviewing in replicates, which are small systematic random samples of the larger sample. Using replicates to control the release of sample to the field ensures that the complete call procedures are followed for the entire sample. The use of replicates also ensures that the regional distribution of numbers called is appropriate. This also works to increase the representativeness of the sample.
In each contacted household, interviewers ask to speak with "the youngest male, 18 years of age or older, who is now at home." If there is no eligible male at home, interviewers ask to speak with "the youngest female, 18 years of age or older, who is now at home." This method of selecting respondents within each household improves participation among young people who are often more difficult to interview than older people because of their lifestyles, and also ensures that we obtain a sufficient number of interviews with men.
Sampling error results from collecting data from some rather than all members of the population. For each of our surveys, we report a margin of sampling error for the total sample and sometimes for key subgroups analyzed in the report (e.g., registered voters, Democrats, Republicans, etc.). For example, the sampling error for a typical People & The Press national survey of 1500 completed interviews is plus or minus 3 percentage points with a 95% confidence interval. This means that in 95 out of every 100 samples of the same size and type, the results we would obtain will vary by no more than plus or minus 3 percentage points from the result we would get if we could interview every member of the population. Thus, the chances are very high (95 out of 100) that our sample will be within 3 points of the true population value. (Also see Why probability sampling)
Cell phones
As the proportion of Americans who rely solely or mostly on a cell phone for their telephone services continues to grow, we periodically sample cellular telephone numbers in addition to landline numbers to assess the possible bias in our landline samples. Our most recent analysis released in January 2008, indicates that when data from both the landline and cell samples are combined and weighted to match the U.S. population on selected demographic measures, the results for key political measures (such as presidential approval, Iraq policy, presidential primary voter preference, and party affiliation) are virtually identical to those from the landline survey alone. (See The Impact of "Cell Onlys" On Public Opinion Polling for more information.) Also see our discussion of Sampling Cell Phones and Conducting Cell Phone Surveys.
Nonresponse
As many as 10 attempts are made to complete an interview at every sampled telephone number. The calls are staggered over times of day and days of the week to maximize the chances of making a contact with a potential respondent. An effort is made to recontact all interview breakoffs and refusals in order to attempt to convert them to completed interviews.
Response rates for our typical telephone polls range from 20 to 25%. These response rates are comparable to those for other major polls. The response rate is calculated using the American Association for Public Opinion Research, AAPOR 3 calculation, multiplying the contact rate (the number of households contacted divided by the total number dialed, accounting for an estimate of nonworking and business numbers) by the cooperation rate (the number of individuals who completed interviews by the total number of households contacted). (For a full description of AAPOR's response rates see Standard definitions)
Nonresponse in telephone interview surveys can produce biases in survey-derived estimates. Survey participation tends to vary for different subgroups of the population, and these subgroups are likely to also vary on questions of substantive interest. Low response rates are not necessarily an indication of nonresponse bias as surveys with low response rates can be more biased because of nonresponse than those with higher response rates.
In order to compensate for these known nonresponse biases, the sample data are weighted in analysis. The demographic weighting parameters are derived from a special analysis of the most recently available U.S. Census Bureau's Current Population Survey (2007 Annual Social and Economic Supplement). This analysis produces population parameters for the demographic characteristics of households with adults 18 years of age or older, which are then compared with the sample characteristics to construct sample weights. The analysis includes only households in the continental United States that contain a telephone. The weights are derived using an iterative technique that simultaneously balances the distributions of all weighting parameters.
But weighting cannot eliminate every source of nonresponse bias. Nonetheless, properly-conducted public opinion polls have a good record in achieving unbiased samples. In particular, election polling - where a comparison of the polls with the actual election results provides an opportunity to validate the survey results - has been very accurate over the years. (National Council on Public Polls evaluation of 2004 Election)
Data analysis
With each of our reports, we provide a "topline questionnaire" that includes all of the questions from that survey with the exact question wording and response options as they were read to respondents. This topline provides the results from the current survey for each question as well as results from previous surveys where the questions were asked.
For our discussion of the results in reports and commentaries, we report differences among groups when we have determined that the relationship is statistically significant and therefore is unlikely to occur by chance. In addition, to support any causal relationships discussed, we have employed more advanced multivariate statistical modeling techniques to test whether these connections exist, although the results of these models may or may not be shown in the actual report.
For most studies, it is our policy to release data from the People & The Press surveys six months after the reports are issued and archive them on our website as quickly as possible. Please visit our data archive page for further information.