As George W. Bush prepares to leave the White House, the United States is in many ways dramatically different from when he took the oath of office in 2001. His first few months as president were largely unremarkable, despite the contentious 2000 election. But the horrific terror attacks of Sept. 11, 2001 greatly altered the course forward.

The attacks transformed American public opinion and fundamentally reshaped Bush’s image. His job approval rating reached 86% by late September. The public expressed broad willingness to use military force to combat terrorism. But then controversies over the build-up to war in Iraq and other Bush policies started to take their toll – at home and abroad.

U.S. forces quickly ousted Saddam Hussein in 2003, but could not create a lasting peace. As the fighting dragged on, Bush won re-election by a narrow margin. In his second term, he failed in his bid to build support for a partial privatization of Social Security. American deaths continued in Iraq, the government bungled the response to the devastating Hurricane Katrina in late 2005 and political scandal reached directly into the White House.

Soon, economic troubles started to mount, and in 2008, the economy went into a dangerous free fall that led to controversial and expensive government intervention in financial markets. The president’s approval ratings slid over time to historic lows. His approval last hit 50% as he started his second term. It stood at just 24% in early December.

Not surprisingly, the public’s verdict on the Bush presidency is overwhelmingly negative. In a December 2008 Pew Research Center survey, just 11% said Bush will be remembered as an outstanding or above average president – by far the lowest positive end-of-term rating for any of the past four presidents. Yet Bush’s impact on public opinion over the past eight years is seen in ways that go well beyond his personal unpopularity.

He helped shape the post-9/11 climate of opinion that was broadly accepting of a muscular approach to U.S. national security. And even after much of the public came to oppose the war in Iraq, there continued to be considerable support for the Bush doctrine of preemptive military action. In spite of the public’s shock over pictures of abuse of detainees at the U.S.-run Abu Ghraib prison, nearly half of Americans consistently said that the torture of terrorists to gain key information was at least sometimes justified.

In the final year of his presidency, even as his approval rating steadily declined to historic lows, most Americans continued to say Bush’s anti-terror policies deserve at least a fair amount of credit for preventing more terror attacks.

In his first term, Bush scored several early legislative successes on domestic issues – such as the No Child Left Behind education reform, two rounds of tax cuts and the launch of a significant Medicare drug plan.

But after those successes, the instances when Bush was able to mobilize – and maintain – public support for his agenda were rare. Even in the realm of national security, the public increasingly rejected the idea that a large military presence overseas would reduce the threat of terror at home.

Public backing for what was to have been Bush’s signature second-term achievement – reforming the Social Security system – withered within months of his reelection. His immigration reform proposal faced opposition within his own party, even though it was an issue – like Social Security – where Americans recognized major change was needed.

What might have damaged Bush’s legacy most was his administration’s mixed record of competent governance. Between Iraq, the government’s flawed relief effort in the aftermath of Hurricane Katrina, and more minor missteps over the Dubai ports issue and other matters, the government “brand” deteriorated badly during the Bush years. In late April 2008, just 37% expressed a favorable view of the federal government, about half of the percentage of five years earlier (73%).

Final Judgments

In a Pew survey conducted Dec. 3-7 among 1,489 adults, the American public paints a harshly negative picture of Bush’s tenure. Nearly two-thirds (64%) say his administration will be remembered more for its failures than its accomplishments, and a plurality (34%) says Bush will go down in history as a poor president. Fully 68% say they disapprove of Bush’s performance and most of those – 53% of the public – say they disapprove strongly. That is the highest rate of strong disapproval measured by Pew surveys in Bush’s eight years in office.

As his second term ends, only 13% say Bush has made progress toward solving the major issues facing the country; 37% say he has made those problems worse and 34% say he has tried but failed in his efforts. Another 11% say he has not addressed the major problems facing the country.

More than three times as many people say Bush will go down in history as a poor president (34%) than said the same of Bill Clinton at the end of his presidency (11%). About a quarter (24%) say Bush will be seen as below average and close to three-in-ten (28%) say he will be seen as average. Just 11% say he will go down in history as above average or outstanding.

Not surprisingly, the most critical assessments come from Democrats. More than half (53%) say Bush will go down in history as a poor president, while 25% say he will be remembered as below average. More than four-in-ten Republicans (44%) say he will be remembered as an average president; 21% say below average and 6% say poor. Two-in-ten say he will be remembered as above average, while 7% say outstanding.

Americans by a wide margin (64% to 24%) also say that in the long run the failures of the Bush administration will outweigh its accomplishments. The assessment of Clinton in 2001, despite controversy over how he had conducted himself in office, was virtually a mirror image. Six-in-ten said the accomplishments would outweigh the failures, and 27% said the failures would outweigh the accomplishments.

Just over half (52%) of Republicans say the Bush administration will be best known for its accomplishments. That number is significantly smaller for independents (20%) and Democrats (8%). When Clinton was leaving office, his own party (77%) and independents (60%) were much more convinced he would be remembered for his accomplishments.

Second Term Approval Slide

Between the start of his second term and December 2008, Bush’s approval rating dropped from 50% to 24%, a level that rivals the historic lows recorded by Gallup for Harry S Truman as he left office in 1952. Declines came across demographic and political groups, though significant divides still exist among those with differing political ideologies.

Approval among moderate and liberal Republicans saw one of the sharpest drops – from 82% to 50%. Conservative Republican approval dropped from 94% to 66%. Independent approval started at below half in 2005 – 47% – but dropped to 18% by December 2008. The change among Democratic groups, already highly critical of Bush, proved less dramatic. Approval among conservative and moderate Democrats dropped from 22% t
o 8%, while approval among liberal Democrats dropped from 7% to 2%.

Bush’s approval dropped significantly among all education levels. In terms of age groups, the largest decline came among the youngest voters – those age 18-29. Within that group, approval dropped from 50% to 19%. The oldest group – age 65 and up – experienced a smaller decline, dropping from 47% to 26%.

A Legacy of War

When people are asked what they think Bush will be most remembered for after he leaves office, the most frequent responses volunteered are tied to the wars in Iraq and Afghanistan and the fight against terrorism. More than half (51%) of responses mention facets of the Global War on Terror, with close to three-in-ten (29%) specifically mentioning Iraq.

About 17% include specific negative assessments of Bush and his performance. Small percentages within that group refer to his competence (2%), his negative impact on the country (2%) or label him the “worst president” (2%). Another 13% refer to the impact of the Sept. 11 terror attacks, with 9% mentioning the attacks specifically and 3% noting that Bush had kept the country safe from major attack since that day. Another 12% mentions economic issues, including 7% who refer to the economy specifically, 4% who mention the current crisis and 3% referring to the recession. Another 4% offer positive assessments of Bush’s performance in office.

Meanwhile, concerns about Bush’s effectiveness are also seen in the one-word answers people give to describe their impressions of the president. In mid-2005, positive one-word descriptions outnumbered negative ones, but in more recent lists, the responses have been more negative. The most frequently mentioned description in the latest survey was “incompetent,” just as it was in February 2007 and March 2006. Many of the terms offered by respondents in the December 2008 survey are negative, though the second most frequently mentioned description was honest. Good and honorable also make the list.

The Global Outlook

On the foreign stage, a solid majority of Americans say the country is significantly less respected than in the past – and many of those people see that as a major problem. Many Americans are eager to turn inward to deal with this nation’s problems: fully 60% said in September 2008 that domestic policy should be the primary focus of the new president. And a greater percentage than before the Iraq war now say the best way to reduce the threat of terror is to reduce America’s military presence overseas, not increase it.

Meanwhile, a 2008 survey by the Pew Global Attitudes Project showed that majorities in 19 of the 24 nations – including several strong U.S. allies – had little confidence in Bush as he neared the end of his presidency. A 2007 survey of 45 nations found anti-American sentiment extensive as well as increasing disapproval for key elements of U.S. foreign policy.

And the image of the U.S. in the Muslim world remained abysmal. Iraq, the war on terrorism and American support for Israel continued to generate animosity in the Middle East, Asia and elsewhere. In many nations considered central to the war on terror, the general public deeply distrusted the United States. Even in countries like Kuwait that have long been considered relatively pro-American, the U.S. image had declined.

Among the few bright spots for Bush in the Global Attitudes surveys were the African nations that had benefited from administration programs to boost economic growth and reduce the spread of AIDS. In 2008, majorities in Tanzania and Nigeria expressed confidence in the president.

Still, Bush has had some success at home building support for tough tactics – including harsh interrogation policies for foreign detainees and government monitoring of phone calls or e-mails without warrants – to gather information about possible terrorists and stop potential attacks. On balance, more Americans say they worry that anti-terror policies have not gone far enough in protecting the United States than say they feel the anti-terror policies have “gone too far in restricting civil liberties.”

The Political Legacy

When Bush took office, Republicans controlled both Congress and the White House. But voter party preferences shifted significantly during Bush’s second term as missteps, bad news and scandals took their toll on Bush and GOP congressional leaders. In the 2006 midterm elections, more independents and moderates aligned themselves with the party out of power and Democrats took control of the House and the Senate.

In 2008, Bush was barely seen during the presidential campaign. Both Barack Obama and John McCain persistently criticized his administration, vowing to bring “change” to Washington. Obama’s significant win and additional Democratic gains in Congress signaled a continuing decline of the Republican Party under Bush.

In surveys conducted in the fall of 2008, 51% of all voters said they thought of themselves as Democrats or leaned toward the Democratic Party. That was up five points from 46% during the same period in 2004. Meanwhile, the number identifying with or leaning toward the Republican Party fell from 45% to 41%.

The greatest gains came among younger voters. Only among voters age 65 and older did the percentage identifying with the Democratic Party decrease – from 49% in 2004 to 47% in 2008. The percentage of voters age 18 to 29 identifying with the Democratic Party increased from 48% in the fall of 2004 to 61% in the fall of 2008. Democrats now outnumber Republicans by a margin of nearly two-to-one (61% to 32%) in this age group, up from only a seven-point advantage in 2004.

The proportion of Americans who rely solely on a cell phone for their telephone service continues to grow, as does the share who still have a landline phone but do most of their calling on their cell phone. With these changes, there is an increased concern that polls conducted only on landline telephones may not accurately measure public opinion. A new Pew Research Center study finds that, while different demographically, Americans who mostly or exclusively rely on cell phones are not substantially different from the landline population in their basic political attitudes and preferences.

On key political measures such as presidential approval, Iraq policy, presidential primary voter preference, and party affiliation, respondents reached on cell phones hold attitudes that are very similar to those reached on landline telephones. Analysis of two separate nationwide studies shows that including interviews conducted by cell phone does not substantially change any key survey findings.

These findings are based on two surveys of adults, conducted Oct. 17-23 and Dec. 19-30, 2007 by the Pew Research Center for the People & the Press. The surveys included interviews with a total of 2,596 adults reached in a conventional landline sample, as well as 841 adults interviewed on their cell phones, using a sample drawn from a nationally representative cell telephone number database. Of those reached on a cell phone, 312 people (or 37%) reported that their cell phone is their only phone.

When data from both samples are combined and weighted to match the U.S. population on key demographic measures, the results are virtually identical to those from the landline survey alone. Across more than 100 political and attitudinal questions on the surveys, including cell phone interviews does not change the results by more than two points in the vast majority of comparisons, and in only one comparison is the difference as large as 4 points.

In particular, there is no evidence that the polling in the Democratic and Republican nomination contests is biased by the fact that most polls rely only on landline interviews. In the December national poll, support for no candidate in the landline sample changed by more than two points when the preferences of cell phone respondents were blended in. The same was true in the October national poll.

There is no doubt that Americans who rely solely on cell phones differ from the rest of the public in some key respects. However, in most cases these differences are the result of their demographic characteristics, particularly the fact they tend to be very young. Since adjustments for age are made in standard landline surveys, adding the cell-only component to the survey substantially increases the raw number of younger people surveyed, but does not alter the overall weight of younger respondents in the final estimates.

In most respects, the political attitudes and behaviors of younger people who are cell-only do not differ substantially from younger people surveys do reach on landlines, meaning that the overall results are virtually identical to those from the landline survey alone.

However, on some non-political topics, and in surveys of certain groups in addition to young people, studies have shown that the inclusion of cell phones in the sample design makes a difference in the combined results. An earlier study by the Pew Research Center for the People & the Press found that blending landline and cell phone samples resulted in higher estimates of young people ages 18 to 25 using new technologies. In addition, small but significant differences were found for lifestyle measures such as attending church and alcohol consumption. In another study from the National Health Interview Survey, Blumberg and Luke (2007) found that for surveys of low-income adults and young adults, the estimates for health risk behaviors, HIV testing, exercise and obesity were all changed when cell phones were included in the sample.

In addition to testing the impact of cell phone sampling, the October and December Pew studies demonstrate the feasibility of including cell phones in telephone surveys. The response rates for the cell and landline samples were virtually identical in both studies, and there is no evidence that the quality of data gathered from cell phone surveys is lower than in landline surveys. Including cell phones, however, is very costly. On average, a cell phone interview costs approximately three times as much as a comparable landline interview.

Although the inclusion of cell phone samples is very costly, and may make little difference in the substantive conclusions one would draw from political surveys, other aspects of the dual frame design provide particular benefits that may argue for the adoption of this type of sampling frame design. Chief among these benefits is the improved demographic representation for certain groups and the attendant increase in the sizes of the samples of these groups for further analysis. This is because it is easier to reach by cell phone than by landline certain groups of respondents who have both types of service.

The inclusion of a cell phone sample may be essential in surveys of population groups that have high rates of cell-only households. More generally, with an estimated 14% of Americans relying solely on cell phones, their exclusion from opinion surveys may call into question the credibility of polls in the mind of the public.

Overview of Differences

Results from both the December and October polls show that the cell-only respondents have somewhat different attitudes and behaviors from those reached on landline telephones. In the December survey, which focused on the public’s campaign news sources, cell-only respondents were significantly less likely to say they have watched a presidential debate on television, but more likely to have seen debate video online. This reflects a more general pattern: cell-only Americans are somewhat less likely to rely on newspapers and network evening news for campaign information, but more apt to get campaign news from the internet, late night comedy shows, and to use social networking sites. Not surprisingly, these behaviors are characteristic of younger respondents in general – whether cell-only or not – and the blended results for none of these measures change by no more than two percentage points.

The October survey included questions that asked registered voters about the importance of 16 issues to their vote. There were a few significant differences between the landline respondents and those who were cell-only: the latter group was 14 points less likely to say Social Security would be important to their vote, and somewhat more likely to say immigration would be important. Again, these differences are understandable, given the fact that cell-only respondents are younger (and thus less concerned about Social Security) and more likely to be Hispanic (who are more concerned about immigration). When the cell-only respondents were combined with the landline respondents, none of the overall survey estimates changed by more than one percentage point.

While including interviews conducted by cell phone in a national sample does not substantially affect survey findings, it does improve the overall representativeness of the sample by reaching more respondents in otherwise hard to reach subpopulations. This reduces sampling error for these groups, and may also mean that the survey requires less statistical adjustment to match the demographic profile of the population. Less clear is whether adding cell phone interviews is the most efficient use of resources. Cell phone interviews cost approximately three times as much as landline interviews, and the sample sizes of underrepresented groups can be boosted more cheaply by simply expanding the overall sample size of the landline survey.

Profile of Cell-Only Respondents

One of the most striking differences between cell-only respondents and people reached on a landline telephone is their age. Nearly half of the cell-only respondents (46%) are under age 30 compared to only 12% in the landline sample. Related to their younger age, only 26% of cell-only respondents are married, compared with 57% percent of those in the landline sample. Similarly, about half of cell-only respondents have never been married (51%), compared with only 16% in the landline sample.

In addition, the landline sample includes a higher proportion of college graduates than the cell-only group (38% vs. 26%), which may also reflect the greater use of cell phones among young people who are still in college. The income distribution also is quite different for the landline and cell-only groups; 29% of people in the landline sample have household incomes of at least $75,000 annually, compared with just 16% in the cell-only group. Similarly, nearly twice as many cell-only than landline respondents earn less than $30,000 a year (41% vs. 21%).

Overall, the landline sample includes more whites (82% vs. 68%) than the cell-only group while the cell-only group includes a greater proportion of minorities. In the cell-only group, there are more African-Americans (19% vs. 11%), Hispanics (13% vs. 6%), and Asians (5% vs. 1%) compared with the landline sample. The cell-only group also includes a larger percentage of males than the landline group (61% vs. 48%). Finally, more cell-only respondents than landline respondents are religiously unaffiliated (27% vs. 14%).

The “Dual” Households

In this study, cell phone interviews were conducted with cell-only individuals (those who have no landline phone), as well as with those who were reached by cell phone but also have a landline telephone. Since these so-called dual-phone respondents could, in fact, be contacted on a landline telephone, some prior studies did not interview them, focusing only on those reachable only on a cell phone.

However, the current study includes dual-phone respondents regardless of whether they were reached on their landline or cell phones. This choice reflects the fact that about half (47%) of the dual-phone respondents who were reached on their cell phone say that they receive more of their calls on their cell phone, in most cases a lot more. While it may be possible to reach these respondents on their landline telephone, it may be more difficult to do so.

The crux of the issue is whether the dual users reached by cell phone are different from those reached by landline. For the most part, the answer is no. Among the dual users, more males than females were reached by cell phone (56% male, compared with 48% male among dual users reached by landline). And more than twice as many Hispanics were reached by cell phone (11% vs. 5%). Those reached by cell phone were somewhat younger (57% under age 50, compared with 47% among those reached by landline). Across a broad range of attitudinal questions in the two surveys, there was very little difference between the dual users reached by cell phone and those reached by landline.

Young Landline vs. Cell Users

In some respects, young people who rely solely on cell phones are quite different demographically from young people who have landline telephones. Much of the difference is driven by the fact that, even within the 18-29 year-old age group, the average age of cell-only respondents is much younger than of landline respondents. Among respondents under age 30, a greater proportion of cell-only respondents than landline respondents are under age 25 (70% vs. 55%). In part because of their younger age, fewer young cell-only people are married (15% vs. 32%) and fewer have children (19% vs. 31%). Nearly half of young people under the age of 30 who rely exclusively on their cell phones (48%) have household incomes of less than $30,000 a year, compared with about a third (32%) of those in the same age category with landline telephones. There also is a substantial gender difference, with men outnumbering women in the cell-only sample (62% vs. 38%), compared with a more even balance in the landline sample (48% male, 52% female).

However, there are no significant differences in education between young people with landlines and those that are cell-only. While both groups have comparable numbers of whites and African Americans, a greater proportion of cell-only people are Asian (8% vs. 2% of the landline sample) Finally, fewer cell-only young people than those with landlines attend religious services once a week (24% vs. 36%) probably because more are religiously unaffiliated (36% vs. 26%).

Although cell-only and landline users under the age of 30 differ demographically, there are very few differences in their political attitudes, ideology, and partisan affiliation. Comparable majorities of young people in the landline and cell-only samples express dissatisfaction with the way things are going in the country, and about the same proportions in both groups disapprove of President Bush’s job performance. Slightly more cell-only than landline people affiliate with the Democratic Party; however, ideologically, more cell-only people report they are conservative than their landline counterparts and neither of these differences are significant.

When it comes to the campaign, young people who are cell-only report that they regularly learn about the campaign from a larger number of news sources (an average of 1.98 “regular” sources per person) than do those with landline telephones (an average of 1.66). In contrast, slightly fewer cell-only young people say they have given a lot or some thought to the 2008 presidential campaign and slightly fewer are registered voters than are those with landlines. Among those registered to vote, there are no differences in whether young people are likely to vote in the presidential primary.

Young people have similar views on the situation in Iraq regardless of whether they rely exclusively on their cell phones or have landline telephones. Identical percentages of cell-only and landline young people (55% each) say that the United States made the wrong decision in using military force against Iraq and that the U.S. military effort there is not going well (53%). Compared with those who have landlines, a slightly larger number of cell-only young people think that the U.S. should keep military troops in Iraq until the situation is stabilized (43% vs. 35%).

Overall, these results suggest that the political attitudes of young people do not vary much by telephone status. As a result, while their inclusion in the study substantially increases the number of younger people interviewed, it does not substantially change overall survey estimates.

Practical Considerations in Conducting Interviews on Cell Phones

This study and several others conducted by the Pew Research Center, as well as those by other survey organizations, demonstrate that it is feasible to conduct random sample surveys by cell phone. But the process is costly, requiring significant additional effort by the survey field house and some additional work in data processing and weighting. Exclusive of the fixed study costs such as CATI programming, pre-testing surveys, creating demographic banners, the marginal cost of a cell phone interview in these two studies was approximately three times larger than the marginal cost of a landline interview. And in terms of reaching the most critical “cell-only” respondents, previous studies suggest that such interviews cost four to five times more than comparable landline interviews, largely because of the additional screening necessary to locate cell-only respondents.

The cost differential for calling cell phones is a result of several operational differences between calling in the landline and cell sample frames. One of the largest differences results from the fact that, due to federal regulations, telephone numbers in the cell frame must be manually dialed by the interviewer. For landline numbers, an “auto-dialer” is used to take a number from the sample and actually dial it before transferring the call to the interviewer.

Another difference is that a significant number of people reached in the cell frame turned out to be under the age of 18 and thus ineligible for the survey. In fact, more than four-in-ten (42%) of the cell phone respondents who were willing to cooperate with the survey could not be interviewed because the phone belonged to an underage person. None of the cooperating households in the landline frame was excluded because they contained no adults. This aspect of the cell sample, along with the fact that the cell phone frame reaches a higher percentage of individuals who do not speak English, meant that the percentage of contacted individuals eligible for the survey was far lower in the cell frame – just 45% and 40% in October and December, respectively, compared with 86% and 85% in the landline frame.

A third difference is that respondents in the cell frame were offered a modest cash reimbursement to offset the cost of airtime they might incur while taking the survey. Beyond the expenses incurred, the collection of contact information in order to reimburse respondents, and the attendant administrative and processing costs, adds to the overall cost of interviewing in the cell frame. The vast majority of respondents (85% in October and 80% in December) who agreed to participate in the interview provided the necessary name and mailing address to receive the reimbursement.

To test the potential impact of different amounts of reimbursement, cell phone respondents in October were randomly assigned to be offered either $10 or $20. Somewhat surprisingly, there was virtually no difference in the response rate between those offered $10 and those offered $20. (There also was no difference in the percentage of cooperating respondents who provided a name and address for reimbursement at the end of the interview.)

Apart from the eligibility rates and the cost differential, however, there were remarkable similarities between the cell and landline samples in several aspects of the fieldwork. The contact and cooperation rates between the cell and landline samples were nearly identical. Similarly, the breakoff rate – the percentage of people who begin the interview but do not complete it – was the same in each sample. As a result, overall response rates were very similar in the cell and landline samples – 23% in each sample in October, and 18% in the landline sample in December – compared with 22% in the cell sample.

Quality of Responses: Landline vs. Cell Phone Interviews

Differences in the ways that people use landline telephones and cell phones could potentially affect the quality of data collected in surveys sampling both kinds of phone numbers. For example, if people are more distracted or more accustomed to short conversations on a cell phone compared to when they use a landline, then they may not respond as carefully when interviewed on a cell phone. However, studies on this topic have found no substantive differences between the quality of answers recorded in landline interviews and those recorded in cell phone interviews. Results from recent Pew surveys are generally consistent with this finding.

People interviewed in cell samples were less likely to refuse to answer or say “don’t know” on at least one question than those interviewed in the landline sample. This result, however, simply reflects the different characteristics of people reachable by landline versus those reachable on a cell phone. For example, adults ages 60 and older are more likely than younger people to decline to answer questions; they also are much more likely to be interviewed in the landline sample. After accounting for such demographic differences, there is no perceptible difference in the rates of Refused/Don’t know responses between cell phone and landline samples.

Another way to gain insight into how carefully people respond is through interviewer evaluations. Immediately after completing each interview, interviewers recorded their impressions of the respondent’s level of cooperation and level of distraction (each on a four-point scale).

There is a slight suggestion that the cell sample respondents were more cooperative and less distracted than those reached on landlines, but again the difference may be attributable to factors other than the type of phone used by the respondent. The difference in the age distributions of the two samples is one factor. The monetary reimbursement, which was offered only to persons in the cell sample, also may have an effect. Presumably, cell sample respondents’ knowledge that they would be remunerated had a positive effect on their attitude during the interview. By this logic, if an incentive had been offered to the landline sample as well, the rates of cooperation would be even more similar.

Benefits of Conducting Cell Phone Samples

Surveys that rely only on landline interviews are more likely to produce biased estimates if the segment of the public unreachable on a landline differs substantially from the landline public. If the cell-only respondents are not very different from the landline respondents, the survey estimates will not be biased by the absence of the cell-only group. For example, the landline survey finds that 54% of Americans favor bringing troops home from Iraq; among the cell-only respondents, 55% favor a U.S. troop withdrawal. Thus the overall survey estimate is unaffected when the cell-only respondents are blended in. One way to consider the impact of adding cell-only interviews to a survey is to ask the question: How different would the cell-only have to be for the total survey estimates to be affected by their inclusion?

For example, in the unlikely instance that 100% of the cell-only adults favored a troop withdrawal from Iraq, and landline respondents remained divided (with 54% favoring withdrawal), then the combined survey estimate would shift to 59% – a five-point increase. The standard survey alone would underestimate national support for withdrawal. Alternatively, if just 30% of the cell-only respondents favored withdrawal, the combined estimate would be 50%, four points below the current estimate from the landline survey.

These effects are potentially greater when analyzing subgroups in the population, such as young people, who are less likely to be reached on a landline. For example, if 100% of cell-only young people (ages 18-29) favored a troop withdrawal, the combined sample estimate for this age group would be 72% in favor of withdrawal, rather than the 60% that the landline sample of young people produces.

Guarding Against Bias

Findings reported here and in other studies demonstrate that standard landline samples still perform well relative to more expensive designs that combine landline and cell phone samples. Currently, this holds true for most overall population estimates. The potential for bias, however, is greater for estimates for subgroups that tend to rely more on cell phones, such as young adults, blacks, Hispanics.

Indeed, for such groups, several standard sample estimates differ from the corresponding combined sample estimates. For example, 46% of Hispanics align with the Democratic Party, based on the standard landline sample. Based on the combined sample, however, 43% of Hispanics consider themselves Democrats. On other items, the standard and combined samples yield similar results, even on estimates for young adults.

When there is a difference between the standard and combined estimates, the natural question is which figure is more accurate. Benchmark data from the American Community Survey (a large multi-mode survey conducted by the Census Bureau) shows that the answer varies.

The combined survey sample yields more accurate estimates for Hispanics on two of the characteristics evaluated here. With regard to African Americans, the combined sample estimate of the proportion of the black population who are parents of children under age 18 is more accurate than the corresponding landline sample estimate. However, the combined sample estimate for the marriage rate among blacks is less accurate. For all 18-29 year olds, the combined sample appears to be slightly less biased in estimating the marriage rate and the proportion who are parents of children under 18.

These results demonstrate that a combined sample is not always superior to a standard sample (and vice versa). This may seem counterintuitive given that the combined sample, by definition, does a better job covering the population (both landline and cell phone users). The primary explanation for the shortcomings of both the standard and combined designs appears to be non-response: Everyone with a telephone has a chance of being interviewed in the combined design, but most either do not answer the call or decline to be interviewed. Those who do respond in landline or cell samples sometimes differ systematically on items in the survey from those who do not participate.

Sample Sizes of Groups Relying Mostly on Cell Phones

One potential advantage of a dual-frame survey is that it may be possible to complete more interviews with groups who rely more on cell phones. For example, 28% of cell phone respondents are under age 30. This is more than double the rate of young adults in landline samples (12%). Thus, a sample of 1,000 cell interviews would yield roughly 280 adults age 18 to 30, while an equally-sized sample of landline numbers would yield roughly 120 adults in this age group.

Having a larger sample size is important because it means more precise estimates. Roughly speaking, the margin of error on an estimate for young adults is 6% with a sample size of 280. With the smaller sample size of 120, the estimate is less reliable and the margin of error is about 9%.

Currently, these advantages are not being realized, largely because of the cost. Cell phone interviews are approximately three times more expensive than landline interviews. Young adults, however, are not three times more likely to be reached in the cell sample (only about twice as likely). When the survey budget is held fixed, the most effective way to maximize the number of interviews – even for groups like 18-29 year olds who rely more heavily on cell phones – is to allocate the entire budget to increasing the overall number of landline interviews. This is because roughly three landline interviews can be completed for the same cost as every one cell phone interview.

Over time the cost differential between landline and cell interviews may narrow. It also is possible that the prevalence of various subgroups may become lower in landline samples and higher in cell samples. Such developments would imply greater sample sizes under a dual frame design (for fixed cost) relative to sample sizes expected under current conditions.

A growing number of Americans rely solely on a cell phone for their telephone service, and many more are considering giving up their landline phones. This trend presents a challenge to public opinion polling, which typically relies on a random sample of the population of landline subscribers. A new study of the issue finds that cell-only Americans — an estimated 7%-9% of the general public — are significantly different in many ways from those reachable on a landline. They are younger, less affluent, less likely to be married or to own their home, and more liberal on many political questions.

Yet despite these differences, the absence of this group from traditional telephone surveys has only a minimal impact on the results. Specifically, the study shows that including cell-only respondents with those interviewed from a standard landline sample, and weighting the resulting combined sample to the full U.S. public demographically, changes the overall results of the poll by no more than one percentage point on any of nine key political questions included in the study.

Estimates of the respondents’ likely congressional vote this fall, approval of President Bush, opinion about the decision to go to war in Iraq, and other important social and political measures are unaffected when cell-only respondents are blended into the sample. The relatively small size of the cell-only group, along with the demographic weighting performed when it is combined with the landline sample, accounts for the minimal change in the overall findings.
This research effort was undertaken by the Pew Research Center, in conjunction with the Associated Press and AOL, to assess the challenge posed by cell phones to random digit dial surveys. The project entailed a survey of 1,503 U.S. adults, with 752 interviewed in a conventional landline sample and 751 interviewed on their cell phones, using a sample drawn from a nationally representative cell telephone number database. The interviews were conducted March 8-28, 2006 and averaged about 11 minutes in length. Among those interviewed on their cell phones, 200 (27%) said that their cell phone was their only phone. Details about the survey, including response rates, costs, and other issues, are discussed in the body of the report below.

The Cell Phone Challenge

The number of people who have given up their landline telephones and rely solely on a cell phone has been increasing, both in the U.S. and internationally, for several years.

According to the U.S. Consumer Expenditure Survey, the percentage of households paying a cell phone bill but not a landline bill rose from 0.4% in 2000 to 7.8% in the first quarter of 2005.

The National Health Interview Survey estimated that, in the second half of 2005, 7.8% of adults lived in households with only a cell phone. And in the 2004 exit poll by the National Election Pool, 7.1% of voters said they relied solely on cell phones.

As the cell-only population has grown, telephone surveys by Pew and other organizations that rely on landline samples have experienced a sharp decline in the percentage of younger respondents interviewed in their samples. In Pew Research Center surveys over the past five years, the average percentage of those ages 18-34 in unweighted samples declined from 31% in 2000 to 20% through March 2006 (the population parameter was essentially unchanged through this period). This decline is consistent with the fact that the cell-only population is heavily tilted toward young people.

Profile of Cell-Only Respondents

Nearly half of the cell-only respondents in the survey (48%) are under age 30. This compares with just 14% in the landline sample (people reached on a landline) and 21% in the population as a whole, according to government statistics. Other characteristics associated with age are also distinctive in the cell-only population. Nearly three-in-ten (29%) cell-only respondents are married, compared with 57% in the landline sample. And only 24% say they own their own home; in the landline sample, 71% do so. The cell-only population also includes a higher proportion of minorities, especially Hispanics (14% vs. 6% among landline users).

The landline sample includes a higher proportion of college graduates than does the cell phone-only group (36% vs. 28%). But more cell-only users say they have some college experience compared with people who have landlines (by 33% vs. 24%); this may reflect the heavy reliance on cell phones among those currently attending college. The cell-only group also is significantly less affluent — more than half (53%) have annual family incomes of under $30,000, compared with just one-quarter (25%) among the landline sample.

Young Cell vs. Landline Users

Young people who rely exclusively on cell phones also are very different — in their lifestyles and family circumstances — from their landline counterparts of similar age. Far fewer cell-only people under age 30 are married, have children, or are homeowners when compared with landline users in this age category. Related to these factors, young cell-only respondents have significantly lower family incomes than young people in the landline sample.

But young cell-only users and landline users do not differ widely in their political attitudes and partisan affiliation. It is true that the cell-only young respondents are more likely to approve of Bush’s performance in office than are under-30 landline respondents (35% vs. 22%). On most other issues, however, they are more liberal and Democratic than their landline counterparts, though most of the differences do not achieve statistical significance. The modest nature of all of these differences suggests that young people — whether cell-only or not — are more similar than different politically.

Seniors Stick With Landlines

According to data collected by the National Center for Heath Statistics, 53% of Americans use both a landline and a cell phone; 37% have only a landline; and 8% rely only on a cell phone.

Like the cell-only population, Americans who rely solely on a landline are distinctive demographically. Fully 41% are ages 65 and older, compared with 16% of the general public. The landline-only group includes a greater proportion of whites than the general public (82% vs. 73%).

Among dual phone users, there are clear differences between those reached on a cell phone and those contacted on a landline. People who were interviewed on a cell phone are somewhat younger (24% under age 30 vs. 15% among those reached on a landline), more likely to be Hispanic (9% vs. 5%), and slightly more likely to have a child under 18 in the household (43% vs. 35%).
Politically, the landline-only and cell-only groups stand out as more Democratic — both in their congressional vote intention and party affiliation — than do those who have both types of phone service. Yet there are only modest differences in approval of President Bush among these four groups.

More striking is the wide divide in views about gay marriage. About half of the cell-only population (51%) favors allowing gay marriage, compared with 39% of the dual phone users and just a third of those who have only a landline phone (33%).

This difference mostly reflects the age patterns of these samples. Pew surveys have consistently found that young people — who make up about half of the cell-only population — are more supportive of gay marriage than are older Americans.

And Pew surveys show that people ages 65 and older, who make up a disproportionate share of the landline-only group, are the most opposed to gay marriage.

Patterns of Cell Phone Use

As might be expected, a solid majority of respondents in the cell phone sample who also have a landline (62%) say that they make more calls on their cell; nearly half (47%) say they make a lot more phone calls on their cell phone. Dual phone owners from the landline sample use landlines only somewhat more frequently than their cell phones; about half (48%) report making more of their calls on their landline while 42% say they make more calls on their cell phone.

Fully 91% of all respondents in the cell sample keep their cell turned on always or most of the time, compared with 73% of cell owners from the landline sample. A small but notable segment (12%) of cell owners from the landline sample say they rarely turn their cell on or do so only to make a call. Hardly anyone from the cell sample (2%) reported having their cell on this infrequently.

Consequently, heavy users of cell phones are more easily reached and interviewed on their cell phones than are lighter users, resulting in a potential bias on some types of measures. One illustration of this is the fact that 27% of respondents in the cell sample identified themselves as cell-only. But U.S. government estimates indicate that only about 13%-15% of cell owners (approximately 7%-9% of the general public) are cell-only.

People in the cell sample use more cell phone features and options than do cell owners from the landline sample. More people in the cell sample say they use a cell to send and receive text messages (45% cell sample vs. 30% landline sample), take still pictures (39% vs. 22%), and surf the web (18% vs. 13%). Three-quarters of those in the cell sample (75%) have personalized their cell phone by changing the wallpaper or ring tone, compared with 59% of cell owners in the landline sample.

Most people in both samples use only one cell phone, and most do not share their cell phone with others. About one-in-five (19%) of those reached in the cell sample say they regularly use more than one cell phone; the comparable number in the landline sample was 14%. And in each sample, 16% said that another adult regularly answers their cell phone.

Dropping Your Landline?

About a quarter of landline users (23%) say they are very (8%) or somewhat likely (15%) to stop using their landline and switch instead to using only a cell phone. A narrow majority (55%) says they are not likely at all to give up their landline in favor of a cell phone. As may be expected, far more young people than older Americans say they are at least somewhat likely to abandon their landline; 40% of those under age 30 say this compared with 19% of current landline users ages 30 and older.

Implications for Tech-Focused Surveys

Asked about their general opinion of computers and technology, cell-only respondents are much more positive toward computers and technology than are landline-only respondents, and somewhat more positive than other cell phone users who are accessible on a landline.

But there is little difference between the cell-only respondents and cell phone users reached on a landline in their use of the internet and their access to broadband. The only significant difference in internet use is how the respondent gets service: cell-only users are less likely than others to use DSL or a dial-up line.

Challenges of Cell Phone Interviews

In addition to providing a look at the cell-only population, this study was designed to assess the feasibility of conducting a telephone survey in a cell phone sampling frame. The conclusion is that such surveys are feasible, but they are more difficult and expensive to conduct than landline surveys.

Because most cell phone users have to pay for incoming calls (or use pre-paid minutes for them), a $10 incentive was offered only to respondents in the cell phone sample. Despite this inducement, gaining cooperation from people on cell phones was notably more difficult than for those on a landline phone.

The response rate was 30% in the landline frame but only 20% in the cell phone frame. It was actually easier to make contact with a respondent through the cell phone frame (the contact rate was 76% in the cell frame vs. 68% in the landline frame). But that greater accessibility did not translate into more cooperation. Half of the people reached in the landline sample (50%) cooperated with the interview, compared with roughly a quarter (28%) of those reached in the cell phone sample.

Aside from difficulties in gaining cooperation, the process of sampling cell phone numbers proved to be reasonably efficient. More of the cell phone numbers (59%) were connected to eligible respondents than were numbers in the landline sample (43%).
Interviewing people on cell phones presents several challenges that require new procedures and have implications for overall costs. Among the most important of these is the fact that federal law prohibits the use of automated dialing devices when calling cell phones; thus each number in the cell phone sample had to be dialed manually.

The $10 incentive offer incurred additional costs. An overwhelming majority of cell phone respondents who completed the interview (86%) accepted this offer and provided a mailing address to which the incentive was sent. In addition to the money paid to the respondent, the use of an incentive also incurs additional administrative work that raises the cost of the survey.

Results from the study suggest that interviews on a cell phone take about the same amount of time to complete as interviews on a landline phone. The same questionnaire was administered to both samples, and the median length was 11 minutes (mean = 11.8) for the cell phone sample and 10 minutes (mean = 10.2) for the landline respondents who reported owning a cell. Most of the small difference in average length between the two sampling frames is likely due to the extra time spent by the cell sample respondents in providing a mailing address for mailing the $10 incentive.

Cell phones tend to be personal devices, and many adolescents and younger children have their own phone. One consequence of this is that more people reached in the cell frame turned out to be ineligible because of their age than is typically the case in a household-based landline sample. Of people contacted in the cell phone frame, 45 cases were dropped from the study because the respondent was under 18. In the landline sample, only 6 cases were dropped because the sampled telephones were used exclusively by children.

Because people may not be accustomed to speaking with an unknown caller on their cell phone, two other modifications in Pew’s regular protocol were used. The survey introduction included the acknowledgement that the respondent had been reached on a cell phone, and an immediate question as to whether it was safe to do an interview at that time. If the interviewer reached voice mail, a message was left explaining the purpose of the survey along with a toll-free number for the respondent to call and complete the interview at their convenience. Approximately 20 of the 751 respondents in the cell phone survey completed the interview in this way.

Data collection costs (apart from overall study design, programming, and analysis costs) were slightly more than twice as high for the cell phone sample as for the landline sample. Adding in the costs of administering and paying the $10 incentive, the total costs of interviewing the cell phone sample were approximately 2.4 times the cost of the landline sample.

Cell Phone Respondents Not More Distracted

According to the interviewers working on the survey, the cell phone respondents were as focused and cooperative as those reached on a landline telephone. The vast majority (93%) of those surveyed on their cell phone demonstrated good or very good cooperation. This compares with 79% of those from the landline sample.

In addition to being cooperative, the cell phone respondents were also relatively focused on the survey task. In each sample only about 10% seemed somewhat or very distracted (8% cell phone vs. 11% landline, respectively), according to interviewers who conducted the survey. Likewise, when interviewers recorded whether it sounded as though the respondent had been doing another activity during the survey, results were quite similar for the two samples. About one-in-five of those from the cell phone sample (20%) and the landline sample (17%) were preparing a meal, watching television, shopping, exchanging comments with another person, or engaged in another activity.

Demographics of the Complete Cell and Landline Samples

People reached in the cell sample have a considerably different demographic profile from those reached in the landline sample, especially with respect to sex, race, age, education, and home ownership. On many variables, the landline sample was closer to the population parameter than the cell sample, though on some measures the cell sample picks up certain kinds of respondents that the landline samples under-represent.

A majority of those interviewed in the cell sample (55%) were men. Most landline surveys interview too few men, and require quotas or other techniques to obtain the proper proportion of men vs. women. As noted earlier, most landline surveys have too few young people in their samples (7% under age 25, vs. 13% in the population), but the cell phone sample had too many (21%). Conversely, the landline sample has too many older respondents (23% are 65 and older, vs. 16% in the population), while the cell phone sample had too few (just 8%).

The cell sample also proved to be effective at reaching African Americans, as 13% of the sample identified themselves as black. Landline samples often fall short of the population parameter (11%), though the landline sample in this project was very close (10%).

Although the survey was conducted only in English, fully 11% of the cell phone sample was Hispanic compared to just 6% of the landline frame sample. Hispanics constitute approximately 12% of the U.S. population.

Both samples include too many people with college experience, compared with the U.S. population. U.S. government figures show that 26% of the public has at least a four-year college degree, compared with 36% in the landline sample and 35% in the cell sample.

The people reached through these two samples differ in other ways as well. Over seven-in-ten (71%) of those interviewed from the landline sample report being a homeowner compared with closer to half (57%) of those reached on a cell phone. (The U.S. government estimates that 69% of the public are homeowners.)

In addition, fewer of the landline sample respondents were parents of children under 18 — a finding that likely reflects the presence of more young adults in the cell phone sample. At the same time, however, the samples were fairly similar in the percentage of respondents who were married (57% in the landline sample vs. 52% in cell sample — compared with 59% from U.S. government data), though the mix of unmarried people is very different in the two samples. One-third (33%) of the cell sample reported having never been married, compared with just 18% in the landline sample; according to the government, 25% of the adult population has never been married.