Estimates of Potential Bias in Telephone Substance Abuse Surveys Due to Exclusion of Households without Telephones
McAuliffe, William E., LaBrie, Richard, Woodworth, Ryan, Zhang, Caroline, Journal of Drug Issues
This article reports estimates of the bias that may result in telephone substance abuse surveys because of the noncoverage of households without telephones. The study analyzed 1995-1998 data from the face-to-face National Household Survey on Drug Abuse. Residents of households with telephones reported less drug use, less dependence on drugs and alcohol, but more alcohol use than residents of households without telephones. The resulting percentage differences between respondents with telephones and respondents in all households averaged one tenth of a percent (0.1%), and ratios of percentage estimates for all households to households with telephones averaged 1.04. The bias varied by substance, time frame, use versus dependence, and demographic characteristics. For example, use of marijuana ever had less bias than past year cocaine dependence. American Indians/Alaska Natives had the greatest amount of potential bias. For most populations and policy objectives, it may not be worth the added expense of conducting supplementary face-to-face interviews with residents of households without telephones in order to eliminate the bias.
This study focused on the amount of potential noncoverage bias in telephone survey estimates of the alcohol and drug use and dependence that may result because telephone surveys fail to include households without telephones. Of the populations missed by a telephone survey, the largest group consists of residents of households without telephones (McAuliffe, Geller, LaBrie, Paletz, & Fournier, 1998). If the substance use disorder rate of residents of households without telephones differs from the rate of residents of all households, failing to adjust for this difference would result in bias if a telephone survey's estimates were generalized to the entire household population. This potential shortcoming of telephone surveys is of course widely recognized by scientists and lay persons, but few researchers can afford to eliminate it by conducting a face-to-face survey or a multi-mode survey that includes face-to-face interviews of households without telephones (McAuliffe et al.). To address the issue, authors of publications of the results of telephone substance abuse surveys routinely caution readers that there is a potential for bias due to noncoverage of households without telephones.
These caveats provide readers with little guidance to interpret a survey's results. The amount of bias depends upon the proportion of households without telephones and the size of the difference between rates for households with and without telephones. In the last decade, the national rate of households without telephones was between 5% and 6%. The most recent national estimate was 5.6% in 2000 (Belinfante, 2001). The median state rate was 5% in 1990 and 5.4% in 2000 (Bureau of the Census, 1994; Belinfante). In the most recent figures, the range for individual states was narrow: only two states had rates in excess of 10% (Mississippi 10.2% and Arkansas 11.4%) (Belinfante).
Because the telephone subscribership is so large in most places, the size of the difference between households with and without telephones would have to be substantial to cause a meaningful amount of bias for most purposes. For example, if a state's telephone noncoverage rate were 5% and the substance use disorder rate for the residents of households with telephones were 10%, the rate for residents of households without telephones would have to be 30%, or three times greater, in order to increase the estimated rate for residents of all households by one percent, that is, to 11%. If the difference in the substance abuse rate between households with and without telephones were much smaller than 30%, even though the difference was statistically significant, the amount of bias may be no more than a few tenths of a percent. Removing bias of this magnitude may not be important enough for many substantive questions to justify increasing the survey cost significantly. …