Academic journal article Demographic Research

Do Low Survey Response Rates Bias Results? Evidence from Japan

Academic journal article Demographic Research

Do Low Survey Response Rates Bias Results? Evidence from Japan

Article excerpt

1. Introduction

Sample surveys (cross-sectional and longitudinal) have become the dominant data source used by population researchers. Response rates, both the initial response rate and attrition rates in longitudinal studies, have historically been an important rough-andready yardstick to judge data quality. Response rates6 have been declining in urbanized, high-income countries to the point where many cross-sectional surveys now have response rates below 50% (Atrostic et al. 2001; Brick and Williams 2013; de Leeuw and de Heer 2002; Groves 2011; Singer 2006).

The survey literature has long recognized that low response rates only indicate potential bias (e.g., Lessler and Kalsbeck 1992), yet the almost automatic response among most in the population field has been to equate low response rates with poor data quality. Low response rates produce bias only to the extent that there are differences between responders and non-responders on the estimate(s) of interest, and then only if such differences cannot be eliminated or controlled for through the use of observable and available characteristics of responders and non-responders. The difficulty in evaluating non-response bias is that measures of the variable(s) of interest and characteristics of non-responders are generally not observed, and hence the population field has tended to rely on the presumption that low response rates necessarily mean low data quality.

In this paper, using surveys designed by demographers and variables that have often been used as dependent variables in demographic research, we examine this common presumption of demographers. We also place our results in the context of the survey research literature in which there are numerous indications that low response rates need not mean the results are biased and, if there is bias within the survey, it varies considerably from variable to variable. It has been unfortunate that the research literature on response rates and bias has tended to be published in journals aimed at survey research experts. These are journals that tend not to be read by population researchers and hence the persistent belief that low response rates equate to low-quality, biased data.

We examine data quality issues across a range of behavioral, knowledge, and attitudinal variables for two data collection efforts conducted in Japan in 2009, one a follow-up of a cross-sectional survey conducted in 2000 and one a new cross section. Both 2009 efforts were conducted by the same data gathering organization, using the same procedures and the same questionnaire. Both had modest response rates: 53% of those surveyed in 2000 for the longitudinal follow-up and 54% of those sampled from Japan's basic residence registration for the new cross section. As is shown below, response rates in this range are now typical in Japan and other industrialized countries. To preview our results, we find evidence that low response rates likely bias simple univariate distributions of some behaviors, knowledge, and attitudinal items, but low response rates seem not to bias estimates of the relationship between various independent variables and these behavioral, knowledge, and attitudinal items.

2. Decline in respondent cooperation

Declines in respondent cooperation in developed countries have been widely reported in the survey research literature (e.g., Groves 2011), including government-conducted surveys (e.g., Atrostic et al. 2001; Bethlehem, Cobben, and Schouten 2011; Brick and Williams 2013; de Leeuw and de Heer 2002), and such declines have been going on for many years (Steeh 1981). These declines are easiest to see in cross-sectional surveys that have been repeated over a long period. Consider the University of Michigan's Survey of Consumer Attitudes (SCA). In the mid-1950s, response rates were close to 90% (Steeh 1981); by 2003, the response rate was below 50% (Curtin, Presser, and Singer 2005). In Japan, the response rate of Mainichi Shimbun's National Opinion Survey on Family Planning declined from 92% in 1950 to 61% in 2004 (Robert Retherford and Naohiro Ogawa, personal communication). …

Search by... Author
Show... All Results Primary Sources Peer-reviewed

Oops!

An unknown error has occurred. Please click the button below to reload the page. If the problem persists, please try again in a little while.