Response Rate

Response rate is the proportion of individuals who complete a survey, questionnaire, or study invitation out of the number of eligible individuals in the sample; dividing by only those who were contacted gives a different measure, the cooperation rate. It is widely reported as an indicator of survey quality, but it is only an indirect indicator of the risk of nonresponse bias, which occurs when respondents differ systematically from nonrespondents on the variables being measured.

Low response rates raise concerns about the generalizability of survey findings because the people who choose to respond may differ in important ways from those who do not. If respondents are healthier, more educated, or more interested in the topic than nonrespondents, the survey results will not accurately reflect the target population. Response rates have declined substantially across many types of surveys over the past several decades, making nonresponse bias an increasingly pressing concern. Journals and funding agencies often expect researchers to report response rates and discuss their implications for the study's validity.

In survey research, response rates are typically calculated using the Standard Definitions published by the American Association for Public Opinion Research (AAPOR; 10th edition, 2023), which provide detailed formulas accounting for different categories of nonresponse (refusals, non contacts, ineligible cases, and cases with unknown eligibility). The most conservative formula, AAPOR Response Rate 1 (RR1), divides the number of completed interviews by all eligible cases plus all cases of unknown eligibility. The other formulas count partial interviews as responses (RR2, RR4, RR6), count only an estimated share of the cases of unknown eligibility as eligible (RR3, RR4), or count none of them (RR5, RR6), so one survey can yield several different response rates, and AAPOR asks researchers to state exactly which rate they report.

Several factors influence response rates. Survey mode matters: in person interviews typically achieve the highest response rates, web surveys have averaged lower response rates than other modes in experimental comparisons, and telephone response rates have fallen steeply over recent decades, so telephone no longer reliably outperforms mail. Survey length is inversely related to response rates; shorter surveys are more likely to be completed. Respondent burden (the cognitive and time effort required) affects willingness to participate. Incentives (financial or nonfinancial) have been shown to increase response rates. Personalized invitations, advance notification letters, and follow up reminders also improve participation.

For a mean or a proportion, the nonresponse bias of the respondents' estimate equals the nonresponse rate multiplied by the difference between respondents and nonrespondents, so bias depends on how different the nonrespondents are as well as on how many there are. A meta analysis of 59 studies found only a weak relationship between nonresponse rates and nonresponse bias (Groves and Peytcheva 2008). Whatever the response rate, and especially when it is low, researchers should assess the potential for nonresponse bias by comparing the characteristics of respondents to those of the target population using available demographic data. Statistical weighting (such as post stratification weighting or propensity score weighting for nonresponse) can reduce but not eliminate bias from differential nonresponse. Sensitivity analyses examining how results might change under different assumptions about nonrespondents strengthen the interpretation.

Report response rates in any study that uses survey methodology, including cross sectional surveys, longitudinal cohort studies with questionnaire components, and clinical trials with patient reported outcome measures. Discuss the implications of the response rate for nonresponse bias. Implement strategies to maximize response rates during the study design phase, including pilot testing the survey, keeping it short, providing incentives, and planning follow up contacts. When interpreting published surveys, consider the response rate as one factor in assessing the reliability and generalizability of the findings.

A common error is reporting response rates without defining the numerator and denominator, making it impossible for readers to assess the calculation. Another mistake is assuming that a high response rate guarantees representativeness; even with 80% response, if the 20% who did not respond differ systematically, bias can occur. Conversely, dismissing a low response rate survey as invalid ignores the possibility that nonresponse is unrelated to the variables being measured, in which case the results can still be representative. Researchers sometimes fail to compare respondent characteristics to the target population, missing an opportunity to assess nonresponse bias. Finally, using only the initial contact as the recruitment strategy without follow up reminders unnecessarily reduces response rates.

A national survey of physician burnout sends questionnaires to 12,000 randomly selected physicians and receives 4,200 completed responses, yielding a 35% response rate (all 12,000 sampled physicians are kept in the denominator, as in AAPOR RR1). Comparison of respondents to the sampling frame shows that women and younger physicians are overrepresented. Post stratification weighting by age, sex, and specialty adjusts the estimates to match the national physician workforce. The weighted burnout prevalence is 44%, compared to 48% in the unweighted analysis, and the researchers note that weighting corrects only for differences in age, sex, and specialty, so bias from unmeasured differences between respondents and nonrespondents may remain.