Do Respondents Really "Mark All That Apply" On Self-Administered Questions?

Abstract

An experiment was conducted to assess the effect of using “mark all that apply” question instructions on survey reporting as part of the field test for the Second Follow-up of the National Education Longitudinal Study of 1988 Eighth Graders (NELS:88). Mark-all-that-apply instructions were compared with instructions asking respondents to indicate “yes” or “no” to each response option on responses to three items dispersed throughout the questionnaire and consisting of different topics and numbers of response options. For the three items, significantly fewer response options were selected with the mark-all-that-apply instructions than with the yes/no instructions, but because external validity criteria were not available, overreporting to the yes/no instructions cannot be ruled out. Instructiondependent primacy effects, predicted under the hypothesis that respondents would engage in more superficial processing when given the mark-all-that-apply instructions, were not found.