Cross-National Surveys of Learning Achievement:\nHow Robust are the Findings?

Abstract

International surveys of learning achievement and functional literacy are increasingly common. We consider two aspects of the robustness of their results. First, we compare results from four surveys: TIMSS, PISA, PIRLS and IALS. This contrasts with the standard approach which is to analyse a single survey with no regard as to whether it agrees or not with other sources. Second, we investigate whether results are sensitive to the choice of item response model used by survey organisers to aggregate respondents' answers. In both cases we focus on countries' average scores, the within-country differences in scores, and on the association between the two. There is mixed news to report.