Test Retest Reliability

Test retest reliability is a measure of the consistency of a measurement instrument over time. It is assessed by administering the same test, questionnaire, or measurement tool to the same group of respondents on two separate occasions and then calculating the correlation or agreement between the two sets of scores. A high test retest reliability coefficient indicates that the instrument produces stable and reproducible results when the underlying construct being measured has not changed, providing evidence that the observed scores reflect true differences between individuals rather than random measurement error.

The stability of a measurement instrument over time is fundamental to its usefulness in both research and clinical practice. If a depression scale gives substantially different scores when administered to the same person two weeks apart, and the person's depression has not changed, the scale cannot be trusted to detect genuine changes in clinical status or to compare groups in a research study. Test retest reliability provides essential evidence for the trustworthiness of outcome measures used in longitudinal studies, clinical monitoring, and program evaluation. Without it, researchers cannot distinguish real change from measurement noise, undermining the validity of any conclusions about treatment effects or disease progression.

Researchers recruit a sample of participants and administer the instrument at two time points separated by an interval chosen to be long enough that participants are unlikely to remember their previous responses but short enough that the construct being measured is unlikely to have truly changed. For most psychological and health measures, intervals of one to four weeks are common. The agreement between the two sets of scores is typically quantified with the intraclass correlation coefficient (ICC) for continuous measures or Cohen's kappa for categorical measures. Pearson's correlation coefficient is sometimes reported but is less appropriate because it does not account for systematic differences between the two administrations: if every participant scores the same number of points higher at retest, r does not change, while an ICC for absolute agreement falls. A widely used rule of thumb from Koo and Li (2016), which they apply to the 95% confidence interval of the ICC, reads values below 0.50 as poor, 0.50 to 0.75 as moderate, 0.75 to 0.90 as good and above 0.90 as excellent, though acceptable thresholds depend on the intended use of the instrument. When interpreting test retest reliability, researchers must consider whether the construct is expected to be stable over the chosen interval, as traits (such as personality) should show higher stability than states (such as mood).

Assess test retest reliability when developing or validating a new measurement instrument, when adapting an existing instrument for a new population or language, or when evaluating whether an instrument is suitable for use in a longitudinal study that requires detecting change over time. Test retest reliability is most informative for instruments intended to measure stable constructs and is less appropriate for constructs that fluctuate naturally, such as acute pain or momentary affect, unless the measurement interval is very short.

A frequent error is choosing an inappropriate time interval. Too short an interval inflates reliability through memory effects, while too long an interval allows genuine change in the construct, deflating reliability estimates. Another mistake is using Pearson's r instead of the ICC, which can yield misleadingly high values even when systematic bias exists between time points. Failing to report the specific ICC model and form (for example, two way mixed effects, single measures, absolute agreement) makes it impossible for readers to evaluate the result. Researchers also sometimes test retest reliability in a convenience sample that does not represent the target population, limiting the generalizability of the reliability estimate.

Researchers developing a new 20 item patient reported outcome measure for chronic fatigue administer the questionnaire to 120 patients at baseline and again two weeks later, selecting a period during which no changes in treatment are expected. The ICC (two way mixed effects, single measures, absolute agreement) between the two administrations is 0.88 (95% CI: 0.83 to 0.92), indicating good to excellent test retest reliability by Koo and Li's guideline, which reads the whole interval. The standard error of measurement (SEM) is calculated as 3.2 points on the 0 to 100 scale, and the minimal detectable change at the 95% confidence level is 8.9 points. These values allow clinicians to determine that a change of 9 or more points likely represents a real change rather than measurement error.