Reliability

Reliability is the degree to which a measurement instrument yields consistent, stable, and reproducible results when administered under the same conditions. It reflects the extent to which observed scores are free from random measurement error. The major types of reliability include test retest reliability (stability over time), inter rater reliability (consistency across different raters), internal consistency (coherence among items within a scale), and parallel forms reliability (equivalence of alternative versions of an instrument).

Reliability is a prerequisite for validity: an instrument that produces inconsistent measurements cannot accurately measure the construct it targets. In clinical research, unreliable measures introduce noise that shrinks standardized effect sizes, requires larger sample sizes, and can lead to erroneous conclusions. In systematic reviews, the reliability of outcome measures affects the interpretability of pooled estimates. Researchers selecting instruments for their studies should prioritize those with established reliability evidence in the relevant population. Failing to account for reliability can also bias findings: random measurement error attenuates correlations, and the regression slopes of error-prone predictors, toward zero, so real associations can look weaker than they are.

Test retest reliability is assessed by administering the same instrument to the same group of individuals on two occasions separated by a time interval and calculating the agreement between the two sets of scores, preferably with the intraclass correlation coefficient (ICC), because Pearson's r ignores systematic shifts between occasions. A high test retest ICC indicates that the instrument produces stable measurements over time. The time interval must be long enough to minimize memory effects but short enough that the true value of the construct has not changed.

Inter rater reliability measures the agreement between two or more independent raters assessing the same phenomenon. Cohen's kappa is used for two raters' categorical ratings and adjusts for chance agreement. The ICC is used for continuous ratings and measures either absolute agreement or consistency, depending on the form chosen. Koo and Li (2016) read ICC values from 0.75 to 0.90 as good and above 0.90 as excellent, applying these rules of thumb to the 95% confidence interval.

Internal consistency assesses how strongly the items within a multi item scale, which are meant to measure the same construct, vary together. Cronbach's alpha is among the most widely reported measures, with values of 0.70 or higher typically considered acceptable for research purposes, while values of 0.90 or higher are often recommended when scores inform decisions about individual patients. McDonald's omega is increasingly recommended as a more accurate alternative. Low internal consistency can mean that the items do not cohere, but alpha also depends on the number of items, and a high alpha does not show that the items measure a single construct.

Parallel forms reliability is assessed by giving the same people two versions of an instrument built to be equivalent, such as two forms of a test, and estimating the agreement between their scores.

Reliability is usually expressed as a coefficient with a maximum of 1, where values closer to 1 indicate higher reliability; sample estimates of kappa, alpha and the ICC can fall below 0. The relationship between reliability and measurement error is captured by the standard error of measurement (SEM), which quantifies the expected variability in individual scores due to random error. A reliable instrument with a small SEM produces precise individual level measurements.

Assess and report the reliability of measurement instruments in any study that uses them for data collection. Select instruments with established reliability evidence for the population being studied. If using a new or adapted instrument, conduct a reliability assessment as part of the study. The type of reliability to assess depends on the instrument and study design: test retest for instruments measuring stable constructs, inter rater for observational or diagnostic assessments, and internal consistency for multi item scales.

A common error is assuming that reliability demonstrated in one population automatically applies to another. An instrument reliable in young adults may have different reliability properties in elderly populations or different cultural groups. Another mistake is reporting only Cronbach's alpha as evidence of reliability, ignoring other relevant forms (such as inter rater reliability for subjective assessments). Researchers sometimes confuse reliability with validity, assuming that a reliable instrument is automatically valid; an instrument can produce highly consistent results that are consistently wrong. Finally, accepting low reliability without considering its impact on statistical power and effect size estimation can lead to underpowered studies and attenuated results.

A team adapts a validated depression screening questionnaire for use in a new cultural context. They administer the adapted version to 200 participants on two occasions two weeks apart, obtaining a test retest ICC of 0.84. Internal consistency is assessed with Cronbach's alpha = 0.89 across the 10 items. Because the adapted version is administered as a structured interview, two trained raters independently score 50 recorded interviews, achieving an ICC (two way random effects, absolute agreement, single rater) of 0.92 for the total score. These reliability estimates support the use of the adapted instrument in subsequent research in this population.