Criterion Validity

Criterion validity is the degree to which scores from a measurement instrument relate to an external criterion: an accepted standard measure of the same construct, or an outcome that the instrument is intended to predict. The external criterion, often called the gold standard, provides an independent benchmark against which the new instrument's performance is judged. Criterion validity is assessed in two forms: concurrent validity, which compares the new instrument with the criterion measured at the same time point, and predictive validity, which evaluates how well the new instrument predicts future criterion scores or outcomes. Together, these forms demonstrate that the instrument has practical utility for classification, screening, or prediction.

Criterion validity is essential for justifying the use of a new or abbreviated instrument as a substitute for a more established but costlier, lengthier, or more invasive measure. In clinical settings, a validated screening questionnaire that predicts diagnoses from structured clinical interviews saves time and resources. In education, a new aptitude test that predicts later academic performance has practical value for admissions decisions. Without criterion validity evidence, there is no empirical basis for claiming that the new measure can serve the practical functions for which it is intended.

Researchers administer the new instrument and the criterion measure to the same sample of participants. For concurrent validity, both measures are collected at the same time. For predictive validity, the criterion is assessed at a later time point. The strength of the association between the two measures is quantified using correlation coefficients, sensitivity and specificity (for binary classifications), area under the receiver operating characteristic (ROC) curve, or regression coefficients. High correlations or strong predictive performance support criterion validity. The choice of criterion is crucial; it must itself be well validated. When no true gold standard exists, researchers may use a "best available" criterion while acknowledging its limitations.

Evaluate criterion validity when developing an instrument that is intended to substitute for an existing measure or to predict a future outcome. It is a standard step in the validation of clinical screening tools, diagnostic tests, educational assessments, and personnel selection instruments. Criterion validity should be assessed in the specific populations and settings where the instrument will be used, because predictive relationships may differ across contexts.

The most fundamental error is using a criterion that itself has questionable validity, which limits the meaningfulness of the comparison. Another mistake is conflating criterion validity with construct validity; criterion validity compares the instrument to a specific external standard, while construct validity evaluates whether the instrument measures the intended theoretical concept through multiple forms of evidence. Sample restriction, studying only a narrow range of the construct, can attenuate correlations and understate criterion validity. Researchers sometimes report area under the ROC curve without also reporting sensitivity and specificity at specific cutoffs, which provides limited actionable information for instrument users. Finally, failing to cross validate predictive models in independent samples risks overstating predictive accuracy.

Researchers validate a brief five item depression screener against the Structured Clinical Interview for DSM 5 (SCID 5) diagnosis of major depressive disorder. In a primary care sample of 450 patients, the screener is administered at the same visit as the SCID 5 interview. A score of 4 or higher on the screener correctly identifies 91% of patients diagnosed with depression by the SCID 5 (sensitivity = 0.91) and correctly classifies 83% of nondepressed patients (specificity = 0.83). The area under the ROC curve is 0.93, and the Pearson correlation between screener scores and the number of depressive symptoms rated present in the SCID 5 interview is 0.79. These results support strong concurrent criterion validity and suggest the screener can serve as a practical first stage tool in busy clinical settings.