Criterion Validity
Reviewed byRemya Krishnan
Definition
Criterion validity is the degree to which scores from a measurement instrument relate to an external criterion: an accepted standard measure of the same construct, or an outcome that the instrument is intended to predict.[1][3] The external criterion, often called the gold standard, provides an independent benchmark against which the new instrument's performance is judged. Criterion validity is assessed in two forms: concurrent validity, which compares the new instrument with the criterion measured at the same time point, and predictive validity, which evaluates how well the new instrument predicts future criterion scores or outcomes.[1][3] Together, these forms demonstrate that the instrument has practical utility for classification, screening, or prediction.
Explore the concept
Hold it to a yardstick
Slide the criterion from the same visit to later, swap in weaker yardsticks, and open each result to see what criterion validity measures and what it depends on.
New instrument
Five item depression screener
Quick, filled in at the visit
External criterion
SCID 5 interview
Accepted standard for diagnosing major depressive disorder
Both measured at the same visit, 450 patients
What is the yardstick?
What the comparison reports
Sensitivity
0.91
Cutoff 4 or higher
Specificity
0.83
Cutoff 4 or higher
ROC area
0.93
All cutoffs at once
Correlation
0.79
With SCID 5 symptom count
What each result means
Pick a result to see what it measures.
Concurrent validity: same visit, trusted standard
The big idea: Criterion validity compares an instrument with an external criterion, measured at the same time (concurrent) or later (predictive). It is only as strong as that criterion: it must be well validated, and a correlation with a merely related measure is construct evidence instead.
The concurrent example (five item screener, SCID 5, 450 patients, cutoff 4 or higher, sensitivity 0.91, specificity 0.83, ROC area 0.93, r = 0.79) is the article's own. The predictive example (800 women, odds ratio 3.2 with 95% CI 2.1 to 4.8, ROC area 0.79, 0.76 in a second cohort of 400) is from the Predictive Validity article in this glossary. The checklist and stress scale yardsticks are illustrative, and no results are shown for them because none were measured. The cross validation bars start at 0.5, the ROC area of chance.
Why it matters
Criterion validity is essential for justifying the use of a new or abbreviated instrument as a substitute for a more established but costlier, lengthier, or more invasive measure. In clinical settings, a validated screening questionnaire that predicts diagnoses from structured clinical interviews saves time and resources. In education, a new aptitude test that predicts later academic performance has practical value for admissions decisions. Without criterion validity evidence, there is no empirical basis for claiming that the new measure can serve the practical functions for which it is intended.
How it works
Researchers administer the new instrument and the criterion measure to the same sample of participants. For concurrent validity, both measures are collected at the same time. For predictive validity, the criterion is assessed at a later time point. The strength of the association between the two measures is quantified using correlation coefficients, sensitivity and specificity (for binary classifications), area under the receiver operating characteristic (ROC) curve, or regression coefficients.[3] High correlations or strong predictive performance support criterion validity. The choice of criterion is crucial; it must itself be well validated. When no true gold standard exists, researchers may use a "best available" criterion while acknowledging its limitations.
When to use it
Evaluate criterion validity when developing an instrument that is intended to substitute for an existing measure or to predict a future outcome. It is a standard step in the validation of clinical screening tools, diagnostic tests, educational assessments, and personnel selection instruments. Criterion validity should be assessed in the specific populations and settings where the instrument will be used, because predictive relationships may differ across contexts.
Key takeaways
- Criterion validity is essential for justifying the use of a new or abbreviated instrument as a substitute for a more established but costlier, lengthier, or more invasive measure.
- For concurrent validity, both measures are collected at the same time.
- Evaluate criterion validity when developing an instrument that is intended to substitute for an existing measure or to predict a future outcome.
- The most fundamental error is using a criterion that itself has questionable validity, which limits the meaningfulness of the comparison.
Common mistakes
- The most fundamental error is using a criterion that itself has questionable validity, which limits the meaningfulness of the comparison.
- Another mistake is conflating criterion validity with construct validity; criterion validity compares the instrument to a specific external standard, while construct validity evaluates whether the instrument measures the intended theoretical concept through multiple forms of evidence. Sample restriction, studying only a narrow range of the construct, can attenuate correlations and understate criterion validity.
- Researchers sometimes report area under the ROC curve without also reporting sensitivity and specificity at specific cutoffs, which provides limited actionable information for instrument users.
- Finally, failing to cross validate predictive models in independent samples risks overstating predictive accuracy.
Example
Researchers validate a brief five item depression screener against the Structured Clinical Interview for DSM 5 (SCID 5) diagnosis of major depressive disorder. In a primary care sample of 450 patients, the screener is administered at the same visit as the SCID 5 interview. A score of 4 or higher on the screener correctly identifies 91% of patients diagnosed with depression by the SCID 5 (sensitivity = 0.91) and correctly classifies 83% of nondepressed patients (specificity = 0.83). The area under the ROC curve is 0.93, and the Pearson correlation between screener scores and the number of depressive symptoms rated present in the SCID 5 interview is 0.79. These results support strong concurrent criterion validity and suggest the screener can serve as a practical first stage tool in busy clinical settings.
Test your understanding
5 questions that adapt as you go: answer two in a row correctly and they get harder.
Question 1 of 5
Scenario:
In the article's example, a five item depression screener with a cutoff of 4 has sensitivity of 0.91 and specificity of 0.83 against SCID 5 diagnoses.
Question:
What does the specificity of 0.83 mean here?
Hardness: Interpret
Sources
- [1]DeVellis RF, Thorpe CT. Scale Development: Theory and Applications. 5th ed. SAGE Publications, 2022. ISBN 9781544379340
- [2]Field A. Discovering Statistics Using IBM SPSS Statistics. 6th ed. SAGE Publications, 2024. ISBN 9781529630008
- [3]Streiner DL, Norman GR, Cairney J. Health Measurement Scales: A Practical Guide to Their Development and Use. 5th ed. Oxford University Press, 2015. ISBN 9780199685219
Cite this term
Paperguide. (2026, October 14). Criterion Validity. In Paperguide Research Glossary. https://paperguide.ai/research-glossary/criterion-validity/(Paperguide, 2026)