Psychometrics is Datametrics: the Test is not Reliable
Educational and Psychological MeasurementPublished 1 April 2000
Bruce Thompson, Tammi Vacha‐Haase
Citations342
SJR quartileQ1
SJR score1.93
SNIP2.21
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
Abstract
The present article responds to selected criticisms of some EPM editorial policies and Vacha-Haase’s “reliability generalization” meta-analytic methods. However, the treatment is more broadly a manifesto regarding the nature of score reliability and what are reasonable expectations for psychometric reporting practices in substantive inquiries. The consequences of misunderstandings of score reliability are explored. It is suggested that paradigmatic misconceptions regarding psychometric issues feed into a spiral of presumptions that measurement training is unnecessary for doctoral students, which then in turn further reinforces misunderstandings of score integrity issues.
Keywords
Decision SciencesEconomics, Econometrics and Finance
American Educational Research JournalStatistical Theories of Mental Test Scores
8,138 Citations1969William W. Rozeboom, Frederic M. Lord +2 more
Journal of the American Statistical AssociationMeasurement, Design, and Analysis: An Integrated Approach.
4,066 Citations1992Sandra G. Funk, Elazar J. Pedhazur +1 more
Occupational Therapy In Health CarePublication Manual of the American Psychological Association
3,198 Citations2011Kate E. DeCleene, Jennifer Fogo
American PsychologistStatistical methods in psychology journals: Guidelines and explanations.
2,773 Citations1999Leland Wilkinson, Task Force on Statistical Inference
Journal of Verbal Learning and Verbal BehaviorThe language-as-fixed-effect fallacy: A critique of language statistics in psychological research
2,263 Citations1973Herbert H. Clark
Test Theory: A Unified Treatment
1,747 Citations1999Roderick P. McDonald
Psychological Testing
1,519 Citations1987Peter W. Zinkus
Educational ResearcherEssentials of Educational Measurement
1,493 Citations1980Alfred D. Garvin, Robert L. Ebel
Measurement and Assessment in Teaching
1,289 Citations1994Robert L. Linn, Norman E. Gronlund
Journal of the American Statistical AssociationMeasurement and Evaluation in Psychology and Education.
1,236 Citations1961R. S., Robert L. Thorndike +1 more
Harvard Educational ReviewThe Case Against Statistical Significance Testing
933 Citations1978Ronald P. Carver
International Statistical ReviewMeasurement and Evaluation in Education and Psychology
494 Citations1979H. Breny, William A. Mehrens
Educational and Psychological MeasurementReliability Generalization: Exploring Variance in Measurement Error Affecting Score Reliability Across Studies
446 Citations1998Tammi Vacha‐Haase
Principles of Educational and Psychological Measurement and Evaluation
349 Citations2019Gilbert Sax
The American StatisticianWhy Permutation Tests are Superior to <i>t</i> and <i>F</i> Tests in Biomedical Research
338 Citations1998John Ludbrook, Hugh Dudley
The Journal of Experimental EducationThe Case Against Statistical Significance Testing, Revisited
323 Citations1993Ronald P. Carver
The Journal of Experimental EducationEvaluating Results Using Corrected and Uncorrected Effect Size Estimates
265 Citations1993Patricia Snyder, Stephen Lawson
American PsychologistGraduate training in statistics, methodology, and measurement in psychology: A survey of PhD programs in North America.
238 Citations1990Leona S. Aiken, Stephen G. West +6 more
Educational and Psychological Measurement
235 Citations2018W. Holmes Finch, Brian F. French
Evaluation PracticeThe program evaluation standards: How to assess evaluations of educational programs
198 Citations1994Michael Quinn Patton
Medical Entomology and ZoologyThe Program Evaluation Standards: How to Assess Evaluations of Educational Programs
164 Citations1994James R. Sanders
Introduction Applying the Standards The Standards Utility Standards Feasibility Standards Propriety Standards Accuracy Standards are applied.
The Journal of Philosophy Psychology and Scientific MethodsAn Introduction to the Theory of Mental and Social Measurements.
133 Citations1905Franklin H. Giddings, Edward L. Thorndike
Educational ResearcherResearch Techniques in <i>AERJ</i> Articles: 1969 to 1978
103 Citations1980Victor L. Willson
The Journal of Experimental EducationPractices Regarding Reporting of Reliability Coefficients: A Review of Three Journals
97 Citations1999Tammi Vacha‐Haase, Carin M. Ness +2 more
Journal of Counseling & DevelopmentTwo and One‐Half Decades of Leadership in Measurement and Evaluation
93 Citations1992Bruce Thompson
Journal of Counseling PsychologyTrends in reporting psychometric properties of scales used in counseling psychology research.
78 Citations1990Scott T. Meier, Susan R. Davis
Journal of Counseling & DevelopmentStatistical Significance and Reliability Analyses in Recent <i>Journal of Counseling & Development</i> Research Articles
77 Citations1998Bruce Thompson, Patricia Snyder
AssessmentTest-Retest Reliability of Psychological and Neurobehavioral Tests Self-Administered by Computer
74 Citations1999Keith A. Campbell, Diane S. Rohlman +6 more
The results suggest that individual test reliability is not affected when tests are administered as part of an extensive multi-measure battery and compared to psychological test reliability, neurobehavioral test reliability appeared more vulnerable to decreases with mixed format testing.
Educational and Psychological MeasurementPsychometrics versus Datametrics: Comment on Vacha-Haase’s “Reliability Generalization” Method and Some Epm Editorial Policies
69 Citations2000Shlomo S. Sawilowsky
Measurement and Evaluation in Counseling and DevelopmentUsing Generalizability Theory in Counseling and Development
56 Citations1988Noreen M. Webb, Glenn Rowley +1 more
ETS Research Bulletin SeriesDO TESTS OF THE SAME LENGTH HAVE THE SAME STANDARD ERROR OF MEASUREMENT?
50 Citations1956Frederic M. Lord
Measurement and assessment in the schools
37 Citations1999Blaine R. Worthen
Early Education and DevelopmentMothers' and Fathers' Perceptions of Family Functioning in Families Having Children With Disabilities
16 Citations1994Susan L. Crowley, Matthew Taylor
Journal of Psychopathology and Behavioral AssessmentPsychometric instability of measures in novel settings: Use of psychiatric rating scales in nursing homes
9 Citations1981William R. Shadish, Richard R. Bootzin +2 more
Methods for assessing changes in both reliability and validity of a measure that is used in a novel setting, along with suggestions for how to deal with these problems are presented.
ETS Research Bulletin SeriesNOTE ON “TESTS OF THE SAME LENGTH DO HAVE THE SAME STANDARD ERROR OF MEASUREMENT”1
6 Citations1958Frances Swineford
Applied Measurement in EducationVariability in Reliability Coefficients and the Standard Error of Measurement From School District to District
6 Citations1999Leonard S. Feldt, Audrey L. Qualls