Improving the Validity of Letters of Recommendation: An Investigation of Three Standardized Reference Forms

Abstract

Although letters of recommendation are (LORs) widely used, little research has examined how accurately they predict job performance. The few existing studies have yielded mixed results, and meta-analytic estimates of validity range from .14 to .27 (Hunter & Hunter, 1984 Hunter, J. E., & Hunter, R. F. (1984). Validity and utility of alternative predictors of job performance. Psychological Bulletin, 96, 72–98.[Crossref], [Web of Science ®] , [Google Scholar]; Reilly & Chao, 1982 Reilly, R. R., & Chao, G. T. (1982). Validity and fairness of some alternate employee selection procedures. Personnel Psychology, 35, 1–62.[Crossref], [Web of Science ®] , [Google Scholar]). This investigation was designed to improve predictive validity by developing a standardized reference form and evaluating 3 different rating formats: Multi-Item scales, Relative Percentile Method (RPM) scales, and Global Trait Rankings. A total of 520 individuals applied to the Canadian military, and 544 LORs were obtained. Complete predictor and criterion data were available for 57 participants. Regression analyses indicated that the validity of the RPM rating format (R2(adj) = .18; R(adj) = .42) was substantially higher than previous estimates of LOR validity. The 2 remaining methods produced nonsignificant results. Limitations of the study, suggestions for future research, and implications for the field are discussed.