Performance-based assessment of expertise: How to decide if someone is an expert or not

Various methods for identifying experts in the absence of a gold standard are explored and one particularly promising approach (labeled CWS for Cochran–Weiss–Shanteau ) is explored in detail.

Abstract

The identification of an expert is vital to any study or application involving expertise. If external criterion (a "gold standard") exists, then identification is straightforward: Simply compare people against the standard and select whoever is closest. However, such criteria are seldom available for domains where experts work; that's why experts are needed in the first place. The purpose here is to explore various methods for identifying experts in the absence of a gold standard. One particularly promising approach (labeled CWS for Cochran–Weiss–Shanteau) is explored in detail. We illustrate CWS through reanalyses of three previous studies of experts. In each case, CWS provided new insights into identifying experts. When applied to auditors, CWS correctly detected group differences in expertise. For agricultural judges, CWS revealed subtle distinctions between subspecialties of experts. In personnel selection, CWS showed that irrelevant attributes were more informative than relevant attributes. We believe CWS provides a valuable tool for identification and evaluation of experts.