Expert judgment about uncertainty: Bayesian decision making in realistic settings
Organizational Behavior and Human PerformancePublished 1 August 1975
Barbara Beach
Citations62
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Questions raised serve as a framework for reviewing the literature on experts' assessments of uncertainty in four separate areas of applied research, the military, meterology, medicine, and business.
Abstract
This article reviews Bayesian research on experts' judgments about uncertainty in the real world. In the first part of the article, six central questions are raised concerning (1) the economic value of probability statements, (2) the use of actuarial versus subjective probabilities, (3) individual versus consensus probability assessments, (4) the role of computers in probability assessment (5) the use of data by assessors, and (6) expertise and the training of individuals to become experts, and laboratory evidence relevant to these questions is discussed. In the second part of the article these questions serve as a framework for reviewing the literature on experts' assessments of uncertainty in four separate areas of applied research, the military, meterology, medicine, and business.
Keywords
Decision Sciences
Cognitive PsychologyAvailability: A heuristic for judging frequency and probability
9,943 Citations1973Amos Tversky, Daniel Kahneman
Psychological ReviewOn the psychology of prediction.
6,171 Citations1973Daniel Kahneman, Amos Tversky
Cognitive PsychologySubjective probability: A judgment of representativeness
4,871 Citations1972Daniel Kahneman, Amos Tversky
Psychological BulletinBelief in the law of small numbers.
3,229 Citations1971Amos Tversky, Daniel Kahneman
Organizational Behavior and Human PerformanceComparison of Bayesian and regression approaches to the study of information processing in judgment
1,841 Citations1971Paul Slovic, Sarah Lichtenstein
This work examines the models that have been developed for describing and prescribing the use of information in decision making, the major experimental paradigms, and the major empirical results and conclusions of these two approaches.
Psychological ReviewBayesian statistical inference for psychological research.
1,746 Citations1963Ward Edwards, Harold R. Lindman +1 more
The likelihood principle emphasized in Bayesian statistics implies that the rules governing when data collection stops are irrelevant to data interpretation, and it is entirely appropriate to collect data until a point has been proven or disproven.
Cambridge University Press eBooksConservatism in human information processing
1,040 Citations1982Ward Edwards
A number of experiments show that a major cause of conservatism is human misaggregation of the data, which means men perceive each datum accurately and are well aware of its individual diagnostic meaning, but are unable to combine its diagnostic meaning well with the diagnostic meaning of other data when revising their opinions.
Journal of Experimental PsychologyConservatism in a simple probability inference task.
852 Citations1966Lawrence D. Phillips, Ward Edwards
Estimates were most nearly Bayesian under the (formally inappropriate) linear payoff, but considerable overestimation resulted; the log payoff condition yielded less conservatism than the quadratic payoff and when Ss estimated odds on a logarithmic scale.
Organizational Behavior and Human PerformanceA comparative study of differences in subjective likelihood estimates made by individuals, interacting groups, Delphi groups, and nominal groups
385 Citations1973David H. Gustafson, Ramesh K. Shukla +2 more
Management ScienceThe Consensus of Subjective Probability Distributions
375 Citations1968Robert L. Winkler
Journal of applied meteorology“Good” Probability Assessors
331 Citations1968Robert L. Winkler, Allan H. Murphy
Journal of the American Statistical AssociationScoring Rules and the Evaluation of Probability Assessors
261 Citations1969Robert L. Winkler
Organizational Behavior and Human PerformanceProbabilistic forecasting: An experiment related to the stock market
250 Citations1972Carl-Axel S. Staël von Holstein
An attempt to tie an experiment to a “practical” situation involving the development of buying prices on the Stock Exchange and the results confirm that people can quantify their beliefs reasonably well in probabilistic terms.
The American Journal of MedicineDecision analysis and clinical judgment
247 Citations1973William B. Schwartz, G. Anthony Gorry +2 more
It is suggested that the theory and technics of decision analysis provide new and useful strategies appropriate for dealing with complex clinical situations and, when applied quantitatively, the formalism affords greater precision than is otherwise readily attainable.
Journal of applied meteorologyOn Subjective Probability Forecasting
217 Citations1963Frederick Sanders
Journal of the Operational Research SocietyA Bayesian Approach to Short-term Forecasting
213 Citations1971P. J. Harrison, C. F. Stevens
A new approach to short-term forecasting is described, based on Bayesian principles, which produces not only single-figure forecasts but distributions of trend and slope values which are relevant to subsequent decisions based on forecasts.
Acta PsychologicaThe utilization of subjective probabilities in production planning
196 Citations1970John Kidd
This paper describes how the subjective probabilities of overhaul completion gleaned from workers in each power station, are combined in the Midlands Region Information Centre, resulting in a prediction of the expected capabilities of the level of power generation over a possible horizon of 30 weeks.
IRE transactions on human factors in electronicsMan-Computer Cooperation in Decisions Requiring Common Sense
179 Citations1961Douwe B. Yntema, Warren S. Torgerson
The problem of enabling a man to convey his decision rules to a machine will in many cases prove less formidable than it might at first appear.
The Journal of BusinessAnalyzing the Use of Information in Investment Decision Making: A Methodological Proposal
167 Citations1972Paul Slovic, Dan Fleissner +1 more
Organizational Behavior and Human PerformanceExperiments in the laboratory and the real world
167 Citations1973Robert L. Winkler, Allan H. Murphy
In this paper conservatism is considered in some detail within the context of the generalizability question, and some of the difficulties inherent in experimentation in realistic settings are discussed, and possible procedures for avoiding or at least alleviating such difficulties are suggested.
Acta PsychologicaScoring rules in probability assessment and evaluation
160 Citations1970Allan H. Murphy, Robert L. Winkler
The American Journal of MedicineDecision analysis as the basis for computer-aided management of acute renal failure
151 Citations1973G. Anthony Gorry, Jerome P. Kassirer +2 more
It is concluded that computer-aided management with the aid of decision analysis is a promising area for further investigation.
IEEE Transactions on Systems Science and CyberneticsProbabilistic Information Processing Systems: Design and Evaluation
148 Citations1968Ward Edwards, Lawrence D. Phillips +2 more
A Probabilistic Information Processing System uses men and machines in a novel way to perform diagnostic information processing that circumvents human conservatism in information processing and fragments the job of evaluating diagnostic information into small separable tasks.
Journal of the American Statistical AssociationProbabilistic Prediction: Some Experimental Results
140 Citations1971Robert L. Winkler
The results reported here show that methods such as scoring rules and bets are useful in leading individuals to make careful probability assessments, with predictions determined by mechanical schemes and by the organized betting market proving superior to those of many of the subjects.
Holt, Rinehart and Winston eBooksEmerging technologies for making decisions
122 Citations1965W. Edwards, Harold R. Lindman +1 more
Medical Entomology and ZoologyAssessment and evaluation of subjective probability distributions
110 Citations1970Staël von Holstein, S Carl-Axel
Journal of Experimental PsychologyResponse bias explanation of conservative human inference.
98 Citations1970Wesley M. Ducharme
JAMAConditional Probability Program for Diagnosis of Thyroid Function
97 Citations1963John E. Overall
The frequencies of occurrence of certain clinical findings were determined and the computer program gave the correct classification (ie, successfully predicted the therapeutic result) in 258 cases, approaching the level of agreement to be expected between an expert consultant and a well-qualified diagnosing physician.
Monthly Weather ReviewTHE ECONOMIC UTILITY OF WEATHER FORECASTS
85 Citations1955J. C. Thompson, Glenn W. Brier
Acta PsychologicaMeasurement of subjective probability
74 Citations1970Carl-Axel S. Staël von Holstein
The paper discusses various properties that scoring rules should possess in order to encourage honesty and includes a survey of assessment techniques which do not rely on scoring rules.
New England Journal of MedicineComputers in Patient Care
71 Citations1968G. Octo Barnett
In most areas of medical care, adaptations in practice have kept pace with advances in knowledge about diseases and therapeutic procedures, however, one vital area of medical practice, information ...
Monthly Weather ReviewTHE RANKED PROBABILITY SCORE AND THE PROBABILITY SCORE: A COMPARISON<sup>1,,</sup><sup>2</sup>
67 Citations1970Allan H. Murphy
Annals of Internal MedicineSymptom Diagnosis. A Mathematical Analysis of Epigastric Pain
56 Citations1963Joseph A. Rinaldo, PERRY SCHEINOK +1 more
Excerpt The process of choosing a diagnosis from subjective symptoms has not been examined in detail and the mechanism behind this mechanism is still unclear.
IEEE Transactions on Human Factors in ElectronicsAccuracy and Consistency in the Revision of Subjective Probabilities
54 Citations1966Lee Roy Beach
Management ScienceAutomating Judgmental Decision Making for a Serious Medical Problem
51 Citations1971Norman E. Betaque, G. Anthony Gorry
A decision theory model of the diagnosis and treatment of a serious medical condition, acute renal failure, has been implemented and its success in duplicating the decisions of expert clinicians in about 90 per cent of the cases used indicates that the method of analysis is potentially a practical way of analyzing such decision problems.
Acta PsychologicaThe assessment of subjective probability distributions: A training experiment
48 Citations1973Ralf E. Schaefer, Katrin Borcherding
In this experiment 22 Ss especially trained to assess subjective probability distributions for proportions concerning student characteristics received feedback about their performance in the previous session(s), and learned to assess more realistic distributions.
Organizational Behavior and Human PerformanceIntuitive statistical inferences about diffuse hypotheses
46 Citations1968Cameron R. Peterson, Richard G. Swensson
Journal of the American Statistical AssociationScoring Rules and the Evaluation of Probability Assessors
46 Citations1969Robert L. Winkler
Organizational Behavior and Human PerformanceEvaluation of probabilistic information processing in medical decision making
44 Citations1969David H. Gustafson
A Bayesian probabilistic information processing system (PIP) was compared, in field and laboratory tests, with four other forecasting methodologics and performed at least as well as any other technique and was superior to all techniques in terms of frequency of error.
Organizational Behavior and Human PerformanceStrategies of information seeking in deferred decision making
43 Citations1969Gordon F. Pitz, Helen Reinhold +1 more
Bulletin of the American Meteorological SocietyForecasters and Probability Forecasts: Some Current Problems
41 Citations1971Allan H. Murphy, Robert L. Winkler
Journal of Financial and Quantitative AnalysisBayesian Models for Forecasting Future Security Prices
41 Citations1973Robert L. Winkler
Journal of applied meteorologyAn Experiment in Probabilistic Weather Forecasting
39 Citations1971Carl-Axel S. Staël von Holstein
Probabilistic forecasts were given for temperature and precipitation for 2 and 5-day periods and the scores were given as feedback to the participants together with the actual outcomes, indicating some explanations for this.
Bulletin of the American Meteorological SocietyCredible Interval Temperature Forecasts
39 Citations1972Cameron R. Peterson, Kurt J. Snapper +1 more
The results of the experiment indicate that forecasters can use credible intervals to quantify this information, information which may be important to potential users of these forecasts.
Organizational Behavior and Human PerformanceThe influence of prior probabilities on information seeking and decision-making
39 Citations1969Gordon F. Pitz
PubMedInitial evaluation of a subjective Bayesian diagnostic system.
38 Citations1971David H. Gustafson, John J. Kestly +2 more
A computer-aided diagnostic system using subjectively estimated probabilities for symptom—disease relationships is described and applied to a sample of 200 cases, indicating that the subjective model's developmental cost and time requirement is much lower, while it performs as well as either actuarial model.
Investigative RadiologyAnalyzing Use of Diagnostic Signs
37 Citations1971Paul Slovic, Leonard G. Rorer +1 more
Organizational Behavior and Human PerformanceResearch on human multistage probabilistic inference processes
36 Citations1973David A. Schum, Wesley M. Du Charme +1 more
Journal of Chronic DiseasesComputer-aided diagnosis of cardiovascular disorders
33 Citations1966Robert A. Bruce, Stephen R. Yarnall
The computer can help in early detection of disease in screening surveys by allowing more sensitive analysis of exercise cardiograms, by actually screening some data which can be obtained prior to physician evaluation, and lastly by optimizing data interpretation and strength of prediction for early disease patterns.
Management ScienceA Bayesian Approach to Quality Control
33 Citations1972Phillip L. Carter
The purpose of this paper is to extend the Bayesian approach to include consideration of the sample size and the sampling interval in the design of the control procedure and to show how the optimal time between samples and the optimal sample size can be found andHow the optimal decision can be made based on the outcome of the samples.
Organizational Behavior and Human PerformanceMultiple regression as a model for human information utilization
32 Citations1967Lee Roy Beach
Journal of the American Statistical AssociationProbabilistic Prediction: Some Experimental Results
32 Citations1971Robert L. Winkler
Journal of Experimental PsychologyInferences based on unreliable reports.
32 Citations1971Kurt J. Snapper, Dennis G. Fryback
American Journal of RoentgenologyA COMPUTER PROGRAM FOR DIAGNOSIS OF THYROID DISEASE
28 Citations1966Lawrence T. Fitzgerald, John E. Overall +1 more
A program, with flow-charts, for the diagnosis of thyroid disease based on experience gained with 1,379 cases of suspected thyroid disease has been written in Fortran II (for the IBM 709) and is available on request.
IEEE Transactions on ReliabilityUtilization of Reliability Measurements in Bayesian Inference: Models and Human Performance
27 Citations1973Edgar M. Johnson, Raymond C. Cavanagh +2 more
Organizational Behavior and Human PerformancePsychological effects of proper scoring rules
25 Citations1973Floyd A. Jensen, Cameron R. Peterson
Journal of the Operational Research SocietyA Predictive Information System for Management
24 Citations1969John Kidd, James Morgan
The design and implementation of a management information system which deals with statements about events that can be expressed only in terms of probabilities is described, giving some insight into problems of co-ordination in decentralized organizations.
Organizational Behavior and Human PerformanceDeviations from optimum information-purchase strategies in human decision-making
24 Citations1970Ramon L. Hershman, Joseph R. Levine
IEEE Transactions on Human Factors in ElectronicsStudies in Probabilistic Information Processing
24 Citations1966Richard J. Kaplan, J. R. Newman
This paper outlines a theory of Probabilistic Information Processing and describes three experimental studies testing that theory, based on certain principles of Bayesian statistical decision theory, which is designed as an aid to human diagnostic decision-making.
Journal of applied meteorologyA Bayesian Approach to Decision Making in Applied Meteorology
21 Citations1962Edward S. Epstein
IEEE Transactions on Man Machine SystemsSubjective Probabilities in Medical Diagnosis
21 Citations1969David H. Gustafson, Ward Edwards +2 more
It is proposed that computer-based medical diagnosis can become an effective and economically feasible tool through the routine use of appropriately obtained physician probability estimates.
Organizational Behavior and Human PerformancePrior uncertainty and amount of diagnostic evidence as variables in a probabilistic inference task
21 Citations1966David A. Schum
RadiologyThe Computer Evaluation and Diagnosis of Congenital Heart Disease, Using Roentgenographic Findings
20 Citations1966Arch W. Templeton, James L. Lehr +1 more
This study evaluates the occurrence of 20 roentgenographic parameters as found in 9 uncomplicated congenital heart lesions, which include secundum type of interatrial septal defect (IASD), primumtype of inter atrial sePTal defect, interventricular septAL defect, patent ductus arteriosus (PDA), tertralogy of Fallot, coarctation of the aorta and complete transposition of the great vessels.
Organizational Behavior and Human PerformanceAction selection and likelihood ratio estimation by individuals and groups
19 Citations1972Barbara C. Goodman
Organizational Behavior and Human PerformanceOn the choice of a consensus distribution in Bayesian analysis
18 Citations1972Robert L. Winkler, L. L. Cummings
The weighted-average method was more frequent and its use increased in the presence of high variance among experts' judgments, while the cognitive abilities, tolerances for ambiguity, and risk-taking propensities of the subjects were not related to their choices of a consensus distribution.
Organizational Behavior and Human PerformanceThe effect of learning on the assessment of subjective probability distributions
17 Citations1971Carl-Axel S. Staël von Holstein
Bulletin of the American Meteorological SocietyForecasters and Probability Forecasts: the Responses to a Questionnaire
16 Citations1971Allan H. Murphy, Robert L. Winkler
The responses to the questionnaire and subsequent discussions with Travelers Weather Service and National Weather Service forecasters suggest the presence of a number of “problems” related to probability forecasting.
Organizational Behavior and Human PerformanceConditional non-independence of data in a practical bayesian decision task
15 Citations1972Sarah Lichtenstein
In classifying 861 M MPI profiles as either neurotic or psychotic, two variants of Bayesian analysis techniques had cross-validated hit rates as high as comparable analyses which correctly handled the interrelationships among the MMPI scales.
PubMed CentralAn initial evaluation of a subjective Bayesian diagnostic system
15 Citations1971David H. Gustafson, John J. Kestly +2 more
Organizational Behavior and Human PerformanceProbabilistic information processing systems: Evaluation with conditionally dependent data
14 Citations1972Patricia A. Domas, Cameron R. Peterson
It is necessary to know if PIP is an appropriate system for the processing of dependent data, and a revised technology of diagnostic information processing based on conditionally dependent data was proposed.
The Medical Journal of AustraliaBAYES' THEOREM
12 Citations1971E.M. Nicholls, Alan E. Stark
Bayes' theorem provides a method of calculating degree of uncertainty and is shown by an example from orthopædic surgery that those medical actions which at the present time are based largely on intuitive summation of loss could be based on calculated numerical values.
Organizational Behavior and Human PerformanceHuman processing of inconclusive evidence from multinomial probability distributions
12 Citations1968David A. Schum, David W. Martin
IEEE Transactions on Systems Man and CyberneticsInformation Aggregation in Probabilistic Prediction
10 Citations1973Robert L. Winkler, Allan H. Murphy
Models of the aggregation process are discussed, with particular emphasis on the question of the conditional dependence of information, and measures of the redundancy of information are developed.
Bulletin of the American Meteorological SocietyHow Some Nonmeteorological Professionals View Meteorology and Weather Forecasting
8 Citations1971J. R. Scoggins, William W. Vaughan
Journal of applied meteorologyProbability Statements in Weather Forecasting
8 Citations1962Halbert E. Root
Journal of the Operational Research SocietyScoring Rules for Subjective Assessments
7 Citations1975John Kidd
The use of an assessment technique which can be used to check the goodness of these completion date estimates is illustrated by reference to experiments upon forecasting stockmarket index changes, which illustrate how good, biased and poor assessors can be named after a short period of forecast evaluations.
Organizational Behavior and Human PerformanceConditional dependence and decomposition strategies in diagnostic inference systems
5 Citations1973James H. Steiger, Charles F. Gettys
One major conclusion of this research is that logically equivalent decomposition rules are not necessarily of equal difficulty, and recommendations as to the superority of a particular decomposition rule should not be made.
Medical CareA Bayesian Approach to Rating Health Status of Geriatric Patients
5 Citations1971R. D. Parker
A Bayesian approach to the development of a geriatric index for rating the level of health care need is investigated in terms of the concepts of “hit rate,” i.e., correct classification rate.
Organizational Behavior and Human PerformanceOn the allocation of inference functions in decision systems
4 Citations1971William C. Howell, Charles F. Gettys +1 more
The conclusion seems to be that performance does improve with manually aggregated (unanticipated) data, but it asymptotes after relatively few (4–6) such items, which implies a limited human capacity.
International Journal of Legal MedicineDas Bayessche Theorem und die Überprüfung seiner Anwendung zur Berechnung der Vaterschaftsplausibilität
3 Citations1971H-D Wehner, Ch. Rittner
…