Scoring rules and the evaluation of probabilities
TestPublished 1 June 1996
Robert L. Winkler, Javier A. V. Muñoz, José Cervera, José M. Bernardo, Gail Blattenberger, Joseph B. Kadane
Citations233
SJR quartileQ2
SJR score0.51
SNIP1.44
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Scoring rules and some related measures for evaluating probabilities are reviewed, including decompositions of scoring rules and attributes of “goodness” of probabilites, comparability of scores, and the design of scoring Rules for specific inferential and decision-making problems are reviewed.
Abstract
Attributes of “Good” probabilities, Decomposition of expected Scores, Evaluation of Probabilities, Probability Assessment, Probability Forecasts, Scoring Rules,
Keywords
Computer ScienceDecision Sciences
Journal of the Operational Research SocietyDecisions with Multiple Objectives: Preferences and Value Tradeoffs.
8,440 Citations1994Ralph L. Keeney, Howard Raiffa +1 more
TechnometricsStatistical Methods in the Atmospheric Sciences
7,690 Citations1996Philip H. Ramsey, Daniel S. Wilks
Uncertainty a guide to dealing with uncertainty in quantitative risk and policy analysis
2,778 Citations1990M. Granger Morgan, Max Henrion +1 more
This chapter discusses the nature and sources of uncertainty, human judgement about and with uncertainty, and the value of knowing how little you know Index.
Management ScienceGames with Incomplete Information Played by “Bayesian” Players, I–III: Part I. The Basic Model
2,181 Citations2004John C. Harsanyi
The paper develops a new theory for the analysis of games with incomplete information where the players are uncertain about some important parameters of the game situation, such as the payoff functions, the strategies available to various players, the information other players have about the game, etc.
The Annals of Mathematical StatisticsOn a Measure of the Information Provided by an Experiment
1,516 Citations1956D. V. Lindley
Statistical Methods in the Atmospheric Sciences: An Introduction
1,474 Citations1995Daniel S. Wilks
Theoretical Probability Distributions, Empirical Distributions and Exploratory Data Analysis, and Methods for Multivariate Data are reviewed.
Principals and Agents: The Structure of Business
1,279 Citations1990John W. Pratt, Richard Zeckhauser
Journal of the American Statistical AssociationElicitation of Personal Probabilities and Expectations
1,190 Citations1971Leonard J. Savage
Experts In Uncertainty
1,083 Citations1991Roger Cooke
Journal of applied meteorologyA New Vector Partition of the Probability Score
1,057 Citations1973Allan H. Murphy
Management ScienceScoring Rules for Continuous Probability Distributions
977 Citations1976James Matheson, Robert L. Winkler
Journal of the Royal Statistical Society Series B (Statistical Methodology)Rational Decisions
919 Citations1952I. J. Good
Cambridge University Press eBooksThe Analytics of Uncertainty and Information
865 Citations1992Jack Hirshleifer, John G. Riley
Weather and ForecastingWhat Is a Good Forecast? An Essay on the Nature of Goodness in Weather Forecasting
847 Citations1993Allan H. Murphy
Three distinct types of goodness are identified in this paper: the correspondence between forecasters’ judgments and their forecasts, the correspondencebetween the forecasts and the matching observations, and the incremental economic and/or other benefits realized by decision makers through the use of the forecasts (type 3 goodness, or value).
Monthly Weather ReviewA General Framework for Forecast Verification
687 Citations1987Allan H. Murphy, Robert L. Winkler
Statistical ScienceBayesian Analysis in Expert Systems
687 Citations1993David Spiegelhalter, A. P. Dawid +2 more
Using a real, moderately complex, medical example, it is illustrated how qualitative and quantitative knowledge can be represented within a directed graphical model, generally known as a belief network in this context.
Management ScienceExceptional Paper—Probability Encoding in Decision Analysis
665 Citations1975Carl Spetzler, Carl-Axel S. Staël von Holstein
The authors recommend and describe a structured interview process that utilizes a trained interviewer and a number of techniques designed to reduce biases and aid in the quantification of judgment.
TestAn overview of robust Bayesian analysis
610 Citations1994James O. Berger, Elı́as Moreno +21 more
An overview of the subject of robust Bayesian analysis is provided, one that is accessible to statisticians outside the field, and recent developments in the area are reviewed.
Journal of the Royal Statistical Society Series D (The Statistician)The Comparison and Evaluation of Forecasters
597 Citations1983Morris H. DeGroot, Stephen E. Fienberg
Journal of the American Statistical AssociationThe Well-Calibrated Bayesian
548 Citations1982A. P. Dawid
Organizational Behavior and Human PerformanceExternal correspondence: Decompositions of the mean probability score
438 Citations1982J. Frank Yates
North-Holland eBooksBayesian inference and decision techniques : essays in honor of Bruno de Finetti
365 Citations1986Arnold Zellner, Bruno de Finetti +1 more
Journal of the American Statistical AssociationThe Assessment of Prior Distributions in Bayesian Analysis
354 Citations1967Robert L. Winkler
Management ScienceState of the Art—Encoding Subjective Probabilities: A Psychological and Psychometric Review
354 Citations1983Thomas S. Wallsten, David V. Budescu
International Statistical ReviewScoring Rules and the Inevitability of Probability
338 Citations1982D. V. Lindley
Journal of applied meteorology“Good” Probability Assessors
331 Citations1968Robert L. Winkler, Allan H. Murphy
IEEE Transactions on Engineering ManagementEliciting probabilities from experts in complex technical problems
312 Citations1991Ralph L. Keeney, Detlof von Winterfeldt
A formal elicitation process to obtain probabilities from experts in a large-scale study involving nuclear safety, using primarily in-house experts and staff members of the project team was strongly criticized in peer review.
Journal of the American Statistical AssociationProbability Forecasting in Meteorology
304 Citations1984Allan H. Murphy, Robert L. Winkler
The history of probability forecasting in meteorology is reviewed to acquaint statisticians with this body of literature and recent methodological, experimental, and operational activities in this field are described.
Journal of the American Statistical AssociationInteractive Elicitation of Opinion for a Normal Linear Model
292 Citations1980Joseph B. Kadane, James Dickey +3 more
International Journal of ForecastingDiagnostic verification of probability forecasts
284 Citations1992Allan H. Murphy, Robert L. Winkler
A diagnostic approach to the evaluation of probability forecasts is outlined, illustrated by presenting some results of an analysis of U.S. National Weather Service probability of precipitation forecasts, which are relatively well-calibrated, unbiased, and skillful, but lacking in accuracy, refinement, resolution, and discrimination.
PsychometrikaAdmissible Probability Measurement Procedures
266 Citations1966Emir H. Shuford, Arthur Albert +1 more
In this case, it is shown that a certain minor modification of a scoring system with the reproducing property yields the desired admissible probability measurement procedure.
Journal of the American Statistical AssociationScoring Rules and the Evaluation of Probability Assessors
261 Citations1969Robert L. Winkler
Journal of the American Statistical AssociationThe Assessment of Prior Distributions in Bayesian Analysis
244 Citations1967Robert L. Winkler
Monthly Weather ReviewThe Value of Climatological, Categorical and Probabilistic Forecasts in the Cost-Loss Ratio Situation
229 Citations1977Allan H. Murphy
Journal of applied meteorologyOn Subjective Probability Forecasting
217 Citations1963Frederick Sanders
Proceedings of the National Academy of SciencesMEASURES OF THE VALUE OF INFORMATION
203 Citations1956John F. McCarthy
Journal of the American Statistical AssociationThe Quantification of Judgment: Some Methodological Suggestions
196 Citations1967Robert L. Winkler
British Journal of Mathematical and Statistical PsychologyMETHODS FOR DISCRIMINATING LEVELS OF PARTIAL KNOWLEDGE CONCERNING A TEST ITEM
194 Citations1965Bruno de Finetti
It is argued that where a person is uncertain as to the true answer in a multiple choice question, he should be encouraged to express his partial knowledge in terms of the subjective probability he attaches to each alternative being correct.
The Annals of StatisticsA General Method for Comparing Probability Assessors
181 Citations1989Mark J. Schervish
The general method for comparing forecasters after a finite number of trials is proven to include calculating all proper scoring rules as special cases and to be translated into a method for deciding who will give better forecasts in the future.
Games with Incomplete Information
155 Citations1994John C. Harsanyi
American Journal of PhysicsThe Scientist Speculates: An Anthology of Partly-baked Ideas
148 Citations1964Thornton Page
Management ScienceEvaluating and Combining Physicians' Probabilities of Survival in an Intensive Care Unit
147 Citations1993Robert L. Winkler, Roy M. Poses
Probability of survival assessed by physicians for patients admitted to an intensive care unit are studied, and the key factor in relative overall performance is the level of discrimination provided by the probabilities.
Cambridge University Press eBooksSeparating Probability Elicitation from Utilities
134 Citations1999Joseph B. Kadane
Monthly Weather ReviewForecast verification: Its Complexity and Dimensionality
130 Citations1991Allan H. Murphy
Two fundamental characteristics of forecast verification problems—complexity and dimensionality—are described and a general framework for the problem of absolute verification (AV) is extended to theproblem of comparative verification (CV).
Journal of applied meteorologyScalar and Vector Partitions of the Probability Score: Part I. Two-State Situation
120 Citations1972Allan H. Murphy
Journal of ForecastingConditional distribution analyses of probabilistic forecasts
116 Citations1985J. Frank Yates, Shawn P. Curley
Journal of Economic LiteratureThe Standard Error of Regressions
113 Citations1996Deirdre N. McCloskey
Medical Entomology and ZoologyAssessment and evaluation of subjective probability distributions
110 Citations1970Staël von Holstein, S Carl-Axel
Journal of the American Statistical AssociationProbabilistic Prediction
108 Citations1965Harry V. Roberts
Journal of the Royal Statistical Society Series D (The Statistician)Graphical Elicitation of a Prior Distribution for a Clinical Trial
107 Citations1993Kathryn Chaloner, Timothy R. Church +2 more
A method to help quantify beliefs in the form of a prior distribution about regression coefficients in a proportional hazards regression model is described, developed for a randomized trial comparing prophylaxes for toxoplasmosis in a population of HIV-positive individuals.
Monthly Weather ReviewImpacts of Feedback and Experience on the Quality of Subjective Probability Forecasts. Comparison of Results from the First and Second Years of the Zierikzee Experiment
105 Citations1984Allan H. Murphy, Harald Daan
Journal of the American Statistical AssociationSeparating Probability Elicitation from Utilities
104 Citations1988Joseph B. Kadane, Robert L. Winkler
Management ScienceEvaluating Probabilities: Asymmetric Scoring Rules
93 Citations1994Robert L. Winkler
Journal of applied meteorologyAssessing the Value of Frost Forecasts to Orchardists: A Dynamic Decision-Making Approach
73 Citations1982Richard W. Katz, Allan H. Murphy +1 more
The methodology of decision analysis is used to investigate the economic value of frost (i.e., minimum temperature) forecasts to orchardists, and a decision-making model for the fruit-frost problem is presented.
Monthly Weather ReviewTHE RANKED PROBABILITY SCORE AND THE PROBABILITY SCORE: A COMPARISON<sup>1,,</sup><sup>2</sup>
67 Citations1970Allan H. Murphy
Monthly Weather ReviewGeneral Decompositions of MSE-Based Skill Scores: Measures of Some Basic Aspects of Forecast Quality
66 Citations1996Allan H. Murphy
The American StatisticianSeparating the Brier Score into Calibration and Refinement Components: A Graphical Exposition
65 Citations1985Gail Blattenberger, Frank Lad
A graphical description of the Brier score (quadratic loss) of assessed probabilities for a sequence of observable events and the implications of long-run calibration for probability theory based solely on the principle of coherence are outlined.
Journal of applied meteorologyHedging and Skill Scores for Probability Forecasts
64 Citations1973Allan H. Murphy
Management ScienceExpert Resolution
63 Citations1986Robert L. Winkler
Some current issues in the combination of probabilities are discussed and the axiomatic approach and the modeling approach to expert resolution are discussed, and some comments on practical issues are given.
Munich Personal RePEc Archive (Ludwig Maximilian University of Munich)Assessing Probability Assessors: Calibration and Refinement.
55 Citations1981Morris H. DeGroot, Stephen E. Fienberg
The Annals of StatisticsA Statistical Diptych: Admissible Inferences--Recurrence of Symmetric Markov Chains
48 Citations1992Morris L. Eaton
Organizational Behavior and Human Decision ProcessesAnalyzing the accuracy of probability judgments for multiple events: An extension of the covariance decomposition
42 Citations1988J. Frank Yates
Monthly Weather ReviewBayesian Correlation Score: A Utilitarian Measure of Forecast Skill
39 Citations1992Roman Krzysztofowicz
International Journal of Man-Machine StudiesAn economic basis for certain methods of evaluating probabilistic forecasts
33 Citations1978Judea Pearl
The paper establishes relations between the compatibility measure and the user's distribution of future pay-offs and shows that each of the commonly used measures represents a natural payoff emanating from an economic environment of a specific character.
Monthly Weather ReviewThe Family of Quadratic Scoring Rules
23 Citations1978Carl-Axel S. Staël von Holstein, Allan H. Murphy
Management ScienceAdaptive Forecasting Models Based on Predictive Distributions
23 Citations1978Robert L. Winkler, Wayne S. Smith +1 more
A general approach to forecasting is developed in this paper, and a specific model, a linear model with normally distributed errors, is considered in some detail.
TestSeveral Bayesians: A review
22 Citations1993Joseph B. Kadane, Javier Girón +6 more
This paper synthesizes a number of papers in which the author has participated that all concern models in which several decision-makers, each modeled as a Bayesian, appear, and reviews the central ideas.
Monthly Weather ReviewA Coherent Method of Stratification within a General Framework for Forecast Verification
18 Citations1995Allan H. Murphy
Approximations in Statistics from a Decision-Theoretical Viewpoint
16 Citations1987José M. Bernardo
The logarithmic divergence is shown to play a rather special role within this approach to the approximation of the probability density p of a random vector x by another probability density q(.) which belongs to a certain class Q.
Management SciencePerformance-Based Incentive Plans
14 Citations1980Rakesh K. Sarin, Robert L. Winkler
This paper shows how an incentive plan can be devised when incentive compensation to managers is based on their actual performance on some predetermined criteria, and considers goal-based plans that encourage managers to set accurate goals and motivate them to strive for a better performance subsequent to goal setting.
Journal of Business and Economic StatisticsAn Application of Operational-Subjective Statistical Methods to Rational Expectations
4 Citations1988Gail Blattenberger, Frank Lad
Journal of Applied EconometricsMoney demand revisited: An operational subjective approach
1 Citations1996Gail Blattenberger
…