Benchmarking default prediction models: pitfalls and remedies in model validation

The components of validating credit default models are discussed with a focus on potential challenges to making inferences from validation under realworld conditions and some empirical examples are presented that highlight instances where these effects can be severe and can thus compromise conclusions drawn from validation tests.

Abstract

We discuss the components of validating credit default models with a focus on potential challenges to making inferences from validation under realworld conditions. We structure the discussion in terms of: (a) the quantities of interest that may be measured (calibration and power) and how they can result in misleading conclusions if not taken in context; (b) a methodology for measuring these quantities that is robust to non-stationarity both in terms of historical time periods and in terms of sample firm composition; and (c) techniques that aid in the interpretation of the results of such tests. The approaches we advocate provide means for controlling for and understanding sample selection and variability. These effects can in some cases be severe and we present some empirical examples that highlight instances where they are and can thus compromise conclusions drawn from validation tests.