Type S error rates for classical and Bayesian single and multiple comparison procedures

Abstract

In classical statistics, the significance of comparisons (e.g., θ1− θ2) is calibrated using the Type 1 error rate, relying on the assumption that the true difference is zero, which makes no sense in many applications. We set up a more relevant framework in which a true comparison can be positive or negative, and, based on the data, you can state "θ1 > θ2 with confidence," "θ2 > θ1 with confidence," or "no claim with confidence." We focus on the Type S (for sign) error, which occurs when you claim "θ1 > θ2 with confidence" when θ2> θ1 (or vice-versa). We compute the Type S error rates for classical and Bayesian confidence statements and find that classical Type S error rates can be extremely high (up to 50%). Bayesian confidence statements are conservative, in the sense that claims based on 95% posterior intervals have Type S error rates between 0 and 2.5%. For multiple comparison situations, the conclusions are similar.