Heteroscedastic multiple comparison procedures for computer simulation

This work uses a transformation called batching to change stochastic systems' data to approximately independent and normally distributed data to find the system with the largest expected performance using a class of confidence interval procedures known as Multiple Comparisons with the Best (MCB).

Abstract

We consider comparing a small, finite number of stochastic systems via computer simulation. We use a transformation called batching to change our stochastic systems' data to approximately independent and normally distributed data. Our goal is to find the system with the largest expected performance. We achieve this goal by using a class of confidence interval procedures known as Multiple Comparisons with the Best (MCB). MCB is a method that provides joint confidence intervals for the difference between the performance of each individual system and the (unknown) best system. Other MCB procedures precede this work, but our procedures uniquely allow for the unknown variances of the data to be different across systems. Unfortunately, an exact solution to our problem for single samples from each system would imply an exact solution to the Behrens-Fisher problem. Fortunately, there are a few approximate solutions to the Behrens-Fisher problem, and we extend one of them (the Welch moment approximation) to address our problem. In addition, we describe exact (generalized-means) and conservative (means) procedures that require two sampling stages from each system, as is common in simulation studies. The one-stage procedures we derive allow simulators to study their experiments results after the experiments are concluded, but do not provide convenient experimental designs. Two stage procedures have advantages over the one-stage procedures: They work with pilot results, require mild variance assumptions, and guarantee inference at a fixed precision. Both our one-stage and two-stage procedures could be easily incorporated into simulation packages. Although our research focuses on computer simulation experiments, our results apply to other settings including laboratory, work-place, and historical studies. Further, this work contributes to Statistics literature by describing heteroscedastic MCB. Additionally, MCB is introduced in an alternative manner using Hsu's Lemmas.