An Introduction to Coding Theory and the Two‐Part Minimum Description Length Principle

This article provides a tutorial introduction to the so‐called two‐part minimum description length (MDL) principle proposed by Rissanen, a powerful methodology for solving many statistical model selection problems.

Abstract

Summary This article provides a tutorial introduction to the so‐called two‐part minimum description length (MDL) principle proposed by Rissanen. This two‐part MDL principle is a powerful methodology for solving many statistical model selection problems. However, it seems that this powerful methodology is only adopted by a small number of statisticians to tackle a small number of problems. One plausible reason for this is that the coding theory results required by the MDL principle are somewhat new to most statisticians, and that there are not many readily accessible articles introducing these results appearing in the statistical literature. The first part of this article is devoted to a discussion of such coding theory results. Then, in the second part of the article, the two‐part MDL principle is introduced and explained. In doing so, only those coding theory results that are presented in the first part of the article are used. Finally, the applicability of the two‐part MDL principle is demonstrated by applying it to tackle four different statistical problems.