On Discriminative vs. Generative Classifiers: A comparison of logistic regression and naive Bayes

It is shown, contrary to a widely-held belief that discriminative classifiers are almost always to be preferred, that there can often be two distinct regimes of performance as the training set size is increased, one in which each algorithm does better.

Abstract

We compare discriminative and generative learning as typied by logistic regression and naive Bayes. We show, contrary to a widelyheld belief that discriminative classiers are almost always to be preferred, that there can often be two distinct regimes of performance as the training set size is increased, one in which each algorithm does better. This stems from the observation|which is borne out in repeated experiments|that while discriminative learning has lower asymptotic error, a generative classier may also approach its (higher) asymptotic error much faster. 1