Large-Scale Bayesian Logistic Regression for Text Categorization

This work presents a simple Bayesian logistic regression approach that uses a Laplace prior to avoid overfitting and produces sparse predictive models for text data and applies this approach to a range of document classification problems and shows that it produces compact predictive models at least as effective as those produced by support vector machine classifiers or ridgelogistic regression combined with feature selection.

Abstract

AbstractLogistic regression analysis of high-dimensional data, such as natural language text, poses computational and statistical challenges. Maximum likelihood estimation often fails in these applications. We present a simple Bayesian logistic regression approach that uses a Laplace prior to avoid overfitting and produces sparse predictive models for text data. We apply this approach to a range of document classification problems and show that it produces compact predictive models at least as effective as those produced by support vector machine classifiers or ridge logistic regression combined with feature selection. We describe our model fitting algorithm, our open source implementations (BBR and BMR), and experimental results.KEY WORDS : Information retrievalLassoPenalizationRidge regressionSupport vector classifierVariable selection