Logistic Regression

Logistic regression is a statistical method used to model the relationship between a binary dependent variable (one with two possible outcomes, such as disease present or absent) and one or more independent variables. It estimates the probability of the outcome occurring and expresses the effect of each predictor in terms of an odds ratio.

Many research outcomes are binary: a patient recovers or does not, a student passes or fails, a participant develops a disease or remains healthy. Linear regression is inappropriate for binary outcomes because it can predict probabilities outside the 0 to 1 range and violates the assumption of normally distributed residuals. Logistic regression resolves these problems by using the logit function to transform the probability of the outcome into a continuous, unbounded scale. It is a standard method for analyzing binary outcomes in clinical research, epidemiology, public health, social science, and machine learning, and it forms the basis for more advanced methods such as multinomial logistic regression (for outcomes with more than two categories) and ordinal logistic regression (for ordered categories).

Logistic regression models the log odds of the outcome as a linear function of the predictors: log(p / (1 minus p)) = b₀ + b₁X₁ + b₂X₂ + ... + bₖXₖ, where p is the probability of the outcome, (1 minus p) is the probability of the outcome not occurring, and the ratio p/(1 minus p) is the odds. The log odds (logit) is a continuous variable that ranges from negative infinity to positive infinity, making it suitable for linear modeling.

Each regression coefficient (b) represents the change in the log odds of the outcome associated with a one unit increase in the corresponding predictor, holding other predictors constant. Exponentiating the coefficient gives the odds ratio (OR), which is more interpretable: an OR greater than 1 indicates that the predictor is associated with higher odds of the outcome, an OR less than 1 indicates that it is associated with lower odds, and an OR of 1 indicates no association.

Model estimation typically uses maximum likelihood estimation (MLE), which finds the parameter values that make the observed data most probable. Calibration (how closely predicted probabilities match observed proportions) can be assessed with the Hosmer Lemeshow test or a calibration plot, discrimination (how well the model separates people with and without the outcome) with the area under the receiver operating characteristic (ROC) curve (AUC, or c statistic), and overall fit with measures such as the Nagelkerke pseudo R². Variable selection strategies include forward, backward, and stepwise methods, though theory guided selection of predictors is preferred.

Assumptions of logistic regression include linearity of the logit (the relationship between continuous predictors and the log odds is linear), absence of multicollinearity among predictors, independence of observations, and an adequate sample size. A common rule of thumb is at least 10 events per candidate predictor, but it is only a rule of thumb: Riley and colleagues (2020) show that the sample size needed to develop a prediction model depends on the number of candidate predictor parameters, how common the outcome is and the expected model fit.

Use logistic regression whenever the dependent variable is binary and you want to estimate the probability of the outcome, identify significant predictors, or calculate odds ratios. It is appropriate for both experimental and observational data. For outcomes with more than two unordered categories, use multinomial logistic regression. For ordered outcomes, use ordinal logistic regression. For time to event outcomes where censoring is present, use survival analysis methods instead. When the outcome is rare (a common rule of thumb is below 10%) and the study design is cross sectional or cohort based, the odds ratio approximates the relative risk; as the outcome becomes more common, the odds ratio lies further from 1 than the risk ratio, and log binomial regression or Poisson regression with robust (sandwich) standard errors may be preferred for directly estimating relative risks.

A common error is interpreting the odds ratio as a relative risk, which is appropriate only when the outcome is rare. For common outcomes, the odds ratio exaggerates the strength of the association relative to the risk ratio. For example, an odds ratio of 2.4 corresponds to a risk ratio of about 2.4 when 1% of the comparison group has the outcome, about 2.1 when 10% do, and about 1.5 when 40% do (risk ratio = OR ÷ (1 − p₀ + p₀ × OR), where p₀ is the risk in the comparison group; for adjusted odds ratios this conversion is only approximate). Another mistake is including too many predictors relative to the number of events (not the total sample size), which leads to overfitting, unstable estimates, and biased odds ratios. Some researchers fail to check the linearity of the logit assumption for continuous predictors, which can be assessed using restricted cubic splines or by categorizing the predictor and examining the pattern. Finally, using stepwise variable selection without theoretical justification produces models that are unstable and may not replicate in new samples.

An epidemiologist investigates risk factors for hospital acquired infections in a cohort of 3,000 surgical patients, of whom 240 (8%) develop an infection. Logistic regression is used to model infection status (yes or no) as a function of age, BMI, diabetes status, surgical duration, and antibiotic prophylaxis. The model yields an odds ratio of 2.4 (95% CI: 1.7 to 3.4) for diabetes, indicating that patients with diabetes have 2.4 times the odds of infection compared to patients without diabetes, controlling for the other variables. The AUC is 0.76, indicating acceptable discrimination.