Multiple Regression

Multiple regression is a statistical technique that models the relationship between a single dependent variable and two or more independent variables, allowing researchers to estimate the unique contribution of each predictor while controlling for the effects of the others. It extends simple linear regression to research questions with several predictors. Because it has a single outcome, it is a multivariable model, not a multivariate one (which has several outcomes).

Very few outcomes in research are determined by a single factor. Disease outcomes are shaped by combinations of demographic, clinical, behavioral, and environmental variables. Academic performance depends on individual ability, teaching quality, and home environment simultaneously. Multiple regression allows researchers to disentangle these intertwined influences, estimating the association of each predictor with the outcome "holding all others constant." This ability to adjust for measured confounders is what makes multiple regression one of the most important tools in observational research and a staple of data analysis across quantitative disciplines.

The multiple regression model takes the form Y = b₀ + b₁X₁ + b₂X₂ + ... + bₖXₖ + e, where Y is the dependent variable, X₁ through Xₖ are the independent variables, b₀ is the intercept, b₁ through bₖ are the regression coefficients (partial slopes), and e is the residual error. Each coefficient bₖ represents the expected change in Y for a one unit increase in Xₖ, holding all other predictors constant.

Model estimation typically uses ordinary least squares (OLS), which minimizes the sum of squared residuals. The overall model fit is assessed using R² (the proportion of variance in Y explained by the predictors) and adjusted R² (which penalizes for the number of predictors). The F test evaluates whether the model as a whole explains a statistically significant amount of variance. Individual predictors are evaluated using t tests (or equivalent Wald tests), which assess whether each coefficient differs significantly from zero.

Variable selection is a critical step. Theory driven approaches select predictors based on subject matter knowledge, specifying a priori which variables should be in the model and why. Data driven approaches (forward selection, backward elimination, stepwise methods) use statistical criteria to select predictors, but these methods are prone to overfitting and producing models that do not replicate. A balance between theory and empirical evidence is recommended. The number of predictors should be proportional to the sample size; a common guideline is a minimum of 10 to 20 observations per predictor for stable estimates.

Assumptions include linearity, independence of residuals, homoscedasticity, and normality of residuals. These are assessed through residual diagnostics (residual plots, Q Q plots) and, if violated, can be addressed through data transformation, heteroscedasticity consistent standard errors (such as White's or Huber White standard errors), or alternative model specifications.

Use multiple regression when the dependent variable is continuous, when two or more predictors are relevant, and when the goal is to estimate associations, adjust for confounders, or predict the outcome. It is a common approach for analyzing observational data where confounding is a concern. For binary outcomes, use logistic regression. For count outcomes, use Poisson or negative binomial regression. For clustered or nested data, use mixed effects models. For very high dimensional data (many predictors relative to observations), use regularization methods such as LASSO or ridge regression.

A common error is including too many predictors relative to the sample size, leading to overfitting and unreliable estimates. Another mistake is interpreting regression coefficients as causal effects in observational studies without acknowledging the potential for unmeasured confounding. Researchers sometimes include both a predictor and a variable that lies on the causal pathway between that predictor and the outcome (a mediator), which inappropriately adjusts away part of the effect they are trying to estimate. Using stepwise variable selection without strong theoretical justification produces models that may not replicate and that capitalize on sampling noise. Finally, failing to assess model assumptions and diagnostics can lead to biased or inefficient estimates.

An educational researcher investigates predictors of standardized test scores in 500 high school students. The multiple regression model includes socioeconomic status (SES, measured as a composite score), hours of homework per week, teacher experience (years), and student absence count. The model explains 42% of the variance in test scores (R² = 0.42, adjusted R² = 0.41). SES has the largest standardized coefficient (beta = 0.38, p < 0.001), followed by hours of homework (beta = 0.22, p < 0.001) and student absences (beta = negative 0.19, p < 0.001). Teacher experience is not statistically significant (beta = 0.06, p = 0.15), which does not show that it has no association with scores. Residual plots show no clear violations of model assumptions. The standardized coefficients depend on how much each predictor varies among these 500 students, so their order is a contested guide to relative importance, and because the data are observational they describe associations, not the effect of changing homework or attendance.