Linear Regression

Linear regression is a statistical method that models the relationship between a continuous dependent variable and one or more independent variables by fitting a linear equation to the observed data. The simplest form (simple linear regression) involves one predictor, while multiple linear regression extends the model to include two or more predictors simultaneously.

Linear regression is arguably the most widely used statistical method in quantitative research. It provides a framework for predicting outcomes, estimating the strength and direction of associations, testing hypotheses about relationships between variables, and controlling for confounders. It appears across virtually every research domain, from clinical medicine (predicting blood pressure from demographic and clinical variables) to economics (estimating the effect of education on earnings) to psychology (modeling cognitive performance as a function of age and training). Understanding linear regression, including its assumptions and limitations, is a foundational competency for any quantitative researcher.

Simple linear regression fits a straight line to the data using the equation Y = b₀ + b₁X + e, where Y is the dependent variable, X is the independent variable, b₀ is the intercept (the predicted value of Y when X equals zero), b₁ is the slope (the change in Y for each one unit increase in X), and e is the error term (the difference between the observed and predicted values of Y). The values of b₀ and b₁ are estimated using ordinary least squares (OLS), which minimizes the sum of the squared residuals (the vertical distances between observed data points and the fitted line).

Multiple linear regression extends this to Y = b₀ + b₁X₁ + b₂X₂ + ... + bₖXₖ + e, where each bₖ represents the change in Y associated with a one unit change in Xₖ, holding all other predictors constant. This "holding constant" interpretation is what makes multiple regression valuable for controlling confounders: by including potential confounders in the model, the coefficient for the primary predictor of interest estimates its association with the outcome independent of the other predictors in the model; it approximates a causal effect only if no important confounder is left out.

Several assumptions underlie OLS regression. Linearity requires that the relationship between each predictor and the outcome is linear. Independence requires that residuals are not correlated with each other. Homoscedasticity requires that the variance of residuals is constant across all levels of the predictors. Normality requires that the residuals follow a normal distribution, which is most important for inference (confidence intervals and hypothesis tests) in small samples. Violations of these assumptions can produce biased, inefficient, or misleading estimates. Diagnostic plots (residual plots, Q Q plots) and tests (Breusch Pagan, Durbin Watson) are used to check assumptions.

The coefficient of determination (R²) indicates the proportion of variance in Y explained by the model. Adjusted R² penalizes for the number of predictors, because R² never decreases when a predictor is added; it helps compare models of different sizes but does not by itself prevent overfitting. Individual predictors are assessed through their coefficients, standard errors, t statistics, p values, and confidence intervals. Standardized coefficients (beta weights) express predictors measured on different scales in standard deviation units, but they depend on how much each predictor varies in the sample, so they are a contested guide to relative importance.

Use linear regression when the dependent variable is continuous and the relationship between variables is approximately linear. It is appropriate for estimation, prediction, and hypothesis testing in both experimental and observational designs. When the dependent variable is binary, use logistic regression instead. When the dependent variable is a count, use Poisson or negative binomial regression. When the data have a hierarchical structure, use multilevel regression. When the relationship between variables is nonlinear, consider polynomial regression, splines, or generalized additive models.

A common error is interpreting regression coefficients as causal effects in observational data without accounting for unmeasured confounders. Regression can estimate associations, but causal claims require additional assumptions (such as no unmeasured confounders) or specialized methods (such as instrumental variables or randomization). Another mistake is including too many predictors relative to the sample size, which leads to overfitting and unstable estimates (a common guideline is at least 10 to 20 observations per predictor). Ignoring multicollinearity (high correlation between predictors) inflates standard errors and makes individual coefficients unreliable. Researchers also sometimes fail to check model assumptions, particularly linearity and homoscedasticity, before interpreting results. Finally, extrapolating predictions beyond the range of the observed data is unreliable because the linear relationship may not hold outside the range of the sample.

A public health researcher models the relationship between air pollution (measured as annual mean PM2.5 concentration in micrograms per cubic meter) and lung function (measured as FEV1 in liters) in 2,000 adults, controlling for age, sex, smoking status, and BMI. The multiple regression yields a coefficient for PM2.5 of negative 0.015 (95% CI: negative 0.022 to negative 0.008, p < 0.001), indicating that adults whose annual mean PM2.5 exposure was 1 microgram per cubic meter higher had, on average, a 0.015 liter lower FEV1, holding the other variables constant. The model explains 34% of the variance in FEV1 (R² = 0.34); this R² describes the whole model, including age, sex, smoking status and BMI, not PM2.5 alone.