Principal Component Analysis (PCA)

Principal component analysis (PCA) is a dimensionality reduction technique that transforms a set of correlated variables into uncorrelated linear combinations called principal components, each capturing the most variance left by earlier ones, with the first few usually kept. The first principal component accounts for the largest share of total variance, with each subsequent component capturing progressively less. PCA is widely used in exploratory data analysis, feature engineering, and multivariate statistics.

Researchers in many disciplines face datasets with dozens or hundreds of variables, many of which are highly correlated. Analyzing all variables simultaneously leads to multicollinearity, computational inefficiency, and difficulty interpreting results. PCA addresses these problems by reducing the number of variables while retaining most of the information in the original dataset. In genomics, PCA identifies population structure from thousands of genetic markers. In psychology, it is often used to summarize large item pools, although factor analysis is the method designed to identify underlying constructs. In environmental science, it reduces complex pollution or climate datasets to interpretable dimensions. PCA also plays a central role in data visualization, enabling researchers to plot high dimensional data in two or three dimensions for pattern detection.

PCA begins by centering each variable at a mean of zero and, when the variables are measured on different scales, standardizing them so that each also has a standard deviation of one. This ensures that variables measured on different scales contribute equally to the analysis. Next, the method computes the covariance matrix of the variables (for standardized variables, this is their correlation matrix), which captures the linear relationships among all pairs.

The algorithm then extracts eigenvectors and eigenvalues from this matrix. Each eigenvector defines a principal component as a weighted combination of the original variables, and its corresponding eigenvalue is the variance that component captures; dividing it by the sum of all the eigenvalues (the number of variables, when they are standardized) gives the proportion of total variance the component explains. The components are mutually orthogonal (uncorrelated), and they are ordered by the amount of variance they capture.

Researchers must decide how many components to retain. Common criteria include the Kaiser rule (retaining components with eigenvalues greater than one, which applies to standardized variables and, used alone, tends to retain too many), parallel analysis, the scree plot (identifying the "elbow" where variance explained levels off), and cumulative variance thresholds (such as retaining enough components to explain 80% or 90% of total variance). The retained components can then be used as input variables for subsequent analyses such as regression or clustering.

Interpreting principal components requires examining the loadings, which indicate how strongly each original variable contributes to each component. High loadings suggest that a variable is closely aligned with that component. Naming components based on their loading patterns is a subjective step that requires domain knowledge. Unrotated principal components seldom show simple structure (each variable loading strongly on only one component); rotation can help, but if the goal is to identify latent constructs, exploratory factor analysis, which models the variance the variables share rather than their total variance, is usually preferred.

Use PCA when your dataset contains many correlated variables and you want to reduce dimensionality before subsequent analysis, remove multicollinearity, or visualize high dimensional data. It is appropriate for continuous variables measured at the interval or ratio level. PCA is not suitable for categorical variables (consider multiple correspondence analysis instead) or when the goal is to identify latent constructs with theoretically meaningful interpretations (factor analysis may be more appropriate). PCA assumes linear relationships among variables; nonlinear patterns require kernel PCA or other nonlinear techniques.

A common error is applying PCA to unstandardized variables measured on different scales, allowing variables with larger scales to dominate the components regardless of their importance. Another mistake is retaining too few or too many components without using systematic criteria, either losing important information or failing to achieve meaningful dimensionality reduction. Researchers sometimes interpret principal components as latent factors with causal meaning, overstating the conceptual significance of what is essentially a mathematical decomposition. Finally, applying PCA to small samples relative to the number of variables produces unstable solutions; a general guideline suggests at least five to ten observations per variable.

A team studying air quality in 150 cities measures 15 pollutant concentrations. Many pollutants are highly correlated (for instance, nitrogen dioxide and carbon monoxide from traffic). PCA reduces the 15 variables to four principal components explaining 82% of total variance. The first component loads heavily on traffic related pollutants, the second on industrial emissions, the third on photochemical byproducts, and the fourth on particulate matter from natural sources. These four component scores are then used as predictors in a regression model of respiratory hospital admissions.