Hypothesis Testing

Hypothesis testing is a formal statistical procedure that uses sample data to evaluate a claim about a population parameter. The process involves specifying a null hypothesis (no effect) and an alternative hypothesis (an effect exists), computing a test statistic from the data, and determining whether the observed result is unlikely enough under the null hypothesis to warrant its rejection.

Hypothesis testing provides the logical framework through which quantitative researchers make evidence based decisions. It is the mechanism by which clinical trials determine whether treatments work, epidemiologists identify risk factors, psychologists evaluate interventions, and economists test policy predictions. Medicines regulators such as the FDA and EMA rely on hypothesis tests in confirmatory trials when evaluating new drugs, following the ICH E9 guideline on statistical principles for clinical trials. Misunderstanding or misapplying hypothesis testing can lead to incorrect conclusions that affect patient care, policy, and scientific progress. A hypothesis test addresses only random error: a significant result does not rule out bias or confounding, and it says nothing on its own about whether the findings apply to other populations or settings.

The hypothesis testing procedure follows a structured sequence. First, the researcher states the null hypothesis (H₀), which typically asserts no difference or no effect, and the alternative hypothesis (H₁), which asserts that an effect exists. Second, before seeing the data, the researcher selects a significance level (alpha), most commonly set at 0.05, which defines the threshold for how unlikely the observed result must be under H₀ before the researcher will reject H₀. Third, the researcher chooses an appropriate test statistic based on the data type, sample size, and research design. Common test statistics include the t statistic (for comparing means), the chi square statistic (for categorical data), the F statistic (for comparing variances or group means in ANOVA), and the z statistic (for large samples).

After data collection, the test statistic is calculated and compared against its theoretical distribution under H₀ to obtain a p value. The p value represents the probability of observing a result as extreme as, or more extreme than, the one obtained, assuming H₀ is true. If the p value falls below alpha, the result is declared statistically significant, and H₀ is rejected in favor of H₁. If the p value exceeds alpha, the researcher fails to reject H₀, which is not the same as confirming that H₀ is true. This procedure blends two historical approaches: Ronald Fisher treated the p value as a graded measure of the evidence against H₀, while Jerzy Neyman and Egon Pearson framed testing as a rule for deciding between H₀ and H₁, with error rates fixed before the data are seen.

Two types of decision errors are inherent in this framework. A Type I error (false positive) occurs when H₀ is true but is incorrectly rejected; when H₀ is true, the test keeps the probability of this error at or below alpha. Alpha is therefore an error rate of the procedure over many tests of true null hypotheses, not the probability that a particular significant result is a false positive. A Type II error (false negative) occurs when H₁ is true but the test fails to reject H₀; its probability is denoted beta, and statistical power (1 minus beta) is the probability of correctly detecting a true effect. Balancing these error rates requires careful consideration of sample size, effect size, and the consequences of each type of error.

Hypothesis testing is appropriate whenever you have a specific, predefined prediction to evaluate with quantitative data. It is standard in clinical trials, experiments, and observational studies that aim to test causal or associative claims. It is less appropriate for purely exploratory analyses, where the goal is to generate hypotheses rather than test them. In those contexts, descriptive statistics, data visualization, and estimation (confidence intervals) may be more informative than dichotomous reject or fail to reject decisions.

A common mistake is interpreting the p value as the probability that the null hypothesis is true, when it actually represents the probability of obtaining data at least as extreme as those observed, assuming that the null hypothesis and the test's other assumptions are true. Nor is it the probability that the result arose by chance alone, as the American Statistical Association's 2016 statement on p values stresses. Another error is treating "statistically significant" as synonymous with "clinically important" or "practically meaningful." A statistically significant result with a trivially small effect size may have no real world relevance. Researchers also sometimes conduct multiple hypothesis tests without adjusting for multiple comparisons, which inflates the overall false positive rate. Finally, the practice of p hacking, selectively reporting analyses that produce significant results while omitting those that do not, undermines the integrity of hypothesis testing.

A clinical researcher conducts an RCT comparing a new analgesic to placebo for postoperative pain in 150 patients, 75 per group. The primary outcome is pain score on a 0 to 10 visual analog scale at 24 hours. Using an independent samples t test, the researcher obtains a mean difference of 1.8 points (new drug: 3.2 vs. placebo: 5.0) with a t statistic of 3.45 (148 degrees of freedom) and a two sided p = 0.0007. Because p falls below the prespecified alpha of 0.05, the null hypothesis is rejected, and the researcher concludes that the new analgesic produces significantly lower pain scores than placebo. The p value does not give the probability that the drug works, and it does not say whether a 1.8 point reduction matters to patients: that judgment rests on the estimate and its 95% confidence interval (about 0.8 to 2.8 points), read against a minimal clinically important difference.