Intraclass Correlation Coefficient (ICC)

The intraclass correlation coefficient (ICC) is a measure of the degree of agreement or consistency among measurements made by multiple raters, across repeated measures of the same quantity, or between observations nested within groups. As a share of variance it ranges from 0 (all of the variation lies within subjects or groups, such as differences between raters or random error) to 1 (perfect agreement), although sample estimates can fall below 0, and the specific ICC model chosen depends on the study design and the nature of the raters.

The ICC serves multiple roles across research disciplines. In reliability studies, it quantifies how consistently raters, instruments, or methods produce the same result when measuring the same quantity. In multilevel modeling, the ICC indicates what proportion of total variance lies between groups (such as between hospitals, schools, or clinics), informing the decision about whether a hierarchical model is needed. In cluster randomized trials, the ICC directly affects sample size calculations because higher ICCs mean less independent information per individual, requiring more clusters to achieve the same statistical power. Accurate estimation and reporting of the ICC is therefore essential in measurement studies, intervention trials, and any research involving nested data.

The ICC is calculated as the ratio of between group variance to total variance (between group variance plus within group variance). However, there are multiple forms of the ICC, and choosing the correct one depends on the study design. Shrout and Fleiss (1979) defined six forms based on three factors: whether each subject is rated by a different set of raters (a one way model) or every subject is rated by the same raters (a two way model), whether those raters are a random sample from a larger population of raters or the only raters of interest (fixed), and whether the unit of analysis is a single measurement or the mean of several raters' measurements. Their two way forms also differ in what counts as agreement: ICC(2,1) measures absolute agreement, so systematic differences between raters lower it, while ICC(3,1) measures consistency, which ignores such differences; McGraw and Wong (1996) later made this choice explicit for each model.

ICC(1,1) is appropriate when each subject is rated by a different set of randomly selected raters. ICC(2,1) applies when all subjects are rated by the same set of randomly selected raters and absolute agreement is of interest. ICC(3,1) is used when all subjects are rated by the same set of fixed raters and consistency (rather than absolute agreement) is of interest. Two way models (ICC(2,1) and ICC(3,1)) are the most common in reliability studies.

A widely used rule of thumb from Koo and Li (2016), which they apply to the 95% confidence interval of the ICC, reads values below 0.50 as poor reliability, 0.50 to 0.75 as moderate, 0.75 to 0.90 as good, and above 0.90 as excellent. However, context matters: measurement tools used for clinical decision making require higher ICCs than those used for group level research comparisons.

In the context of multilevel modeling, the ICC is calculated as the between cluster variance divided by the total variance (between cluster plus within cluster) from an empty (intercept only) mixed effects model. An ICC of 0.05 in a cluster randomized trial of schools means that 5% of the variation in the outcome lies between schools. This value enters the design effect formula, which adjusts the effective sample size for clustering.

Calculate and report the ICC whenever you are assessing the reliability of a measurement involving multiple raters, repeated measures, or different instruments. Use it in the planning phase of cluster randomized trials to estimate the design effect and calculate the required number of clusters. Report it when fitting multilevel models to justify the need for a hierarchical approach. For binary outcomes, state which scale the ICC is on: the standard formula applied to 0/1 data gives an ICC on the proportion scale, which cluster trial sample size calculations often use, while a multilevel logistic model gives an ICC on the latent scale, and the two values differ. For two raters' categorical classifications, kappa is the usual agreement statistic.

A frequently made error is selecting the wrong ICC form for the study design, which can substantially affect the reported reliability. For example, using ICC(3,1) (which assumes fixed raters) when the raters are a random sample from a larger population produces estimates that do not generalize beyond the specific raters in the study. Another mistake is reporting the ICC without specifying which form was used, making it impossible for readers to evaluate or replicate the analysis. In cluster randomized trials, using an inaccurate ICC estimate in the sample size calculation can lead to underpowered studies. Finally, interpreting the ICC from a reliability study without considering the confidence interval (which is often wide with small samples) can give a false impression of precision.

A rehabilitation research team assesses the inter rater reliability of a new functional mobility scale by having three physical therapists independently rate 50 patients. Using a two way random effects model with absolute agreement (ICC(2,1)), they obtain an ICC of 0.88 (95% CI: 0.81 to 0.93), indicating good to excellent reliability. Based on this result, they conclude that the scale shows good to excellent inter rater reliability for therapists like these. They separately calculate the ICC from pilot data in a cluster randomized trial of 15 clinics, finding an ICC of 0.08, which they use to estimate the design effect (1 + (average cluster size minus 1) times ICC): a plan of 30 clinics with 25 patients each (750 patients) has a design effect of 2.92, so it carries about as much information as 257 independent patients.