Data Linkage

Data linkage is the process of combining records from two or more separately maintained data sources that relate to the same individual, event, or entity. It uses matching algorithms, either deterministic (based on exact agreement on a unique identifier or on a fixed set of identifiers) or probabilistic (based on weighted similarity across multiple variables), to connect records across databases that were not originally designed to be combined. Data linkage enables researchers to build comprehensive profiles of individuals or events by joining information scattered across multiple systems.

No single data source captures the full picture of a person's health, behavior, or outcomes. Hospital records do not include community pharmacy dispensing data. Cancer registries usually learn the date and cause of death of registered patients from death registration records, through linkage. Many surveys collect no clinical measurements. Data linkage makes it possible to answer research questions that require combining information from these separate sources, such as studying whether a medication dispensed in the community is associated with subsequent hospitalization, or whether participants in a clinical trial experienced long term outcomes recorded in a death registry. Pharmacoepidemiology, health services research, population health, and social science all rely heavily on linked data.

Deterministic linkage accepts a pair of records as a match when they meet fixed agreement rules: most simply, exact agreement on a unique identifier present in all datasets, such as national health insurance numbers, social security numbers, or study specific participant IDs, or exact agreement on a set of identifiers such as name, date of birth, sex and postcode, sometimes applied in a series of steps. When all records contain the same reliable identifier, deterministic linkage is fast and accurate. However, unique identifiers are often unavailable due to privacy restrictions, inconsistent data entry, or the absence of a universal identifier system.

Probabilistic linkage addresses this gap by using combinations of partially identifying variables, such as name, date of birth, sex, and address, to estimate how likely it is that two records refer to the same individual. The Fellegi Sunter model is the theoretical foundation for most probabilistic linkage systems. For each variable, it compares how often values agree among true matches (the m probability) with how often they agree by chance among non matches (the u probability), so agreement on a highly discriminating variable such as full date of birth adds more weight than agreement on sex, and disagreement subtracts weight. Frequency based weights also let agreement on a rare surname count for more than agreement on a common one. The weights are summed into a total match weight for each pair of records. Pairs with a total weight above an upper threshold are classified as links, those below a lower threshold are classified as non links, and those in between are possible links that may require clerical review. Where the pairs between the thresholds are not resolved by clerical review and stay unlinked, raising the upper threshold reduces false matches but increases missed matches, and lowering it does the reverse.

Privacy preserving linkage techniques allow matching without exposing personal information. Approaches include using encrypted or hashed identifiers (where a trusted third party or a secure multiparty computation protocol performs the linkage without any party seeing the other's raw data), and k anonymity based methods that link on generalized quasi identifiers. Hashing makes exact comparison of identifiers possible but hides near matches, so a single typing error in a hashed name prevents a match; encodings such as Bloom filters allow approximate matching of encoded identifiers. These techniques are increasingly important as data governance requirements become more stringent.

Quality assessment is essential. Researchers evaluate linkage quality by examining the false match rate (proportion of linked records that do not actually belong to the same individual) and the missed match rate (proportion of true matches that the algorithm failed to detect). Clerical review of a random sample of linked pairs estimates the false match rate. Estimating the missed match rate needs records whose true match status is known, such as a gold standard subset, or a search for the true matches of a sample of records that failed to link. Comparing the characteristics of linked and unlinked records shows whether linkage errors fall unevenly across groups. False matches attach information from the wrong person, which typically (but not always) dilutes associations and biases effect estimates toward the null. Missed matches reduce sample size and statistical power when unlinked records are excluded, and when an unlinked record is taken to mean that no event happened (for example, no death registration), they make outcomes look rarer than they are. Because linkage errors are often more common in some groups, such as people who move or change their names, both kinds of error can bias results.

Data linkage projects that use personal data generally need ethics approval (or a formal waiver) and data governance agreements, under rules that vary by jurisdiction. Because linkage creates richer datasets with more information about each individual, the re identification risk increases, and governance frameworks must address who can access the linked data, how long it can be retained, and what purposes it can serve.

Data linkage is appropriate when the research question requires information from multiple data sources that cannot be obtained from a single dataset. It is particularly useful for long term follow up of clinical trial participants, evaluation of healthcare policies using administrative data, pharmacoepidemiological safety studies, and population based research using registry data. It is not appropriate when a single data source contains all necessary information, when identifiers are too poor to produce reliable matches, or when ethical and legal constraints prevent the sharing of identifying information needed for linkage.

A common error is failing to assess linkage quality, assuming that linked records are correct without estimating error rates. Another mistake is using deterministic linkage with a single identifier (such as name) that may contain errors or refer to multiple individuals, producing false matches. Researchers sometimes neglect to account for records that could not be linked, which may represent a biased subset of the population (for example, people who changed their names, moved, or used different identifiers across systems). Ignoring data governance requirements can result in legal liability and damage to institutional reputation. Finally, treating linked data as complete and accurate without recognizing the inherent uncertainty in the linkage process leads to overconfident conclusions.

A pharmacoepidemiology team links prescription dispensing records from a national pharmacy database to hospital admission records using encrypted health insurance numbers managed by a trusted third party linkage unit. Because most patients are never admitted to hospital, the share of dispensing records with a linked admission is not a measure of linkage quality; instead, the team reports that 96.5% of dispensing records carry a valid insurance number that can be linked, and compares the other 3.5% with the rest to check whether they differ, for example in age or region. Clerical review by the linkage unit of 1,000 randomly selected linked pairs finds 3 false matches, an estimated false match rate of 0.3% (95% confidence interval about 0.1% to 0.9%). A patient with no linked admission is counted as not admitted, so any admissions that fail to link would make stroke hospitalization rates look lower than they are. The resulting dataset of 2.3 million patients enables a study of statin use and subsequent stroke hospitalization rates, adjusting for comorbidities recorded in the hospital data.