Data Curation

Data curation encompasses the management, processing, and enrichment of research data to produce well organized, thoroughly documented, and interoperable datasets suitable for analysis, sharing, and long term preservation. It extends beyond simple data storage to include activities such as quality assurance, metadata creation, format standardization, and the application of disciplinary standards that ensure datasets remain usable over time. Data curation bridges the gap between raw data as collected and analysis ready data as needed.

Raw research data are rarely in a form that others can understand and use without significant effort. Variable names may be cryptic, coding schemes undocumented, file formats proprietary, and metadata absent. As open science policies gain traction and funders increasingly require data management plans with provisions for sharing, the ability to curate data effectively has become a critical skill for research teams. Well curated data extend the value of research investments by enabling secondary analyses, meta analyses, and interdisciplinary reuse that the original investigators may not have anticipated. Poorly curated data, by contrast, may become unusable within years as team members leave, software versions change, and institutional knowledge is lost.

Curation begins with planning. A data management plan, which many funders require either with the grant application or early in the funded project, outlines how data will be organized, documented, stored, and eventually shared. During data collection, curation activities include applying consistent file naming conventions, maintaining a data dictionary that defines every variable, and using standardized formats and coding schemes.

After data collection, curation involves several interrelated activities. Quality assurance checks verify data integrity, identifying corrupted files, incomplete records, and inconsistencies. Metadata creation applies descriptive information following established standards such as Dublin Core (for general descriptive metadata), Data Documentation Initiative (DDI) for social science data, or the standards of the Clinical Data Interchange Standards Consortium (CDISC) for clinical trial data. This metadata enables discoverability by making the dataset findable in search engines and data catalogs.

Format standardization converts data from proprietary formats to open, widely supported formats such as CSV, JSON, or discipline specific standards. This ensures that the data remain accessible regardless of changes in commercial software licensing or availability. Version control systems track changes to datasets over time, allowing researchers to identify when and why modifications were made and to revert to earlier versions if needed.

Preparation for repository deposit is the final step. Curators assess whether the dataset meets the repository's submission requirements, resolve any remaining quality issues, prepare documentation, and assign or verify persistent identifiers such as DOIs. Many university libraries and data centers provide curation services, and domain specific repositories often offer guidance tailored to their fields.

Data curation should be integrated into the research workflow from the outset, not treated as an afterthought at the end of a project. It is essential whenever data will be shared with collaborators, deposited in a repository, or preserved for long term access. It is especially important for longitudinal studies where data are collected over many years, multisite studies where harmonization is required, and interdisciplinary projects where team members from different fields need to work with the same data. Even for single investigator projects, good curation practices protect against data loss and make it possible for the researcher to return to their own data years later and understand it.

A common mistake is treating curation as a one time activity at the end of a project rather than an ongoing process integrated into data collection. By the time a project ends, key details about variable definitions, coding decisions, and data collection anomalies may be forgotten. Another error is using file naming conventions that are inconsistent or uninformative (such as "data_final_v2_FINAL.xlsx"), making it hard to tell which file is current. Researchers sometimes document metadata at a level that is too abstract to be useful, providing high level descriptions of the dataset without specifying the content, format, and meaning of individual variables. Finally, relying on proprietary file formats without maintaining open format copies risks making the data inaccessible when the software is no longer available or affordable.

A research data manager prepares a 15 year longitudinal cohort dataset for deposit in ICPSR. The work involves standardizing variable names across 15 annual collection waves (from inconsistent naming such as "bp_sys," "SystolicBP," and "sys_bp" to a uniform convention), creating a comprehensive codebook with 450 variable definitions, converting all data files from SPSS format to CSV with accompanying DDI XML metadata, and documenting three major changes in measurement methodology that occurred during the study. Once ICPSR releases it, the curated dataset has a DOI and is discoverable through Google Dataset Search.