A survey of data provenance in e-science
ACM SIGMOD RecordPublished 1 September 2005
Yogesh Simmhan, Beth Plale, Dennis Gannon
Citations1,157
SJR quartileQ2
SJR score0.69
SNIP0.92
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The main aspect of the taxonomy categorizes provenance systems based on why they record provenance, what they describe, how they represent and storeprovenance, and ways to disseminate it.
Abstract
Data management is growing in complexity as large-scale applications take advantage of the loosely coupled resources brought together by grid middleware and by abundant storage capacity. Metadata describing the data products used in and generated by these applications is essential to disambiguate the data and enable reuse. Data provenance, one kind of metadata, pertains to the derivation history of a data product starting from its original sources.In this paper we create a taxonomy of data provenance characteristics and apply it to current research efforts in e-science, focusing primarily on scientific workflow approaches. The main aspect of our taxonomy categorizes provenance systems based on why they record provenance, what they describe, how they represent and store provenance, and ways to disseminate it. The survey culminates with an identification of open research problems in the field.
Keywords
Computer ScienceDecision Sciences
Lecture notes in computer scienceWhy and Where: A Characterization of Data Provenance
1,310 Citations2001Peter Buneman, Sanjeev Khanna +1 more
An approach to computing provenance when the data of interest has been created by a database query is described, adopting a syntactic approach and present results for a general data model that applies to relational databases as well as to hierarchical data such as XML.
Chimera: a virtual data system for representing, querying, and automating data derivation
608 Citations2002Ian Foster, Jens Vöckler +2 more
The Chimera virtual data system is developed, which combines avirtual data catalog for representing data derivation procedures and derived data, with a virtual data language interpreter that translates user requests into data definition and query operations on the database.
Trio: A System for Integrated Management of Data, Accuracy, and Lineage
477 Citations2004Jennifer Widom
This paper provides numerous motivating applications for Trio and lays out preliminary plans for the data model, query language, and prototype system.
ACM Computing SurveysLineage retrieval for scientific data processing: a survey
415 Citations2005Rajendra Bose, James Frew
Based on a comprehensive survey of lineage research and previous prototypes, a metamodel is presented to help identify and assess the basic components of systems that provide lineage retrieval for scientific data products.
The VLDB JournalLineage tracing for general data warehouse transformations
348 Citations2003Yangfan Cui, Jennifer Widom
This work formally defines the lineage tracing problem in the presence of general data warehouse transformations, and presents algorithms for lineage tracing in this environment, and can be used as the basis for a lineage tracing tool in a general warehousing setting.
The VLDB JournalAn annotation management system for relational databases
277 Citations2005Deepavali Bhagwat, Laura Chiticariu +2 more
An annotation management system for relational databases where every piece of data in a relation is assumed to have zero or more annotations associated with it and annotations are propagated along, from the source to the output, as data is being transformed through a query.
Supporting fine-grained data lineage in a database visualization environment
200 Citations2002Allison Woodruff, Michael Stonebraker
This paper proposes a novel method to support fine-grained data lineage that lazily computes the lineage using a limited amount of information about the processing operators and the base data, and introduces the notions of weak inversion and verification.
A survey of data provenance techniques
162 Citations2005Yogesh Simmhan, Beth Plale +1 more
The main aspect of the taxonomy categorizes provenance systems based on why they record provenance, what they describe, how they represent and storeprovenance, and ways to disseminate it can help those building scientific and business metadata-management systems to understand existing provenance system designs.
Practical lineage tracing in data warehouses
155 Citations2002Yangfan Cui, Jennifer Widom
A lineage tracing package for relational views with aggregation is implemented in the WHIPS data warehousing system prototype at Stanford, and a number of schemes for storing auxiliary views that enable consistent and efficient lineage tracing in a multi-source data warehouse are proposed.
The Virtual Data Grid: A New Model and Architecture for Data-Intensive Collaboration.
124 Citations2003Ian Foster, Jens-S. Vöckler +2 more
Cartography and Geographic Information SystemsDesign of a Lineage-Based Meta-Data Base for GIS
116 Citations1991David P. Lanter
A conceptual design is presented for a lineage meta-data base system that documents data sources and geographic information system transformations applied to derive cartographic products that enables GIS users to determine the fitness of spatial data sets.
ePrints Soton (University of Southampton)Provenance of e-Science Experiments - Experience from Bioinformatics
112 Citations2003Mark Greenwood, Carole Goble +7 more
An overview of initial work on the provenance of bioinformatics e-Science experiments within myGrid uses two kinds of provenance: the derivation path of information and annotation and explores how the resulting Webs of experimental data holdings can be mined for useful information and presentations for the e-Scientist.
Earth System Science Workbench: a data management infrastructure for earth science products
108 Citations2002James Frew, Rajendra Bose
An implementation of ESSW to track the processing of locally received satellite imagery is presented, demonstrating the Workbench's transparent and robust support for archiving and publishing data products.
Knowledge Provenance Infrastructure.
90 Citations2003Paulo Pinheiro da Silva, Deborah L. McGuinness +1 more
This article defines a broad notion of information provenance called knowledge provenance that includes proof-like information on how a question answering system arrived at its answer(s) and describes an approach for aknowledge provenance infrastructure supporting the extraction, maintenance and usage of knowledgeprovenance related to answers of web applications and services.
Lecture notes in computer scienceSemantically Linking and Browsing Provenance Logs for E-science
77 Citations2004Jun Zhao, Carole Goble +2 more
This paper describes how to assemble a Semantic Web of workflow provenance logs that allows a bioinformatician to browse and navigate between experimental components by generating hyperlinks based on semantic annotations associated with them.
ePrints Soton (University of Southampton)The requirements of recording and using provenance in e- Science experiments
76 Citations2005Simon Miles, Paul Groth +2 more
This paper presents use cases for a provenance architecture from current experiments in biology, chemistry, physics and computer science, and analyse the use cases to determine the technical requirements of a generic, application-independent architecture.
Lecture notes in computer scienceTowards Dynamically Adaptive Weather Analysis and Forecasting in LEAD
70 Citations2005Beth Plale, Dennis Gannon +5 more
An architectural framework that is forming the thinking about adaptability and give early solutions in workflow and monitoring is discussed.
ACM SIGMOD RecordDatabase management for life sciences research
63 Citations2004H. V. Jagadish, Frank Olken
The life sciences provide a rich application domain for data management research, with a broad diversity of problems that can make a significant difference to progress in life sciences research.
Multi-scale Science: Supporting Emerging Practice with Semantically Derived Provenance
62 Citations2003James D. Myers, Carmen M. Pancerella +4 more
This work is developing a general-purpose informatics-based approach that emphasizes ''on-demand'' metadata creation, configurable data translations, and semantic mapping to support the rapidly increasing and continually evolving requirements for managing data, metadata, and data relationships in multi-scale science projects.
Metadata in the collaboratory for multi-scale chemical science
58 Citations2003Carmen M. Pancerella, John C. Hewson +28 more
The project's metadata-related requirements, the relevant software infrastructure, core metadata schema, and tools that use the metadata to enhance science are discussed.
Composing lineage metadata with XML for custom satellite-derived data products
50 Citations2004Rajendra Bose, James Frew
This work proposes two techniques for composing lineage metadata, both based on accessory XML metadata documents that are paired with the data products and versioned data transformations they describe, and provides guidelines for compose lineage metadata that are applicable to other domains.
Lecture notes in computer scienceUsing Digital Library Techniques – Registration of Scientific Primary Data
42 Citations2004Jan Brase
The German national library of science and technology (TIB) has become the first registration agency worldwide for scientific primary data, to make these data citable as a unique piece of work and not only a part of a publication.
Improving Data Cleaning Quality using a Data Lineage Facility
39 Citations2001Héléna Galhardas, Inria Rocquencourt +5 more
A solution to handle the problem of removing inconsistencies and errors from original data sets by enabling users to express user interactions declaratively and tune data cleaning programs is presented.
Lecture notes in computer scienceWebservices Infrastructure for the Registration of Scientific Primary Data
7 Citations2005Uwe Schindler, Jan Brase +1 more
This paper shall give an overview about the technical realization of this important usage field for a digital library in the field of earth science.