DataHub: Collaborative Data Science & Dataset Version Management at Scale
arXiv (Cornell University)Published 2 September 2014Open access
Anant Bhardwaj, Souvik Bhattacherjee, Amit Chavan, Amol Deshpande, Aaron J. Elmore, Samuel Madden
Citations112
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This work proposes a dataset version control system, giving users the ability to create, branch, merge, difference and search large, divergent collections of datasets, and a platform, DATAHUB, that gives users the able to perform collaborative data analysis building on this versions control system.
Abstract
Relational databases have limited support for data collaboration, where teams collaboratively curate and analyze large datasets. Inspired by software version control systems like git, we propose (a) a dataset version control system, giving users the ability to create, branch, merge, difference and search large, divergent collections of datasets, and (b) a platform, DataHub, that gives users the ability to perform collaborative data analysis building on this version control system. We outline the challenges in providing dataset version control at scale.
Keywords
Computer ScienceDecision Sciences
Lecture notes in computer scienceA Digital Signature Based on a Conventional Encryption Function
1,803 Citations1988Ralph C. Merkle
A new digital signature based only on a conventional encryption function (such as DES) is described which is as secure as the underlying encryption function -- the security does not depend on the difficulty of factoring and the high computational costs of modular arithmetic are avoided.
Elsevier eBooksRole-Based Access Control
1,796 Citations2002
Role-based Access Control is a new authorization schema that implements authorization and revocation of privilege by assigning and revoking roles, and provides role-assignment rules and realizes the logical separation of users and privileges.
Scientific ProgrammingPegasus: A Framework for Mapping Complex Scientific Workflows onto Distributed Systems
1,229 Citations2005Ewa Deelman, Gurmeet Singh +11 more
The results of improving application performance through workflow restructuring which clusters multiple tasks in a workflow into single entities are presented.
Chimera: a virtual data system for representing, querying, and automating data derivation
608 Citations2002Ian Foster, Jens Vöckler +2 more
The Chimera virtual data system is developed, which combines avirtual data catalog for representing data derivation procedures and derived data, with a virtual data language interpreter that translates user requests into data definition and query operations on the database.
IEEE Transactions on Knowledge and Data EngineeringTemporal and real-time databases: a survey
470 Citations1995Gültekin Özsoyoğlu, Richard T. Snodgrass
First the time domain is characterized; then the real-time data models are investigated; then time-varying query languages are evaluated along several dimensions.
ACM Computing SurveysComparison of access methods for time-evolving data
326 Citations1999Betty Salzberg, Vassilis J. Tsotras
This paper compares different indexing techniques proposed for supporting efficient access to temporal data based on a collection of important performance criteria, including the space consumed, update processing, and query time for representative queries.
VisTrails: Enabling Interactive Multiple-View Visualizations
295 Citations2006Louis Bavoil, Steven P. Callahan +5 more
The design and implementation of VisTrails are described, the effectiveness of the system is shown, and its effectiveness in different application scenarios is shown.
ACM Transactions on Database SystemsExtended algebra and calculus for nested relational databases
279 Citations1988Mark A. Roth, Herry F. Korth +1 more
This paper takes a first step towards unifying the various theories of ¬1NF databases by determining an appropriate model to couch their formalisms in and defining an extended relational calculus as the theoretical basis for their database query language.
Querying data provenance
196 Citations2010Grigoris Karvounarakis, Zachary G. Ives +1 more
A query language for provenance is developed, which can express all of the aforementioned types of queries, as well as many more, and the feasibility of provenance querying and the benefits of the indexing techniques across a variety of application classes and queries are experimentally validated.
IEEE Data(base) Engineering BulletinProvenance in Scientific Workflow Systems
167 Citations2007Susan B. Davidson, Sarah Cohen‐Boulakia +6 more
The Choosing Wisely campaign “is about physicians and patients having conversations, informed conversations, about making wise choices, about their care, and avoiding unnecessary tests and procedures that are not supported by evidence,” said Christine Cassel at a press briefing in Washington, DC.
Efficient snapshot retrieval over historical graph data
165 Citations2013Udayan Khurana, Amol Deshpande
An in-memory graph data structure called GraphPool that can maintain hundreds of historical graph instances in main memory in a non-redundant manner and develop analytical models for both the storage space needed and the snapshot retrieval times to aid in choosing the right construction parameters for a specific scenario.
ACM Transactions on Database SystemsArchiving scientific data
136 Citations2004Peter Buneman, Sanjeev Khanna +2 more
This article develops an archiving technique that is both efficient in its use of space and preserves the continuity of elements through versions of the database, something that is not provided by traditional minimum-edit-distance diff approaches and uses timestamps.
ORCHESTRA : Rapid, collaborative sharing of dynamic data
92 Citations2005Zachary G. Ives, Nitin Khandelwal +2 more
This paper describes the initial design and prototype of the ORCHESTRA system, which focuses on managing disagreement among multiple data representations and instances and represents an important evolution of the concepts of peer-to-peer data sharing, which considers revision, disagreement, authority, and intermittent participation.
ArXiv.orgA Case for A Collaborative Query Management System
64 Citations2009Nodira Khoussainova, Magdalena Bałazińska +3 more
Lecture notes in computer scienceDatabase-as-a-Service for Long-Tail Science
46 Citations2011Bill Howe, Garret Cole +5 more
It is concluded that the SQLShare system and associated services have the potential to increase uptake of relational database technology in the long tail of science.
Efficient Versioning for Scientific Array Databases
39 Citations2012Adam Seering, Philippe Cudré-Mauroux +2 more
This paper describes a versioned database storage manager for SciDB, designed to efficiently store and retrieve array-oriented data, exposing a "no-overwrite" storage model in which each update creates a new "version" of an array.
Big Data Storytelling Through Interactive Maps
14 Citations2012Jayant Madhavan, S. Balakrishnan +11 more
This paper highlights Google Fusion Tables support for map visualizations over large complex geospatial datasets in order to help journalists in numerous high-profile stories.