Dremel
Proceedings of the VLDB EndowmentPublished 1 September 2010
Sergey Melnik, Andrey Gubarev, Jing Jing Long, Geoffrey Romer, Shiva Shivakumar, Matt Tolton
Citations709
SJR quartileQ1
SJR score1.83
SNIP1.80
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper describes the architecture and implementation of Dremel, and explains how it complements MapReduce-based computing, and presents a novel columnar storage representation for nested records.
Abstract
Dremel is a scalable, interactive ad-hoc query system for analysis of read-only nested data. By combining multi-level execution trees and columnar data layout, it is capable of running aggregation queries over trillion-row tables in seconds. The system scales to thousands of CPUs and petabytes of data, and has thousands of users at Google. In this paper, we describe the architecture and implementation of Dremel, and explain how it complements MapReduce-based computing. We present a novel columnar storage representation for nested records and discuss experiments on few-thousand node instances of the system.
Keywords
Computer Science
Communications of the ACMMapReduce
18,538 Citations2008Jay B. Dean, Sanjay Ghemawat
This presentation explains how the underlying runtime system automatically parallelizes the computation across large-scale clusters of machines, handles machine failures, and schedules inter-machine communication to make efficient use of the network and disks.
ACM SIGOPS Operating Systems ReviewThe Google file system
5,003 Citations2003Sanjay Ghemawat, Howard Gobioff +1 more
Foundations of databases
3,400 Citations1994Serge Abiteboul, Richard Hull +1 more
Bigtable: a distributed storage system for structured data
1,955 Citations2006Fay W. Chang, Jay B. Dean +7 more
Synthesis lectures on computer architectureThe Datacenter as a Computer: An Introduction to the Design of Warehouse-Scale Machines
1,746 Citations2009Luiz André Barroso, Urs Hölzle
The architecture of WSCs is described, the main factors influencing their design, operation, and cost structure, and the characteristics of their software base are described.
Pig latin
1,744 Citations2008Christopher Olston, Benjamin Reed +3 more
A new language called Pig Latin is described, designed to fit in a sweet spot between the declarative style of SQL, and the low-level, procedural style of map-reduce, which is an open-source, Apache-incubator project, and available for general use.
The Google file system
1,366 Citations2003Sanjay Ghemawat, Howard Gobioff +1 more
This paper presents file system interface extensions designed to support distributed applications, discusses many aspects of the design, and reports measurements from both micro-benchmarks and real world use.
Communications of the ACMMapReduce
1,182 Citations2009Jay B. Dean, Sanjay Ghemawat
MapReduce advantages over parallel databases include storage-system independence and fine-grain fault tolerance for large jobs.
Proceedings of the VLDB EndowmentHadoopDB
856 Citations2009A. A. ABOUZEID, Kamil Bajda-Pawlikowski +3 more
This paper explores the feasibility of building a hybrid system that takes the best features from both technologies; the prototype built approaches parallel databases in performance and efficiency, yet still yields the scalability, fault tolerance, and flexibility of MapReduce-based systems.
Proceedings of the VLDB EndowmentSCOPE
731 Citations2008Ronnie Chaiken, B. Keith Jenkins +5 more
A new declarative and extensible scripting language, SCOPE (Structured Computations Optimized for Parallel Execution), targeted for this type of massive data analysis, designed for ease of use with no explicit parallelism, while being amenable to efficient parallel execution on large clusters.
Operating Systems Design and ImplementationDryadLINQ: a system for general-purpose distributed data-parallel computing using a high-level language
699 Citations2008Yuan Yu, Michael Isard +5 more
Scientific ProgrammingInterpreting the Data: Parallel Analysis with Sawzall
616 Citations2005Rob Pike, Sean Dorward +2 more
The design -- including the separation into two phases, the form of the programming language, and the properties of the aggregators -- exploits the parallelism inherent in having data and computation distributed across many machines.
Lecture notes in computer scienceCounting Distinct Elements in a Data Stream
473 Citations2002Ziv Bar-Yossef, T. S. Jayram +3 more
Three algorithms to count the number of distinct elements in a data stream to within a factor of 1 ± ?
Communications of the ACMMapReduce and parallel DBMSs
427 Citations2009Michael Stonebraker, Daniel J. Abadi +5 more
MapReduce complements DBMSs since databases are not designed for extract-transform-load tasks, a MapReduce specialty.
ORDPATHs
382 Citations2004Patrick E. O׳Neil, Elizabeth O’Neil +4 more
A hierarchical labeling scheme called ORDPATH that is implemented in the upcoming version of Microsoft® SQL Server™ and supports insertion of new nodes at arbitrary positions in the XML tree, their ORDPath values "careted in" between OrDPATHs of sibling nodes, without relabeling any old nodes.
FlumeJava
336 Citations2010Craig Chambers, Ashish Raniwala +5 more
The combination of high-level abstractions for parallel data and computation, deferred evaluation and optimization, and efficient parallel primitives yields an easy-to-use system that approaches the efficiency of hand-optimized pipelines.
Proceedings of the VLDB EndowmentColumn-oriented database systems
233 Citations2009Daniel J. Abadi, Peter Boncz +1 more
This tutorial presents an overview of column-oriented database system technology and addresses questions about how easily a major row-based system achieve column-store performance and the new applications that can be potentially enabled by column-stores.
XMill
200 Citations2000Hartmut Liefke, Dan Suciu
A tool for compressing XML data, with applications in data exchange and archiving, which usually achieves about twice the compression ratio of gzip at roughly the same speed.
Synthesis lectures on computer architectureThe Datacenter as a Computer
194 Citations2009Luiz André Barroso, Urs Hölzle
Challenges in building large-scale information retrieval systems
190 Citations2009Jay B. Dean
This talk will discuss the evolution of Google's hardware infrastructure and information retrieval systems and some of the design challenges that arise from ever-increasing demands in all of these dimensions.
ACM SIGMOD RecordA recursive algebra and query optimization for nested relations
100 Citations1989Latha S. Colby