Dryad
Published 21 March 2007
Michael Isard, Mihai Budiu, Yuan Yu, Andrew Birrell, Dennis Fetterly
Citations2,460
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.
Abstract
Dryad is a general-purpose distributed execution engine for coarse-grain data-parallel applications. A Dryad application combines computational "vertices" with communication "channels" to form a dataflow graph. Dryad runs the application by executing the vertices of this graph on a set of available computers, communicating as appropriate through flies, TCP pipes, and shared-memory FIFOs.
Keywords
Computer Science
Communications of the ACMMapReduce
18,538 Citations2008Jay B. Dean, Sanjay Ghemawat
This presentation explains how the underlying runtime system automatically parallelizes the computation across large-scale clusters of machines, handles machine failures, and schedules inter-machine communication to make efficient use of the network and disks.
ACM SIGOPS Operating Systems ReviewThe Google file system
5,003 Citations2003Sanjay Ghemawat, Howard Gobioff +1 more
ACM Transactions on Computer SystemsThe click modular router
2,424 Citations2000Eddie Kohler, Robert Morris +3 more
On conventional PC hardware, the Click IP router achieves a maximum loss-free forwarding rate of 333,000 64-byte packets per second, demonstrating that Click's modular and flexible architecture is compatible with good performance.
Concurrency and Computation Practice and ExperienceDistributed computing in practice: the Condor experience
1,896 Citations2005Douglas Thain, Todd Tannenbaum +1 more
The history and philosophy of the Condor project is provided and how it has interacted with other projects and evolved along with the field of distributed computing is described.
The Google file system
1,366 Citations2003Sanjay Ghemawat, Howard Gobioff +1 more
This paper presents file system interface extensions designed to support distributed applications, discusses many aspects of the design, and reports measurements from both micro-benchmarks and real world use.
Communications of the ACMParallel database systems
1,281 Citations1992David J. DeWitt, Jim Gray
Over the last decade 'Eradata, Tandem, and a host of startup companies have successfully developed and marketed highly parallel machines that refutes a 1983 paper predicting the demise of database machines.
Concurrency Practice and ExperiencePVM: A framework for parallel distributed computing
1,173 Citations1990Vaidy Sunderam
The PVM system is a programming environment for the development and execution of large concurrent or parallel applications that consist of many interacting, but relatively independent, components that operate on a collection of heterogeneous computing elements interconnected by one or more networks.
Cilk
1,031 Citations1995Robert D. Blumofe, Christopher F. Joerg +4 more
This paper shows that on real and synthetic applications, the “work” and “critical path” of a Cilk computation can be used to accurately model performance, and proves that for the class of “fully strict” (well-structured) programs, the Cilk scheduler achieves space, time and communication bounds all within a constant factor of optimal.
IEEE MicroA case for NOW (Networks of Workstations)
797 Citations1995Thomas E. Anderson, David Culler +1 more
This paper identifies three opportunities for NOWs that will benefit end- users: dramatically improving virtual memory and file system performance by using the aggregate DRAM of a NOW as a giant cache for disk, achieving cheap, highly available, and scalable file storage by using redundant arrays of workstation disks, and multiple CPUs for parallel computing.
IEEE Transactions on Knowledge and Data EngineeringThe Gamma database machine project
667 Citations1990David J. DeWitt, Shahram Ghandeharizadeh +4 more
The design of the Gamma database machine and the techniques employed in its implementation are described and a thorough performance evaluation of the iPSC/s hypercube version of Gamma is presented.
Scientific ProgrammingInterpreting the Data: Parallel Analysis with Sawzall
616 Citations2005Rob Pike, Sean Dorward +2 more
The design -- including the separation into two phases, the form of the programming language, and the properties of the aggregators -- exploits the parallelism inherent in having data and computation distributed across many machines.
Cg
541 Citations2003William R. Mark, R. Steven Glanville +2 more
A programming language and supporting system that are designed for programming programmable floating-point vertex and fragment processors, with support for data-dependent control flow in the vertex processor.
Cluster-based scalable network services
527 Citations1997Armando Fox, Steven D. Gribble +3 more
A general, layered architecture for building cluster-based scalable network services that encapsulates the above requirements for reuse, and a service-programming model based on composable workers that perform transformation, aggregation, caching, and customization (TACC) of Internet content is proposed.
Proceedings of the 5th symposium on Operating systems design and implementation - OSDI '02
485 Citations2002
The implementation and performance-setting algorithms used in Vertigo, the authors' power management extensions for Linux, are described and it is shown that unlike conventional interval-based algorithms like Lon-gRun, Vertigo is successful at focusing in on a small range of performance levels that are sufficient to meet an application's deadlines.
Communications of the ACMProgramming parallel algorithms
449 Citations1996Guy E. Blelloch
This research on parallel algorithms has not only improved the general understanding of par-allelism but in several cases has led to improvements in sequential algorithms.
Encapsulation of parallelism in the Volcano query processing system
342 Citations1990Goetz Graefe
The reasons for not choosing the bracket model, the novel operator model, and details of Volcano's exchange operator that parallelizes all other operators are described, which makes implementation of parallel database algorithms significantly easier and more robust.
Accelerator
297 Citations2006David Tarditi, Sidd Puri +1 more
This work describes Accelerator, a system that uses data parallelism to program GPUs for general-purpose uses instead of C, and compares the performance of Accelerator versions of the benchmarks against hand-written pixel shaders.
The Imagine Stream Processor
230 Citations2003Ujval J. Kapasi, William J. Dally +3 more
The Imagine architecture and programming model is presented in the first half and the scalability of the Imagine architecture is explored in the second half to provide a scalable architecture that supports 48 ALUs on a single chip.
Highly available, fault-tolerant, parallel dataflows
205 Citations2004Mehul A. Shah, Joseph M. Hellerstein +1 more
A technique that masks failures in a cluster to provide high availability and fault-tolerance for long-running, parallelized dataflows that can use these dataflows to implement a variety of continuous query applications that require high-throughput, 24x7 operation.
Fault-tolerance in the Borealis distributed stream processing system
203 Citations2005Magdalena Bałazińska, Hari Balakrishnan +2 more
This work presents a replication-based approach to fault-tolerant distributed stream processing in the face of node failures, network failures, and network partitions, and aims to reduce the degree of inconsistency in the system while guaranteeing that available inputs capable of being processed are processed within a specified time threshold.
The CODE 2.0 graphical parallel programming language
127 Citations1992P. Newton, James C. Browne
This paper reports results obtained through experimental use of a prototype implementation of the CODE 2.0 system, a major conceptual advance over its predecessor systems in terms of the expressive power of the model of computation which is implemented.
Using Cohort-Scheduling to Enhance Server Performance
109 Citations2002James R. Larus, Michael Parkes
Paralex
65 Citations1992Özalp Babaoğlu, Lorenzo Alvisi +3 more
An overview of Paralex is given and the experiences with a prototype implementation are presented, aimed at exploring the extent to which the parallel application programmer can be liberated from the complexities of distributed systems.
Journal of Functional ProgrammingParallel and Distributed Haskells
64 Citations2002Phil Trinder, Hans Loidl +1 more
This paper addresses the question of why Haskell is a common functional computation language, and outlines parallel and distributed language concepts and classify Haskell extensions using them.
ACM SIGMOD RecordLoading databases using dataflow parallelism
41 Citations1994Tom Barclay, Robert Barnes +2 more
This paper describes the optimizer's cost-based hierarchical optimization strategy in some detail and preliminary measurements indicate that this design will give excellent scaleups.
Stream Computations Organized for Reconfigurable Execution (SCORE): Introduction and Tutorial
37 Citations2000Eylon Caspi, Michael Chu +4 more
This paper introduces SCORE (Stream Computation Organized for Reconfigurable Execution), a streambased compute model which virtualizes reconfigurable computing resources (compute, storage, and communication) by dividing a computation up into fixed-size “pages” and time-multiplexing the virtual pages on available physical hardware.
An overview of DB2 parallel edition
37 Citations1995Chaitanya Baru, Gilles Fecteau
An overview of the storage model, query optimization, runtime system, utilities, and performance of DB2 Parallel Edition is provided.
arXiv (Cornell University)Data Mining the SDSS SkyServer Database
30 Citations2002Jim Gray, Alexander S. Szalay +5 more
This paper built a database and interfaces to support both the query load and also a website for ad-hoc access, and reports on the query implementation and performance.
ACM Transactions on Computer SystemsRun-time adaptation in river
27 Citations2003Remzi H. Arpaci-Dusseau
This study presents the design, implementation, and evaluation of run-time adaptation within the River dataflow programming environment, comparing results from idealized models of system behavior, targeted simulations, and a prototype implementation.
IEEE ConcurrencyP-RIO: a modular parallel-programming environment
14 Citations1998Orlando Loques, Julius Leite +1 more
The authors discuss the development of their P-RIO environment, which offers a graphical programming tool, modular construction, high portability, a separate configuration language, and runtime support mechanisms for parallel programs.
RePEc: Research Papers in EconomicsScience in an Exponential World
14 Citations2008Alexander S. Szalay
The talk presents the challenges of this new paradigm for scientific publishing, and shows examples of how some disciplines are trying to cope with the data avalanche.
Design and implementation of parallel programs with LGDF2
8 Citations2003David C. DiNucci, Robert G. Babb
The authors present a method of parallel program design using large-grain data-flow constructs that approaches these problems head-on and efficient, automatic scheduling of these modules is possible for both shared- and distributed-memory processors.