Apache Hadoop YARN
Published 1 October 2013
Vinod Kumar Vavilapalli, Arun C. Murthy, Chris Douglas, Sharad Agarwal, Mahadev Konar, Robert Evans
Citations1,816
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
The design, development, and current state of deployment of the next generation of Hadoop's compute platform: YARN is summarized, which decouples the programming model from the resource management infrastructure, and delegates many scheduling functions to per-application components.
Abstract
The initial design of Apache Hadoop [1] was tightly focused on running massive, MapReduce jobs to process a web crawl. For increasingly diverse companies, Hadoop has become the data and computational agorá---the de facto place where data and computational resources are shared and accessed. This broad adoption and ubiquitous usage has stretched the initial design well beyond its intended target, exposing two key shortcomings: 1) tight coupling of a specific programming model with the resource management infrastructure, forcing developers to abuse the MapReduce programming model, and 2) centralized handling of jobs' control flow, which resulted in endless scalability concerns for the scheduler.
Keywords
Computer ScienceDecision Sciences
Communications of the ACMMapReduce
18,538 Citations2008Jay B. Dean, Sanjay Ghemawat
This presentation explains how the underlying runtime system automatically parallelizes the computation across large-scale clusters of machines, handles machine failures, and schedules inter-machine communication to make efficient use of the network and disks.
The Hadoop Distributed File System
4,780 Citations2010Konstantin V. Shvachko, Hairong Kuang +2 more
The architecture of HDFS is described and experience using HDFS to manage 25 petabytes of enterprise data at Yahoo! is reported on.
Spark: cluster computing with working sets
4,236 Citations2010Matei Zaharia, Mosharaf Chowdhury +3 more
Spark can outperform Hadoop by 10x in iterative machine learning jobs, and can be used to interactively query a 39 GB dataset with sub-second response time.
Pregel
3,534 Citations2010Grzegorz Malewicz, Matthew H. Austern +5 more
A model for processing large graphs that has been designed for efficient, scalable and fault-tolerant implementation on clusters of thousands of commodity computers, and its implied synchronicity makes reasoning about programs easier.
Dryad
2,460 Citations2007Michael Isard, Mihai Budiu +3 more
The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.
Concurrency and Computation Practice and ExperienceDistributed computing in practice: the Condor experience
1,896 Citations2005Douglas Thain, Todd Tannenbaum +1 more
The history and philosophy of the Condor project is provided and how it has interacted with other projects and evolved along with the field of distributed computing is described.
Pig latin
1,744 Citations2008Christopher Olston, Benjamin Reed +3 more
A new language called Pig Latin is described, designed to fit in a sweet spot between the declarative style of SQL, and the low-level, procedural style of map-reduce, which is an open-source, Apache-incubator project, and available for general use.
UC BerkeleyMesos: a platform for fine-grained resource sharing in the data center
1,593 Citations2011Benjamin Hindman, Andy Konwinski +6 more
The results show that Mesos can achieve near-optimal data locality when sharing the cluster among diverse frameworks, can scale to 50,000 (emulated) nodes, and is resilient to failures.
Hive - a petabyte scale data warehouse using Hadoop
925 Citations2010Ashish Thusoo, Joydeep Sen Sarma +7 more
Hive is presented, an open-source data warehousing solution built on top of Hadoop that supports queries expressed in a SQL-like declarative language - HiveQL, which are compiled into map-reduce jobs that are executed using Hadoops.
Towards predictable datacenter networks
746 Citations2011Hitesh Ballani, Paolo Costa +2 more
The case for extending the tenant-provider interface to explicitly account for the network is made, and the design of virtual network abstractions that capture the trade-off between the performance guarantees offered to tenants, their costs and the provider revenue are proposed.
Proceedings of the VLDB EndowmentSCOPE
731 Citations2008Ronnie Chaiken, B. Keith Jenkins +5 more
A new declarative and extensible scripting language, SCOPE (Structured Computations Optimized for Parallel Execution), targeted for this type of massive data analysis, designed for ease of use with no explicit parallelism, while being amenable to efficient parallel execution on large clusters.
Operating Systems Design and ImplementationDryadLINQ: a system for general-purpose distributed data-parallel computing using a high-level language
699 Citations2008Yuan Yu, Michael Isard +5 more
Addison-Wesley Longman Publishing Co., Inc. eBooksThe mythical man-month (anniversary ed.)
655 Citations1995Jr. Frederick P. Brooks
The added chapters contain a crisp condensation of all the propositions asserted in the original book, including Brooks' central argument in The Mythical Man-Month: that large programming projects suffer management problems different from small ones due to the division of labor.
Omega
638 Citations2013Malte Schwarzkopf, Andy Konwinski +2 more
This work presents a novel approach to address increasing scale and the need for rapid response to changing requirements using parallelism, shared state, and lock-free optimistic concurrency control to address monolithic cluster scheduler architectures.
Managing data transfers in computer clusters with orchestra
559 Citations2011Mosharaf Chowdhury, Matei Zaharia +3 more
This work proposes a global management architecture and a set of algorithms that improve the transfer times of common communication patterns, such as broadcast and shuffle, and allow scheduling policies at the transfer level,such as prioritizing a transfer over other transfers.
Lecture notes in computer scienceCore Algorithms of the Maui Scheduler
363 Citations2001David Jackson, Quinn Snell +1 more
This paper focuses on three areas of Maui scheduling, specifically, backfill, job prioritization, and fairshare and briefly discusses the goals of each component, the issues and corresponding design decisions, and the algorithms enabling the Maui policies.
ACM SIGCOMM Computer Communication ReviewManaging data transfers in computer clusters with orchestra
301 Citations2011Mosharaf Chowdhury, Matei Zaharia +3 more
A batch scheduler with high level components
209 Citations2005Nicolas Capit, Georges Da Costa +6 more
The design choices and the evaluation of a batch scheduler for large clusters, named OAR, which is based upon an original design that emphasizes on low software complexity by using high level tools is presented.
ACM SIGCOMM Computer Communication ReviewTowards predictable datacenter networks
207 Citations2011Hitesh Ballani, Paolo Costa +2 more
The Making of TPC-DS
158 Citations2010Raghunath Nambiar, Meikel Poess
arXiv (Cornell University)A batch scheduler with high level components
99 Citations2005Nicolas Capit, Georges Da Costa +6 more
Oozie
86 Citations2012Mohammad Samiul Islam, Angelo K. Huang +6 more
This paper finds that conventional workflow management tools lack at least one of these qualities of Scalability, Security, Multi-tenancy, and Operability, and therefore presents Apache Oozie, a workflow management system specialized for Hadoop.
Lecture notes in computer scienceDynamic Virtual Clustering with Xen and Moab
52 Citations2006Wesley Emeneker, Dave Jackson +2 more
The overhead of Xen-based DVC vs. native cluster performance is examined for workloads consisting of both serial and MPI-based parallel jobs.
IEEE Data(base) Engineering BulletinChallenges, Techniques and Directions in Building XSeek: an XML Search Engine.
45 Citations2009Brian F. Cooper, Eric Baldeschwieler +9 more
The vision and requirements for a comprehensive web-scale cloud infrastructure, emphasizing data storage and processing capabilities are discussed, as well as the components that will go into the cloud.
Proceedings of the VLDB EndowmentREEF
22 Citations2013Byung-Gon Chun, Tyson Condie +10 more
REEF, a framework that makes it easy to implement scalable, fault-tolerant runtime environments for a range of computational models, aims for composability of jobs across computational models, providing significant performance and usability gains, even with legacy code.