Cumulon
Published 22 June 2013
Botong Huang, Shivnath Babu, Jun Yang
Citations79
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A suite of benchmarking, simulation, modeling, and search techniques are applied to support effective cost-based optimization over this rich space of deployment plans to help users rapidly develop and intelligently deploy matrix-based big-data analysis programs in the cloud.
Abstract
We present Cumulon, a system designed to help users rapidly develop and intelligently deploy matrix-based big-data analysis programs in the cloud. Cumulon features a flexible execution model and new operators especially suited for such workloads. We show how to implement Cumulon on top of Hadoop/HDFS while avoiding limitations of MapReduce, and demonstrate Cumulon's performance advantages over existing Hadoop-based systems for statistical data analysis. To support intelligent deployment in the cloud according to time/budget constraints, Cumulon goes beyond database-style optimization to make choices automatically on not only physical operators and their parameters, but also hardware provisioning and configuration settings. We apply a suite of benchmarking, simulation, modeling, and search techniques to support effective cost-based optimization over this rich space of deployment plans.
Keywords
Computer Science
Communications of the ACMMapReduce
18,538 Citations2008Jay B. Dean, Sanjay Ghemawat
This presentation explains how the underlying runtime system automatically parallelizes the computation across large-scale clusters of machines, handles machine failures, and schedules inter-machine communication to make efficient use of the network and disks.
Probabilistic latent semantic indexing
3,916 Citations1999Thomas Hofmann
Probabilistic Latent Semantic Indexing is a novel approach to automated document indexing which is based on a statistical latent class model for factor analysis of count data.
Communications of the ACMA bridging model for parallel computation
3,680 Citations1990Leslie G. Valiant
The bulk-synchronous parallel (BSP) model is introduced as a candidate for this role, and results quantifying its efficiency both in implementing high-level language features and algorithms, as well as in being implemented in hardware.
Pig latin
1,744 Citations2008Christopher Olston, Benjamin Reed +3 more
A new language called Pig Latin is described, designed to fit in a sweet spot between the declarative style of SQL, and the low-level, procedural style of map-reduce, which is an open-source, Apache-incubator project, and available for general use.
Proceedings of the VLDB EndowmentHive
1,539 Citations2009Ashish Thusoo, Joydeep Sen Sarma +7 more
Hadoop is a popular open-source map-reduce implementation which is being used as an alternative to store and process extremely large data sets on commodity hardware.
Proceedings of the VLDB EndowmentHaLoop
688 Citations2010Yingyi Bu, Bill Howe +2 more
HaLoop is presented, a modified version of the Hadoop MapReduce framework that is designed to serve iterative applications and dramatically improves their efficiency by making the task scheduler loop-aware and by adding various caching mechanisms.
PEGASUS: A Peta-Scale Graph Mining System Implementation and Observations
637 Citations2009U Kang, Charalampos E. Tsourakakis +1 more
This paper describes PEGASUS, an open source Peta Graph Mining library which performs typical graph mining tasks such as computing the diameter of the graph, computing the radius of each node and finding the connected components, and describes a very important primitive for PEGasUS, called GIM-V (Generalized Iterated Matrix-Vector multiplication).
Proceedings of the VLDB EndowmentMAD skills
467 Citations2009Jeffrey P. Cohen, Brian Dolan +3 more
This paper highlights the emerging practice of Magnetic, Agile, Deep (MAD) data analysis as a radical departure from traditional Enterprise Data Warehouses and Business Intelligence, and describes database design methodologies that support the agile working style of analysts in these settings.
SIAM Journal on Matrix Analysis and ApplicationsA Randomized Algorithm for Principal Component Analysis
433 Citations2009Vladimir Rokhlin, Arthur Szlam +1 more
This work describes an efficient algorithm for the low-rank approximation of matrices that produces accuracy that is very close to the best possible accuracy, for matrices of arbitrary sizes.
ARIA
410 Citations2011Abhishek Verma, Ludmila Cherkasova +1 more
This work designs a MapReduce performance model and implements a novel SLO-based scheduler in Hadoop that determines job ordering and the amount of resources to allocate for meeting the job deadlines and validate the approach using a set of realistic applications.
Proceedings of the VLDB EndowmentProfiling, what-if analysis, and cost-based optimization of MapReduce programs
370 Citations2011Herodotos Herodotou, Shivnath Babu
This work introduces, to its knowledge, the first Cost-based Optimizer for simple to arbitrarily complex MapReduce programs, which focuses on the optimization opportunities presented by the large space of configuration parameters for these programs.
Proceedings of the VLDB EndowmentThe MADlib analytics library
369 Citations2012Joseph M. Hellerstein, Christoper Ré +9 more
Overview of sciDB
363 Citations2010Paul Brown
This talk describes a set of motivating examples and uses them to explain the features of SciDB, explaining the novel storage manager, array data model, query language, and extensibility frameworks.
SystemML: Declarative machine learning on MapReduce
300 Citations2011Amol Ghoting, Rajasekar Krishnamurthy +6 more
This paper proposes SystemML in which ML algorithms are expressed in a higher-level language and are compiled and executed in a MapReduce environment and describes and empirically evaluate a number of optimization strategies for efficiently executing these algorithms on Hadoop, an open-source mapReduce implementation.
No one (cluster) size fits all
220 Citations2011Herodotos Herodotou, Fei Dong +1 more
The Elastisizer is introduced, a system to which users can express cluster sizing problems as queries in a declarative fashion and provides reliable answers to these queries using an automated technique that uses a mix of job profiling, estimation using black-box and white-box models, and simulation.
HAMA: An Efficient Matrix Computation with the MapReduce Framework
209 Citations2010Sangwon Seo, Edward J. Yoon +4 more
The state-of-the-art framework providing high-level matrix computation primitives with MapReduce is explored through the case study approach, and these primitives are demonstrated with different computation engines to show the performance and scalability.
Towards optimizing hadoop provisioning in the cloud
197 Citations2009Karthik Kambatla, Abhinav Pathak +1 more
This paperMotivates the importance of optimally provisioning a MapReduce job, and demonstrates that existing approaches can result in far from optimal provisioning, and presents a preliminary approach that improves Map Reduce provisioning by analyzing and comparing resource consumption of the application at hand with a database of similar resource consumption signatures of other applications.
Proceedings of the VLDB EndowmentAutomatic optimization for MapReduce programs
177 Citations2011Eaman Jahani, Michael Cafarella +1 more
Manimal is shown, which automatically analyzes MapReduce programs and applies appropriate data-aware optimizations, thereby requiring no additional help at all from the programmer, and that it yields speedups of up to 1,121% on previously-written Map Reduce programs.
Ricardo
166 Citations2010Sudipto Das, Yannis Sismanis +4 more
R Ricardo is part of the eXtreme Analytics Platform (XAP) project at the IBM Almaden Research Center, and rests on a decomposition of data-analysis algorithms into parts executed by the R statistical analysis system and parts handled by the Hadoop data management system.
DIMACS series in discrete mathematics and theoretical computer scienceA survey of out-of-core algorithms in numerical linear algebra
141 Citations1999Sivan Toledo
Algorithms that efficiently solve linear equations or compute eigenvalues even when the matrices involved are too large to fit in the main memory of the computer and must be stored on disks are surveyed.
SciHadoop
134 Citations2011Joe B. Buck, N. W. Watkins +5 more
This work describes the implementation of a Sci-Hadoop prototype for NetCDF data sets and quantifies the performance of five separate optimizations that address the following goals for several representative aggregate queries: reduce total data transfers, reduce remote reads, and reduce unnecessary reads.
Declarative Systems for Large-Scale Machine Learning.
33 Citations2012Vinayak Borkar, Yingyi Bu +6 more
This article argues for the use of recursive queries to program a variety of machine learning algorithms using database query optimization techniques to identify effective execution plans and the resulting runtime plans can be executed on a single unified data-parallel query processing engine.
arXiv (Cornell University)RIOT: I/O-Efficient Numerical Computing without SQL
30 Citations2009Yi Zhang, Herodotos Herodotou +1 more
Computing environment for the statistical analysis of large and complex data
17 Citations2010William S. Cleveland, Saptarshi Guha
RHIPE is a software system that integrates the R open source project for statistical computing and visualization with the Apache Hadoop Distributed File System (HDFS) and the Apache MapReduce software framework for the distributed processing of massive data sets across a cluster.
Journal of Parallel and Distributed ComputingTransparent runtime parallelization of the R scripting language
14 Citations2010Jiangtian Li, Xiaosong Ma +3 more
Meeting service level objectives of Pig programs
5 Citations2012Zhuoyao Zhang, Ludmila Cherkasova +2 more
A simple yet elegant performance model is developed that provides completion time estimates of a Pig program as a function of allocated resources and is used as a basis for solving the inverse resource provisioning problem for Pig programs.