Twister
Published 21 June 2010
Jaliya Ekanayake, Hui Li, Bingjing Zhang, Thilina Gunarathne, Seunghee Bae, Judy Qiu
Citations777
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper presents the programming model and the architecture of Twister an enhanced MapReduce runtime that supports iterative Map Reduce computations efficiently and shows performance comparisons of Twisters with other similar runtimes such as Hadoop and DryadLINQ for large scale data parallel applications.
Abstract
MapReduce programming model has simplified the implementation of many data parallel applications. The simplicity of the programming model and the quality of services provided by many implementations of MapReduce attract a lot of enthusiasm among distributed computing communities. From the years of experience in applying MapReduce to various scientific applications we identified a set of extensions to the programming model and improvements to its architecture that will expand the applicability of MapReduce to more classes of applications. In this paper, we present the programming model and the architecture of Twister an enhanced MapReduce runtime that supports iterative MapReduce computations efficiently. We also show performance comparisons of Twister with other similar runtimes such as Hadoop and DryadLINQ for large scale data parallel applications.
Keywords
Computer Science
Munich Personal RePEc Archive (Ludwig Maximilian University of Munich)Some methods for classification and analysis of multivariate observations
22,787 Citations1967James B. MacQueen
Communications of the ACMMapReduce
18,538 Citations2008Jay B. Dean, Sanjay Ghemawat
This presentation explains how the underlying runtime system automatically parallelizes the computation across large-scale clusters of machines, handles machine failures, and schedules inter-machine communication to make efficient use of the network and disks.
Computer Networks and ISDN SystemsThe anatomy of a large-scale hypertextual Web search engine
15,828 Citations1998Sergey Brin, Lawrence M. Page
This paper provides an in-depth description of Google, a prototype of a large-scale search engine which makes heavy use of the structure present in hypertext and looks at the problem of how to effectively deal with uncontrolled hypertext collections where anyone can publish anything they want.
PsychometrikaMultidimensional Scaling by Optimizing Goodness of Fit to a Nonmetric Hypothesis
7,398 Citations1964Joseph B. Kruskal
The fundamental hypothesis is that dissimilarities and distances are monotonically related, and a quantitative, intuitively satisfying measure of goodness of fit is defined to this hypothesis.
ACM SIGOPS Operating Systems ReviewThe Google file system
5,003 Citations2003Sanjay Ghemawat, Howard Gobioff +1 more
Dryad
2,460 Citations2007Michael Isard, Mihai Budiu +3 more
The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.
Journal of Molecular BiologyAn improved algorithm for matching biological sequences
1,739 Citations1982Osamu Gotoh
The algorithm of Waterman et al. (1976) for matching biological sequences was modified under some limitations to be accomplished in essentially MN steps, instead of the M 2 N steps necessary in the original algorithm.
The MIT Press eBooksMap-Reduce for Machine Learning on Multicore
1,253 Citations2007Cheng-Tao Chu, Sang Kyun Kim +5 more
This work shows that algorithms that fit the Statistical Query model can be written in a certain "summation form," which allows them to be easily parallelized on multicore computers and shows basically linear speedup with an increasing number of processors.
Trends in GeneticsRepbase Update: a database and an electronic journal of repetitive elements
1,022 Citations2000Jerzy Jurka
This work was supported in part by the National Library of Medicine, grant 5 P41 LM06252-3.
Evaluating MapReduce for Multi-core and Multiprocessor Systems
968 Citations2007Colby Ranger, R. Raghuraman +3 more
It is established that, given a careful implementation, MapReduce is a promising model for scalable performance on shared-memory systems with simple parallel code.
PsychometrikaNonmetric Individual Differences Multidimensional Scaling: An Alternating Least Squares Method with Optimal Scaling Features
960 Citations1977Yoshio Takane, Forrest W. Young +1 more
Operating Systems Design and ImplementationDryadLINQ: a system for general-purpose distributed data-parallel computing using a high-level language
699 Citations2008Yuan Yu, Michael Isard +5 more
Pattern Recognition LettersA deterministic annealing approach to clustering
384 Citations1990Kenneth Rose, E. Gurewitz +1 more
It is shown that as the temperature approaches zero, the algorithm becomes the basic ISODATA algorithm and the method is independent of the initial choice of cluster means.
MapReduce for Data Intensive Scientific Analyses
378 Citations2008Jaliya Ekanayake, Shrideep Pallickara +1 more
This paper presents CGL-MapReduce, a streaming-based MapReduce implementation and compares its performance with Hadoop, and presents the experience in applying the MapReduced technique for two scientific data analyses: high energy physics data analyses; and K-means clustering.
Swift: Fast, Reliable, Loosely Coupled Parallel Computation
352 Citations2007Yong Zhao, Mihael Hategan +7 more
Swift adopts and adapts ideas first explored in the GriPhyN virtual data system, improving on that system in many regards and describes application experiences and performance experiments that quantify the cost of Swift operations.
eScholarship (California Digital Library)Applications of Convex Analysis to Multidimensional Scaling
328 Citations2000Jan de Leeuw
An algorithm for metric and nonmetric multidimensional scaling that is very similar to the C-matrix algorithm of Guttman is discussed, extended to cover general Minkovski metrics.
Exploiting the Block Structure of the Web for Computing PageRank
316 Citations2003Sepandar Kamvar, Taher H. Haveliwala +2 more
Lecture notes of the Institute for Computer Sciences, Social Informatics and Telecommunications EngineeringHigh Performance Parallel Computing with Clouds and Cloud Technologies
187 Citations2010Jaliya Ekanayake, Geoffrey Fox
This paper first discusses large scale data analysis using different MapReduce implementations and then, a performance analysis of high performance parallel applications on virtualized resources is presented.
Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering SciencesSector and Sphere: the design and implementation of a high-performance data cloud
158 Citations2009Yunhong Gu, Robert L. Grossman
The design and implementation of the Sector storage cloud and the Sphere compute cloud are described and some experimental studies comparing Sector/Sphere and Hadoop using the Terasort benchmark are described.
IEEE Transactions on Parallel and Distributed SystemsAll-Pairs: An Abstraction for Data-Intensive Computing on Campus Grids
73 Citations2009Christopher Moretti, Hoang Bui +4 more
This work argues that campus grids should provide end users with high-level abstractions that allow for the easy expression and efficient execution of data-intensive workloads and presents one example of an abstraction-All-Pairs-that fits the needs of several applications in biometrics, bioinformatics, and data mining.
Distributed PageRank computation based on iterative aggregation-disaggregation methods
64 Citations2005Yangbo Zhu, Shaozhi Ye +1 more
A distributed PageRank computation algorithm based on iterative aggregation-disaggregation (IAD) method with Block Jacobi smoothing with major advantages for PageRank on large scale Web graphs is proposed.
DryadLINQ for Scientific Analyses
42 Citations2009Jaliya Ekanayake, Thilina Gunarathne +5 more
The experience in applying Dryad LINQ for a series of scientific data analysis applications is presented, their mapping to the DryadLINQ programming model is identified, and their performances with Hadoop implementations of the same applications are compared.
Advances in parallel computingParallel Data Mining from Multicore to Cloudy Grids
37 Citations2009Geoffrey Fox, Seunghee Bae +3 more
A suite of data mining tools that cover clustering, information retrieval and the mapping of high dimensional data to low dimensions for visualization are described, stressing that data analysis/mining of large datasets can be a supercomputer application.
BioinformaticsData Intensive Computing for Bioinformatics
15 Citations2013Judy Qiu, Jaliya Ekanayake +10 more