HSim: A MapReduce simulator in enabling Cloud Computing
Future Generation Computer SystemsPublished 27 May 2011
Yang Liu, Maozhen Li, Nasullah Khalid Alham, Suhel Hammoud
Citations69
SJR quartileQ1
SJR score1.55
SNIP2.23
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
HSim, a MapReduce simulator which builds on top of Hadoop, models a large number of parameters that can affect the behaviors of Map Reduce nodes, and thus it can be used to tune the performance of a Mapreduce cluster.
Abstract
MapReduce is an enabling technology in support of Cloud Computing. Hadoop which is a MapReduce implementation has been widely used in developing MapReduce applications. This paper presents HSim, a MapReduce simulator which builds on top of Hadoop. HSim models a large number of parameters that can affect the behaviors of MapReduce nodes, and thus it can be used to tune the performance of a MapReduce cluster. HSim is validated with both benchmark results and user customized MapReduce applications.
Keywords
Computer Science
Communications of the ACMMapReduce
18,538 Citations2008Jay B. Dean, Sanjay Ghemawat
This presentation explains how the underlying runtime system automatically parallelizes the computation across large-scale clusters of machines, handles machine failures, and schedules inter-machine communication to make efficient use of the network and disks.
A comparison of approaches to large-scale data analysis
1,072 Citations2009Andrew Pavlo, Erik K. Paulson +5 more
A benchmark consisting of a collection of tasks that are run on an open source version of MR as well as on two parallel DBMSs shows a dramatic performance difference between the two paradigms.
Mars
715 Citations2008Bingsheng He, Wenbin Fang +3 more
Mars hides the programming complexity of the GPU behind the simple and familiar MapReduce interface, and is up to 16 times faster than its CPU-based counterpart for six common web applications on a quad-core machine.
A simulation approach to evaluating design decisions in MapReduce setups
206 Citations2009Guanying Wang, Ali R. Butt +2 more
The resulting simulator, MRPerf, captures such aspects of MapReduce setups as node, rack and network configurations, disk parameters and performance, data layout and application I/O characteristics, among others, and uses this information to predict expected application performance and can serve as a tool for optimizing existing MapReduces setups as well as designing new ones.
Apress eBooksPro Hadoop
127 Citations2009Jason Venner
The ins and outs of MapReduce; how to structure a cluster, design, and implement the Hadoop file system; and how to build your first cloudcomputing tasks using Hadooper, from Apress.
Using realistic simulation for performance analysis of mapreduce setups
98 Citations2009Guanying Wang, Ali R. Butt +2 more
The design of an accurate MapReduce simulator, MRPerf, is presented, which can serve as a design tool for Map Reduce infrastructure, and as a planning tool for making Map reduce deployment far easier via reduction in the number of parameters that currently have to be hand-tuned using rules of thumb.
Phoenix
59 Citations2003Kenjiro Taura, Kenji Kaneda +2 more
Phoenix model is close enough to regular message passing to be a general programming model that facilitates porting many parallel applications/algorithms to more dynamic environments and is a useful programming abstraction and platform for emerging parallel applications that must be deployed across multiple LANs and/or shared clusters having dynamically varying resource conditions.
2010 Seventh International Conference on Fuzzy Systems and Knowledge DiscoveryA MapReduce based distributed LSI
16 Citations2010Yang Liu, Maozhen Li +3 more
A novel MapReduce based distributed LSI is proposed using Hadoop distributed computing architecture to implement K-means algorithm to cluster the documents and then using LSI on the clustered results to show a great improvement of LSI's performance in terms of speed.
2010 Seventh International Conference on Fuzzy Systems and Knowledge DiscoveryA distributed SVM for image annotation
16 Citations2010Nasullah Khalid Alham, Maozhen Li +3 more
This paper presents a MapReduce based distributed implementation of SMO using Hadoop, which shows the efficiency of the distributed SMO increases with the increase of the number of processors, and the training speed of distributedSMO with 12 Map task is about 11times higher than standalone SMO.
A distributed SVM for scalable image annotation
12 Citations2011Nasullah Khalid Alham, Maozhen Li +3 more
MRSVM, a distributed SVM algorithm for large scale image annotation which partitions the training data set into smaller subsets and train SVM in parallel using a cluster of computing nodes is presented.