Boosting energy efficiency with mirrored data block replication policy and energy scheduler
ACM SIGOPS Operating Systems ReviewPublished 23 July 2013
Sara Arbab Yazd, S. Venkatesan, Neeraj Mittal
Citations12
SJR quartileQ3
SJR score0.32
SNIP0.48
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A scheduling algorithm is introduced, which takes into account energy efficiency in addition to fairness and data locality properties, and computer simulations of the proposed method suggest its superiority over Hadoop's standard settings.
Abstract
Energy efficiency is one of the major challenges in big datacenters. To facilitate processing of large data sets in a distributed fashion, the MapReduce programming model is employed in these datacenters. Hadoop is an open-source implementation of MapReduce which contains a distributed file system. Hadoop Distributed File System provides a data block replication scheme to preserve reliability and data availability. The distribution of the data block replicas over the nodes is performed randomly by meeting some constraints (e.g., preventing storage of two replicas of a data block on a single node). This study makes use of flexibility in the data block placement policy to increase energy efficiency in datacenters. Furthermore, inspired by Zaharia et al.'s delay scheduling algorithm, a scheduling algorithm is introduced, which takes into account energy efficiency in addition to fairness and data locality properties. Computer simulations of the proposed method suggest its superiority over Hadoop's standard settings.
Keywords
Computer Science
Communications of the ACMMapReduce
18,538 Citations2008Jay B. Dean, Sanjay Ghemawat
This presentation explains how the underlying runtime system automatically parallelizes the computation across large-scale clusters of machines, handles machine failures, and schedules inter-machine communication to make efficient use of the network and disks.
ACM SIGOPS Operating Systems ReviewXen and the art of virtualization
5,948 Citations2003Paul Barham, Boris Dragovic +7 more
The Hadoop Distributed File System
4,780 Citations2010Konstantin V. Shvachko, Hairong Kuang +2 more
The architecture of HDFS is described and experience using HDFS to manage 25 petabytes of enterprise data at Yahoo! is reported on.
ComputerThe Case for Energy-Proportional Computing
2,479 Citations2007Luiz André Barroso, Urs Hölzle
Energy-proportional designs would enable large energy savings in servers, potentially doubling their efficiency in real-life use, particularly the memory and disk subsystems.
Dryad
2,460 Citations2007Michael Isard, Mihai Budiu +3 more
The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.
Xen and the art of virtualization
2,075 Citations2003Paul Barham, Boris Dragovic +7 more
Xen, an x86 virtual machine monitor which allows multiple commodity operating systems to share conventional hardware in a safe and resource managed fashion, but without sacrificing either performance or functionality, considerably outperform competing commercial and freely available solutions.
Delay scheduling
1,380 Citations2010Matei Zaharia, Dhruba Borthakur +4 more
This work proposes a simple algorithm called delay scheduling, which achieves nearly optimal data locality in a variety of workloads and can increase throughput by up to 2x while preserving fairness.
Energy aware consolidation for cloud computing
778 Citations2008Shekhar Srikantaiah, Aman Kansal +1 more
Reducing network energy consumption via sleeping and rate-adaptation
649 Citations2008Sergiu Nedevschi, Lucian Popa +3 more
It is shown that even simple schemes for sleeping or rate-adaptation can offer substantial savings and that both sleeping and rate adaptation are valuable depending (primarily) on the power profile of network equipment and the utilization of the network itself.
Energy-aware server provisioning and load dispatching for connection-intensive internet services
645 Citations2008Chen Gong, Wenbo He +5 more
This paper characterize unique properties, performance, and power models of connection servers, based on a real data trace collected from the deployed Windows Live Messenger, and shows that these algorithms can save a significant amount of energy without sacrificing user experiences.
Energy-Efficient Management of Data Center Resources for Cloud Computing: A Vision, Architectural Elements, and Open Challenges
444 Citations2010Rajkumar Buyya, Anton Beloglazov +1 more
The results demonstrate that Cloud computing model has immense potential as it offers significant performance gains as regards to response time and cost saving under dynamic workload scenarios.
ACM SIGOPS Operating Systems ReviewOn the energy (in)efficiency of Hadoop clusters
339 Citations2010Jacob Leverich, Christos Kozyrakis
It is found that running Hadoop clusters in fractional configurations can save between 9% and 50% of energy consumption, and that there is a tradeoff between performance energy consumption.
Job Scheduling for Multi-User MapReduce Clusters
324 Citations2009Matei Zaharia, Dhruba Borthakur +4 more
Two simple techniques, delay scheduling and copy-compute splitting, are developed which improve throughput and response times in multi-user MapReduce workloads by factors of 2 to 10 and can also raise throughput in a single-user, FIFO workload by a factor of 2.
arXiv (Cornell University)Energy-Efficient Management of Data Center Resources for Cloud\n Computing: A Vision, Architectural Elements, and Open Challenges
237 Citations2010Rajkumar Buyya, Anton Beloglazov +1 more
Minimizing data center cooling and server power costs
220 Citations2009Ehsan Pakbaznia, Massoud Pedram
The resulting optimization problem is formulated as an Integer Linear Programming problem and a heuristic algorithm that solves it in polynomial time is presented, showing an average of 13% power saving for different data center utilization rates.
Proceedings of the VLDB EndowmentEnergy management for MapReduce clusters
217 Citations2010Willis Lang, Jignesh M. Patel
This paper develops a framework for systematically considering various MapReduce node power down strategies, and their impact on the overall energy consumption and workload response time and proposes the All-In Strategy (AIS), which is often the right energy saving strategy.
Purlieus
196 Citations2011Balaji Palanisamy, Aameek Singh +2 more
Purlieus provisions virtual Map Reduce clusters in a locality-aware manner enabling MapReduce virtual machines (VMs) access to input data and importantly, intermediate data from local or close-by physical machines.
I-THERM :/I-THERM - Intersociety Conference on Thermal Phenomena in Electronic SystemsThe effect of data center temperature on energy efficiency
192 Citations2008Michael K. Patterson
Performance EvaluationServer farms with setup costs
158 Citations2010Anshul Gandhi, Mor Harchol‐Balter +1 more
File and Storage TechnologiesMore than an interface: scsi vs. ata
138 Citations2003Dave Anderson, Jim Dykes +1 more
This paper attempts to clarify the differences between SCSI disc drives and IDE (ATA) disc drives by illuminating some of these design choices and their consequences on final device characteristics to allow the community to build better storage systems with better knowledge of the trade-offs being made and the performance characteristics that result.
Matchmaking: A New MapReduce Scheduling Technique
118 Citations2011Chen He, Ying Lü +1 more
Experimental results show that this new MapReduce scheduling technique often leads to the highest data locality rate and the lowest response time for map tasks, and does not require an intricate parameter tuning process.
IEEE Communications MagazineOn energy efficiency for enterprise and data center networks
77 Citations2011Priya Mahadevan, Sujata Banerjee +3 more
The challenges relating to life cycle energy management of network devices are described, a sustainability analysis of these devices is presented, and techniques to significantly reduce network operational power are developed.
Application-Aware Power Management
62 Citations2006Karthick Rajamani, Heather Hanson +3 more
This paper combines continuous monitoring of critical workload indicators, online power and performance model usage and timely p-state control to realize application-aware power management and presents two new power management solutions enabled by the methodology.
Journal of Internet Services and ApplicationsOn Cloud computational models and the heterogeneity challenge
49 Citations2011Raouf Boutaba, Cheng Lü +1 more
It is shown that dealing with the heterogeneity of workloads and machine capabilities is a key challenge in today’s cloud environment and that although advances are made that partially address some of the outlined challenges, there are even more open challenges yet to be explored, and this topic at large is ripe for scientific contributions.
Mathematical Problems in EngineeringEnergy‐Efficient Multi‐Job Scheduling Model for Cloud Computing and Its Genetic Algorithm
43 Citations2012Xiaoli Wang, Yuping Wang +1 more
A new energy-efficient multi-job scheduling model based on Google’s massive data processing framework is proposed and a local search operator is introduced in order to accelerate the convergent speed of the algorithm and enhance its searching ability.
IEEE International Conference on High Performance Computing, Data, and AnalyticsT: a data-centric cooling energy costs reduction approach for big data analytics cloud
34 Citations2012Rini T. Kaushik, Klara Nahrstedt
Migration energy-aware workload consolidation in enterprise clouds
21 Citations2012Mohammad Motaher Hossain, Jen-Cheng Huang +1 more
New schemes to pack applications that targets a joint optimization of energy and performance overhead of VM migrations are devised that can improve energy savings up to 23% compared to state of the art power-aware workload consolidation strategy.