MapReduce as a programming model for association rules algorithm on Hadoop

Published 1 June 2010

Xin Yue Yang, Zhen Liu, Yan Fu

DOI10.1109/icicis.2010.5534718

Citations122

TL;DR

The improved Apriori algorithm based on MapReduce mode is described, which can handle massive datasets with a large number of nodes on Hadoop platform.

Abstract

As association rules widely used, it needs to study many problems, one of which is the generally larger and multi-dimensional datasets, and the rapid growth of the mount of data. Single-processor's memory and CPU resources are very limited, which makes the algorithm performance inefficient. Recently the development of network and distributed technology makes cloud computing a reality in the implementation of association rules algorithm. In this paper we describe the improved Apriori algorithm based on MapReduce mode, which can handle massive datasets with a large number of nodes on Hadoop platform.

Keywords

Computer Science

Communications of the ACMMapReduce

18,538 Citations2008Jay B. Dean, Sanjay Ghemawat

This presentation explains how the underlying runtime system automatically parallelizes the computation across large-scale clusters of machines, handles machine failures, and schedules inter-machine communication to make efficient use of the network and disks.

View PDF

Fast algorithms for mining association rules

10,739 Citations1998Rakesh Agrawal, Ramakrishnan Srikant

UC BerkeleyImproving MapReduce performance in heterogeneous environments

1,619 Citations2008Matei Zaharia, Andy Konwinski +3 more

A new scheduling algorithm, Longest Approximate Time to End (LATE), that is highly robust to heterogeneity and can improve Hadoop response times by a factor of 2 in clusters of 200 virtual machines on EC2.

The MIT Press eBooksMap-Reduce for Machine Learning on Multicore

1,253 Citations2007Cheng-Tao Chu, Sang Kyun Kim +5 more

This work shows that algorithms that fit the Statistical Query model can be written in a certain "summation form," which allows them to be easily parallelized on multicore computers and shows basically linear speedup with an increasing number of processors.

MapReduce for Data Intensive Scientific Analyses

378 Citations2008Jaliya Ekanayake, Shrideep Pallickara +1 more

This paper presents CGL-MapReduce, a streaming-based MapReduce implementation and compares its performance with Hadoop, and presents the experience in applying the MapReduced technique for two scientific data analyses: high energy physics data analyses; and K-means clustering.

Pairwise document similarity in large collections with MapReduce

204 Citations2008Tamer Elsayed, Jimmy Lin +1 more

This paper presents a MapReduce algorithm for computing pairwise document similarity in large document collections that exhibits linear growth in running time and space in terms of the number of documents.

View PDF

MRPGA: An Extension of MapReduce for Parallelizing Genetic Algorithms

116 Citations2008Chao Jin, Christian Vecchiola +1 more

This paper describes the design and implementation of the extended MapReduce model featuring a hierarchical reduction phase, called MRPGA (MapReduce for parallel GAs), which can automatically parallelize GAs on a .NET-based enterprise grid system in detail.

IEEE Distributed Systems OnlineODAM: an optimized distributed association rule mining algorithm

101 Citations2004Mafruz Zaman Ashrafi, David Taniar +1 more

This work has developed a distributed algorithm, called optimized distributed association mining, for geographically distributed data sets, called ODAM, which generates support counts of candidate itemsets quicker than the other DARM algorithms and reduces the size of average transactions, data set, and message exchanges.

Map-reduce as a Programming Model for Custom Computing Machines

80 Citations2008Jackson H.C. Yeung, C.C. Tsang +5 more

This work presents an implementation of a map-reduce library supporting parallel field programmable gate arrays (FPGAs) and graphics processing units (GPUs) and describes the experience in developing a number of benchmark problems in signal processing, Monte Carlo simulation and scientific computing.

Communications in computer and information scienceAn Improved Algorithm of Apriori

20 Citations2009Binhua Liao

This paper puts forward a kind of improved algorithm after analyzing the classical Apriori algorithm, where the unnecessary data are deleted in time, and the joining and pruning steps become simple.