User-Based Collaborative-Filtering Recommendation Algorithms on Hadoop
Published 1 January 2010
Zhi-Dan Zhao, Mingsheng Shang
Citations362
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This paper implements user-based CF algorithm on a cloud computing platform, namely Hadoop, to solve the scalability problem of CF.
Abstract
Collaborative Filtering (CF) algorithms are widely used in a lot of recommender systems, however, the computational complexity of CF is high thus hinder their use in large scale systems. In this paper, we implement user-based CF algorithm on a cloud computing platform, namely Hadoop, to solve the scalability problem of CF. Experimental results show that a simple method that partition users into groups according to two basic principles, i.e., tidy arrangement of mapper number to overcome the initiation of mapper and partition task equally such that all processors finish task at the same time, can achieve linear speedup.
Keywords
Computer Science
Communications of the ACMMapReduce
18,538 Citations2008Jay B. Dean, Sanjay Ghemawat
This presentation explains how the underlying runtime system automatically parallelizes the computation across large-scale clusters of machines, handles machine failures, and schedules inter-machine communication to make efficient use of the network and disks.
IEEE Transactions on Knowledge and Data EngineeringToward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions
10,115 Citations2005Gediminas Adomavičius, Alexander Tuzhilin
This paper presents an overview of the field of recommender systems and describes the current generation of recommendation methods that are usually classified into the following three main categories: content-based, collaborative, and hybrid recommendation approaches.
ACM Transactions on Information SystemsEvaluating collaborative filtering recommender systems
5,762 Citations2004Jonathan L. Herlocker, Joseph A. Konstan +2 more
The key decisions in evaluating collaborative filtering recommender systems are reviewed: the user tasks being evaluated, the types of analysis and datasets being used, the ways in which prediction quality is measured, the evaluation of prediction attributes other than quality, and the user-based evaluation of the system as a whole.
ACM SIGOPS Operating Systems ReviewThe Google file system
5,003 Citations2003Sanjay Ghemawat, Howard Gobioff +1 more
Dynamo
3,454 Citations2007Giuseppe DeCandia, Deniz Hastorun +7 more
D Dynamo is presented, a highly available key-value storage system that some of Amazon's core services use to provide an "always-on" experience and makes extensive use of object versioning and application-assisted conflict resolution in a manner that provides a novel interface for developers to use.
Dryad
2,460 Citations2007Michael Isard, Mihai Budiu +3 more
The Dryad execution engine handles all the difficult problems of creating a large distributed, concurrent application: scheduling the use of computers and their CPUs, recovering from communication or computer failures, and transporting data between vertices.
Google news personalization
1,511 Citations2007Abhinandan Das, Mayur Datar +2 more
This paper describes the approach to collaborative filtering for generating personalized recommendations for users of Google News using MinHash clustering, Probabilistic Latent Semantic Indexing, and covisitation counts, and combines recommendations from different algorithms using a linear model.
The Google file system
1,348 Citations2003Sanjay Ghemawat, Howard Gobioff +1 more
Evaluating MapReduce for Multi-core and Multiprocessor Systems
968 Citations2007Colby Ranger, R. Raghuraman +3 more
It is established that, given a careful implementation, MapReduce is a promising model for scalable performance on shared-memory systems with simple parallel code.
Chinese Physics LettersDiffusion-Based Recommendation in Collaborative Tagging Systems
24 Citations2009Shang Mingsheng, Zi‐Ke Zhang
Experimental results demonstrate that the usage of tag information can significantly improve the accuracy of personalized recommendations in a real-world dataset: Del.icio.us.
Optimizing data aggregation for cluster-based internet services
24 Citations2003Lingkun Chu, Hong Tang +2 more
The design and implementation of a programming primitive -- Data Aggregation Call (DAC) -- to exploit partition parallelism for cluster-based Internet services and the effectiveness of the proposed optimization techniques for reducing response time, improving throughput, and gracefully handling server unresponsiveness are validated.
Physica A Statistical Mechanics and its ApplicationsCollaborative filtering based on multi-channel diffusion
22 Citations2009Mingsheng Shang, Ci-Hang Jin +2 more
By applying a diffusion process, a new index to quantify the similarity between two users in a user–object bipartite graph is proposed, which is further demonstrated to be more accurate than the classical Pearson correlation coefficient under the standard collaborative filtering framework.
Kahn Process Networks are a Flexible Alternative to MapReduce
16 Citations2009eljko Vrba, Pål Halvorsen +2 more
Kahn process networks are proposed, which are based on message-passing and shared-nothing model, as a simple and flexible tool for modeling parallel applications and are argued that they are more flexible than MapReduce, which is widely recognized for its efficiency and simplicity.