A Comparative Study on Parallel LDA Algorithms in MapReduce Framework
Lecture notes in computer sciencePublished 1 January 2015
Yang Gao, Zhenlong Sun, Yi Wang, Xiaosheng Liu, Jianfeng Yan, Jia Zeng
Citations9
SJR quartileQ2
SJR score0.35
SNIP0.55
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A novel multi-channel MapReduce framework is built to compare fairly three representative parallel LDA algorithms such as parallel variational Bayes (PVB), parallel Gibbs sampling (PGS) and parallel belief propagation (PBP), confirming that PGS yields the best application performance in search engine and online advertising system of Tencent.
Abstract
Although several parallel latent Dirichlet allocation (LDA) algorithms have been implemented to extract topic features from large-scale text data sets, very few studies compare their performance in real-world industrial applications. In this paper, we build a novel multi-channel MapReduce framework to compare fairly three representative parallel LDA algorithms such as parallel variational Bayes (PVB), parallel Gibbs sampling (PGS) and parallel belief propagation (PBP). Experimental results confirm that PGS yields the best application performance in search engine and online advertising system of Tencent, one of the biggest Internet companies in China, while PBP has the highest topic modeling accuracy. Moreover, PGS is more scalable in MapReduce framework than PVB and PBP because of its low memory usage and efficient sampling technique.
Keywords
Computer Science
Journal of Machine Learning ResearchLatent dirichlet allocation
27,049 Citations2003David M. Blei, Andrew Y. Ng +1 more
The Annals of Mathematical StatisticsA Stochastic Approximation Method
9,581 Citations1951Herbert Robbins, Sutton Monro
Proceedings of the National Academy of SciencesFinding scientific topics
6,017 Citations2004Thomas L. Griffiths, Mark Steyvers
A generative model for documents is described, introduced by Blei, Ng, and Jordan, and a Markov chain Monte Carlo algorithm is presented for inference in this model, which is used to analyze abstracts from PNAS by using Bayesian model selection to establish the number of topics.
Communications of the ACMProbabilistic topic models
5,549 Citations2012David M. Blei
Surveying a suite of algorithms that offer a solution to managing large document archives suggests they are well-suited to handle large amounts of data.
Spark: cluster computing with working sets
4,236 Citations2010Matei Zaharia, Mosharaf Chowdhury +3 more
Spark can outperform Hadoop by 10x in iterative machine learning jobs, and can be used to interactively query a 39 GB dataset with sub-second response time.
Online Learning for Latent Dirichlet Allocation
1,294 Citations2010Matthew D. Hoffman, Francis Bach +1 more
An online variational Bayes (VB) algorithm for Latent Dirichlet Allocation (LDA) based on online stochastic optimization with a natural gradient step is developed, which shows converges to a local optimum of the VB objective function.
Fast collapsed gibbs sampling for latent dirichlet allocation
567 Citations2008Ian R. Porteous, David Newman +4 more
A novel collapsed Gibbs sampling method for the widely used latent Dirichlet allocation (LDA) model, which can be as much as 8 times faster than the standard collapsed Gibbs sampler for LDA and results in significant speedups on real world text corpora.
A Collapsed Variational Bayesian Inference Algorithm for Latent Dirichlet Allocation
516 Citations2007Yee Whye Teh, David Newman +1 more
2009 IEEE Conference on Computer Vision and Pattern RecognitionSimultaneous image classification and annotation
515 Citations2009Chong Wang, David M. Blei +1 more
A new probabilistic model for jointly modeling the image, its class label, and its annotations is developed, which derives an approximate inference and estimation algorithms based on variational methods, as well as efficient approximations for classifying and annotating new images.
arXiv (Cornell University)On Smoothing and Inference for Topic Models
452 Citations2012Arthur Asuncion, Max Welling +2 more
Proceedings of the VLDB EndowmentAn architecture for parallel topic models
431 Citations2010Alexander J. Smola, Shravan Narayanamurthy
This paper describes a high performance sampling architecture for inference of latent topic models on a cluster of workstations and shows that this architecture is entirely general and that it can be extended easily to more sophisticated latent variable models such as n-grams and hierarchies.
Efficient methods for topic model inference on streaming document collections
425 Citations2009Limin Yao, David Mimno +1 more
Empirical results indicate that SparseLDA can be approximately 20 times faster than traditional LDA and provide twice the speedup of previously published fast sampling methods, while also using substantially less memory.
Scalable inference in latent variable models
261 Citations2012Amr Ahmed, Moahmed Aly +3 more
A scalable parallel framework for efficient inference in latent variable models over streaming web-scale data by introducing a novel delta-based aggregation system with a bandwidth-efficient communication protocol, schedule-aware out-of-core storage, and approximate forward sampling to rapidly incorporate new data.
Lecture notes in computer sciencePLDA: Parallel Latent Dirichlet Allocation for Large-Scale Applications
190 Citations2009Yi Wang, Hongjie Bai +3 more
PLDA smooths out storage and computation bottlenecks and provides fault recovery for lengthy distributed computations and can be applied to large, real-world applications and achieves good scalability.
ACM Transactions on Intelligent Systems and TechnologyPLDA+
161 Citations2011Zhiyuan Liu, Yuzhou Zhang +2 more
Data placement, pipeline processing, word bundling, and priority-based scheduling are proposed to improve scalability of LDA and significantly reduce the unparallelizable communication bottleneck and achieve good load balancing.
A fast parallel SGD for matrix factorization in shared memory systems
153 Citations2013Yong Zhuang, Wei-Sheng Chin +2 more
A fast parallel SGD method, FPSGD, for shared memory systems is developed by dramatically reducing the cache-miss rate and carefully addressing the load balance of threads, which is more efficient than state-of-the-art parallel algorithms for matrix factorization.
Neural Information Processing SystemsAsynchronous Distributed Learning of Topic Models
148 Citations2008Padhraic Smyth, Max Welling +1 more
It is demonstrated that the asynchronous algorithms presented are able to learn global topic models that are statistically as accurate as those learned by the standard LDA and HDP samplers, but with significant improvements in computation time and memory.
Mr. LDA
126 Citations2012Ke Zhai, Jordan Boyd‐Graber +2 more
This paper introduces a novel and flexible large scale topic modeling package in MapReduce (Mr. LDA), which uses variational inference, which easily fits into a distributed environment and is easily extensible.
Parallel Inference for Latent Dirichlet Allocation on Graphics Processing Units
104 Citations2009Feng Yan, Ningyi Xu +1 more
A novel data partitioning scheme is proposed that effectively reduces the memory cost of parallelizing two inference methods on GPUs for latent Dirichlet Allocation models, collapsed Gibbs sampling and collapsed variational Bayesian.
IEEE Transactions on Pattern Analysis and Machine IntelligenceLearning Topic Models by Belief Propagation
69 Citations2012Jia Zeng, William K. Cheung +1 more
The collapsed LDA is represented as a factor graph, which enables the classic loopy belief propagation (BP) algorithm for approximate inference and parameter estimation and is validated by encouraging experimental results on four large-scale document datasets.
arXiv (Cornell University)Towards Topic Modeling for Big Data
24 Citations2014Yi Wang, Xuemin Zhao +9 more