MLlib: Machine Learning in Apache Spark

arXiv (Cornell University)Published 26 May 2015Open access

Xiangrui Meng, Joseph K. Bradley, Burak Yavuz, Evan Sparks, Shivaram Venkataraman, Davies Liu

DOI10.48550/arxiv.1505.06807

Citations961

Abstract

Apache Spark is a popular open-source platform for large-scale data processing that is well-suited for iterative machine learning tasks. In this paper we present MLlib, Spark's open-source distributed machine learning library. MLlib provides efficient functionality for a wide range of learning settings and includes several underlying statistical, optimization, and linear algebra primitives. Shipped with Spark, MLlib supports several languages and provides a high-level API that leverages Spark's rich ecosystem to simplify the development of end-to-end machine learning pipelines. MLlib has experienced a rapid growth due to its vibrant open-source community of over 140 contributors, and includes extensive documentation to support further growth and to let users quickly get up to speed.

Keywords

Computer Science

ComputerMatrix Factorization Techniques for Recommender Systems

11,675 Citations2009Yehuda Koren, Robert Bell +1 more

As the Netflix Prize competition has demonstrated, matrix factorization models are superior to classic nearest neighbor techniques for producing product recommendations, allowing the incorporation of additional information such as implicit feedback, temporal effects, and confidence levels.

Resilient distributed datasets: a fault-tolerant abstraction for in-memory cluster computing

3,577 Citations2012Matei Zaharia, Mosharaf Chowdhury +7 more

Resilient Distributed Datasets is presented, a distributed memory abstraction that lets programmers perform in-memory computations on large clusters in a fault-tolerant manner and is implemented in a system called Spark, which is evaluated through a variety of user applications and benchmarks.

arXiv (Cornell University)API design for machine learning software: experiences from the scikit-learn project

1,791 Citations2013Lars Buitinck, Gilles Louppe +13 more

View PDF

Hidden factors and hidden topics

1,567 Citations2013Julian McAuley, Jure Leskovec

This paper aims to combine latent rating dimensions (such as those of latent-factor recommender systems) with latent review topics ( such as those learned by topic models like LDA), which more accurately predicts product ratings by harnessing the information present in review text.

Discretized streams

964 Citations2013Matei Zaharia, Tathagata Das +4 more

D-Streams enable a parallel recovery mechanism that improves efficiency over traditional replication and backup schemes, and tolerates stragglers, and can easily be composed with batch and interactive query models like MapReduce, enabling rich applications that combine these modes.

View PDF

MLbase: A Distributed Machine-learning System

294 Citations2013Tim Kraska, Ameet Talwalkar +4 more

This work presents the vision for MLbase, a novel system harnessing the power of machine learning for both end-users and ML researchers, which provides a simple declarative way to specify ML tasks and a novel optimizer to select and dynamically adapt the choice of learning algorithm.

Proceedings of the VLDB EndowmentPLANET

268 Citations2009Biswanath Panda, Joshua S. Herbach +2 more

This paper describes PLANET: a scalable distributed framework for learning tree models over large datasets, and shows how this framework supports scalable construction of classification and regression trees, as well as ensembles of such models.

The R JournalPMML: An Open Standard for Sharing Models

159 Citations2009Alex Guazzelli, M. Zeller +2 more

The PMML package exports a variety of predictive and descriptive models from R to the Predictive Model Markup Language, an XML-based language that allows for the interchange of models among different tools and environments, mostly avoiding proprietary issues and incompatibilities.

View PDF

arXiv (Cornell University)TuPAQ: An Efficient Planner for Large-scale Predictive Analytic Queries

18 Citations2015Evan Sparks, Ameet Talwalkar +3 more

TuPAQ, a component of the MLbase system, is proposed, which solves the PAQ planning problem with comparable quality to exhaustive strategies but an order of magnitude more efficiently than the standard baseline approach, and can scale to models trained on terabytes of data across hundreds of machines.

View PDF