Large-scale collaborative prediction using a nonparametric random effects model
Published 14 June 2009Open access
Kai Yu, John Lafferty, Shenghuo Zhu, Yihong Gong
Citations61
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A nonparametric model is introduced that allows multiple related regression tasks to take inputs from a common data space and treats such output responses as conditionally independent, given known predictors and appropriate unobserved random effects.
Abstract
A nonparametric model is introduced that allows multiple related regression tasks to take inputs from a common data space. Traditional transfer learning models can be inappropriate if the dependence among the outputs cannot be fully resolved by known input-specific and task-specific predictors. The proposed model treats such output responses as conditionally independent, given known predictors and appropriate unobserved random effects. The model is nonparametric in the sense that the dimensionality of random effects is not specified a priori but is instead determined from data. An approach to estimating the model is presented uses an EM algorithm that is efficient on a very large scale collaborative prediction problem. The obtained prediction accuracy is competitive with state-of-the-art results.
Keywords
Computer Science
Journal of the Royal Statistical Society Series B (Statistical Methodology)Probabilistic Principal Component Analysis
3,730 Citations1999Michael E. Tipping, Chris Bishop
It is demonstrated how the principal axes of a set of observed data vectors may be determined through maximum likelihood estimation of parameters in a latent variable model that is closely related to factor analysis.
Neural Information Processing SystemsProbabilistic Matrix Factorization
3,578 Citations2007Andriy Mnih, Ruslan Salakhutdinov
The Probabilistic Matrix Factorization (PMF) model is presented, which scales linearly with the number of observations and performs well on the large, sparse, and very imbalanced Netflix dataset and is extended to include an adaptive prior on the model parameters.
Restricted Boltzmann machines for collaborative filtering
1,882 Citations2007Ruslan Salakhutdinov, Andriy Mnih +1 more
This paper shows how a class of two-layer undirected graphical models, called Restricted Boltzmann Machines (RBM's), can be used to model tabular data, such as user's ratings of movies, and demonstrates that RBM's can be successfully applied to the Netflix data set.
Bayesian probabilistic matrix factorization using Markov chain Monte Carlo
1,453 Citations2008Ruslan Salakhutdinov, Andriy Mnih
This paper presents a fully Bayesian treatment of the Probabilistic Matrix Factorization (PMF) model in which model capacity is controlled automatically by integrating over all model parameters and hyperparameters and shows that Bayesian PMF models can be efficiently trained using Markov chain Monte Carlo methods by applying them to the Netflix dataset.
Matrix Variate Distributions
1,322 Citations2018Arjun K. Gupta, Daya K. Nagar
Probabilistic Matrix Factorization.
1,077 Citations2007Ruslan Salakhutdinov, Andriy Mnih
Fast maximum margin matrix factorization for collaborative prediction
968 Citations2005Jasson D. M. Rennie, Nathan Srebro
This work investigates a direct gradient-based optimization method for MMMF and finds that MMMf substantially outperforms all nine methods he tested and demonstrates it on large collaborative prediction problems.
Maximum-Margin Matrix Factorization
952 Citations2004Nathan Srebro, Jason D. M. Rennie +1 more
A novel approach to collaborative prediction is presented, using low-norm instead of low-rank factorizations, inspired by, and has strong connections to, large-margin linear discrimination.
Edinburgh Research ExplorerMulti-task Gaussian Process Prediction
851 Citations2007Edwin V. Bonilla, Kian Ming A. Chai +1 more
A model that learns a shared covariance function on input-dependent features and a "free-form" covariance matrix over tasks allows for good flexibility when modelling inter-task dependencies while avoiding the need for large amounts of data for training.
BiometrikaSome matrix-variate distribution theory: Notational considerations and a Bayesian application
421 Citations1981A. P. Dawid
Learning Gaussian processes from multiple tasks
388 Citations2005Kai Yu, Volker Tresp +1 more
This work considers the problem of multi-task learning, that is, learning multiple related functions, and presents a hierarchical Bayesian framework, that exploits the equivalence between parametric linear models and nonparametric Gaussian processes.
Journal of the American Statistical AssociationBilinear Mixed-Effects Models for Dyadic Data
343 Citations2005Peter D. Hoff
This article discusses the use of a symmetric multiplicative interaction effect to capture certain types of third-order dependence patterns often present in social networks and other dyadic datasets.
Learning to learn with the informative vector machine
333 Citations2004Neil D. Lawrence, John Platt
The multi-task IVM (MTIVM) saves computation by greedily selecting the most informative examples from the separate tasks and is shown to be more efficient than random sub-sampling on an artificial data-set and more effective than the traditional IVM in a speaker dependent phoneme recognition task.
Semiparametric Latent Factor Models
202 Citations2005Yee Whye Teh, Matthias Seeger +1 more
A semiparametric model for regression and classification problems involving multiple response variables makes use of a set of Gaussian processes to model the relationship to the inputs in a nonparametric fashion.
The MIT Press eBooksStochastic Relational Models for Discriminative Link Prediction
201 Citations2007Kai Yu, Wei Chu +3 more
A Gaussian process (GP) framework, stochastic relational models (SRM), for learning social, physical, and other relational phenomena where interactions between entities are observed is introduced and extensions of SRM to general relational learning tasks are discussed.
UCL Discovery (University College London)Variational Bayesian Approach to Movie Rating Prediction
166 Citations2007YJ Lim, YW Teh
A Bayesian approach is proposed to alleviate overfitting in SVD, where priors are introduced and all parameters are integrated out using variational inference, and it is shown that this gives significantly improved results over vanilla SVD.
Fast nonparametric matrix factorization for large-scale collaborative filtering
165 Citations2009Kai Yu, Shenghuo Zhu +2 more
These experiments on EachMovie and Netflix, the two largest public benchmarks to date, demonstrate that the nonparametric models make more accurate predictions of user ratings, and are computationally comparable or sometimes even faster in training, in comparison with previous state-of-the-art parametric matrix factorization models.
SZTAKI Publication Repository (Hungarian Academy of Sciences)Methods for large scale SVD with missing values
138 Citations2007Miklós Kurucz, András A. Benczúr +1 more
A systematic exploration of expectation maximization methods based both on the Lanczos algorithm and power iteration for recommenders based solely on low rank approximations of the rating matrix.
The BellKor 2008 Solution to the Netflix Prize
132 Citations2007Robert M. Bell, Yehuda Koren +1 more
The final solution (RMSE=0.8712) consists of blending 107 individual results, and the main approaches behind them are described, which can achieve slightly more accurate results than the newer one, at the expense of a significant increase in running time.
Collaborative ordinal regression
54 Citations2006Shipeng Yu, Kai Yu +2 more
Empirical studies show that the collaborative ordinal regression model outperforms the individual counterpart in preference learning applications and explores the dependency between ranking functions through a hierarchical Bayesian model and assign a common Gaussian Process prior to all individual functions.
Stochastic Relational Models for Large-scale Dyadic Data using MCMC
17 Citations2008Shenghuo Zhu, Kai Yu +1 more
This paper introduces a Markov chain Monte Carlo (MCMC) algorithm for equivalent models of SRMs in order to scale the computation to very large dyadic data sets and demonstrates superior scalability and predictive accuracy on a collaborative filtering problem.