A Unifying View of Sparse Approximate Gaussian Process Regression
Journal of Machine Learning ResearchPublished 1 December 2005
Joaquin Quiñonero-Candela, Carl Edward Rasmussen
Citations1,759
SJR quartileQ1
SJR score2.02
SNIP3.07
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
A new unifying view, including all existing proper probabilistic sparse approximations for Gaussian process regression, relies on expressing the effective prior which the methods are using, and highlights the relationship between existing methods.
Abstract
We provide a new unifying view, including all existing proper probabilistic sparse approximations for Gaussian process regression. Our approach relies on expressing the effective prior which the methods are using. This allows new insights to be gained, and highlights the relationship between existing methods. It also allows for a clear theoretically justified ranking of the closeness of the known approximations to the corresponding full GPs. Finally we point directly to designs of new better sparse approximations, combining the best of the existing strategies, within attractive computational constraints.
Keywords
Computer ScienceEngineering
The Nature of Statistical Learning Theory
39,279 Citations1995Vladimir Vapnik
Machine LearningSupport-Vector Networks
33,035 Citations1995Corinna Cortes, Vladimir Vapnik
High generalization ability of support-vector networks utilizing polynomial input transformations is demonstrated and the performance of the support- vector network is compared to various classical learning algorithms that all took part in a benchmark study of Optical Character Recognition.
The MIT Press eBooksGaussian Processes for Machine Learning
10,409 Citations2005Carl Edward Rasmussen, Christopher K. I. Williams
The book provides a long-needed, systematic and unified treatment of theoretical and practical aspects of GPs in machine learning, targeted at researchers and students in machine learning and applied statistics.
Applied Physics Letters10.1162/15324430152748236
1,869 Citations2000
It is demonstrated that by exploiting a probabilistic Bayesian learning framework, the 'relevance vector machine' (RVM) can derive accurate prediction models which typically utilise dramatically fewer basis functions than a comparable SVM while offering a number of additional advantages.
Using the Nyström Method to Speed Up Kernel Machines
1,522 Citations2000Christopher K. I. Williams, Matthias Seeger
It is shown that an approximation to the eigendecomposition of the Gram matrix can be computed by the Nystrom method (which is used for the numerical solution of eigenproblems) and the computational complexity of a predictor using this approximation is O(m2n).
Sparse Gaussian Processes using Pseudo-inputs
1,329 Citations2005Edward Snelson, Zoubin Ghahramani
It is shown that this new Gaussian process (GP) regression model can match full GP performance with small M, i.e. very sparse solutions, and it significantly outperforms other approaches in this regime.
Aston Publications Explorer (Aston University)Gaussian Processes for Regression
1,142 Citations1995Christopher K. I. Williams, Carl Edward Rasmussen
This paper investigates the use of Gaussian process priors over functions, which permit the predictive Bayesian analysis for fixed values of hyperparameters to be carried out exactly using matrix operations.
Journal of the Royal Statistical Society Series B (Statistical Methodology)Some Aspects of the Spline Smoothing Approach to Non-Parametric Regression Curve Fitting
1,129 Citations1985B. W. Silverman
The topics and examples discussed in this paper are intended to promote the understanding and extend the practicability of the spline smoothing methodology.
Neural ComputationSparse On-Line Gaussian Processes
763 Citations2002Lehel Csató, Manfred Opper
An approach for sparse representations of gaussian process (GP) models (which are Bayesian types of kernel machines) in order to overcome their limitations for large data sets is developed based on a combination of a Bayesian on-line algorithm and a sequential construction of a relevant subsample of data that fully specifies the prediction of the GP model.
Neural ComputationA Bayesian Committee Machine
418 Citations2000Volker Tresp
It is found that the performance of the BCM improves if several test points are queried at the same time and is optimal if the number of test points is at least as large as the degrees of freedom of the estimator.
Fast Forward Selection to Speed Up Sparse Gaussian Process Regression
389 Citations2003Matthias Seeger, Christopher K. I. Williams +1 more
A method for the sparse greedy approximation of Bayesian Gaussian process regression, featuring a novel heuristic for very fast forward selection, which leads to a sufficiently stable approximation of the log marginal likelihood of the training data, which can be optimised to adjust a large number of hyperparameters automatically.
Sparse Greedy Gaussian Process Regression
325 Citations2000Alex Smola, Peter L. Bartlett
A simple sparse greedy technique to approximate the maximum a posteriori estimate of Gaussian Processes with much improved scaling behaviour in the sample size m, and shows applications to large scale problems.
Journal of Machine Learning ResearchAssessing Approximate Inference for Binary Gaussian Process Classification
247 Citations2005Malte Kuß, Carl Edward Rasmussen
This work reviews and compares Laplace's method and Expectation Propagation for approximate Bayesian inference in the binary Gaussian process classification model, and presents a comprehensive comparison of the approximations, their predictive performance and marginal likelihood estimates to results obtained by MCMC sampling.
Neural Information Processing SystemsPerspectives on Sparse Bayesian Learning
103 Citations2003Jason Palmer, Bhaskar D. Rao +1 more
It is demonstrated that SBL can be recast as the application of a rigorous vari-ational approximation to the full model by expressing the prior in a dual form, which obviates the necessity of assuming any hyperpriors and leads to natural, intuitive explanations of why sparsity is achieved in practice.
Healing the relevance vector machine through augmentation
94 Citations2005Carl Edward Rasmussen, Joaquin Quiñonero-Candela
The purpose of the resulting model, RVM*, is primarily to corroborate the theoretical and experimental analysis and show that sparsity comes at the expense of worse predictive.
Transductive and Inductive Methods for Approximate Gaussian Process Regression
76 Citations2002Anton Schwaighofer, Volker Tresp
It is found that subset of representers methods can give good and particularly fast predictions for data sets with high and medium noise levels and on complex low noise data sets, the Bayesian committee machine achieves significantly better accuracy, yet at a higher computational cost.
MPG.PuRe (Max Planck Society)Learning with Uncertainty: Gaussian Processes and Relevance Vector Machines
75 Citations2004Joaquin Quiñonero-Candela
This paper modify this method for training generalized linear models by adapting automatically the width of the basis functions to the optimal for the data at hand, and tries the Adaptive RVM for prediction on the chaotic Mackey-Glass time series.
A matching pursuit approach to sparse Gaussian process regression
67 Citations2005Sathiya Keerthi, Wei Chu
A new basis selection criterion for building sparse GP regression models is proposed that provides promising gains in accuracy as well as efficiency over previous methods, especially on the quality of predictive distributions.
The Bias-Variance Tradeoff and the Randomized GACV
41 Citations1998Grace Wahba, Xiwu Lin +4 more
Observations on the Nyström Method for Gaussian Processes
6 Citations2002Christopher K. I. Williams, Carl Edward Rasmussen +2 more