Joint Optimization and Variable Selection of High-dimensional Gaussian\n Processes

arXiv (Cornell University)Published 27 June 2012Open access

Bo Chen, Rui M. Castro, Andreas Krause

DOI10.48550/arxiv.1206.6396

Citations38

TL;DR

This paper modeling the unknown function as a sample from a high-dimensional Gaussian process (GP) distribution shows that it is possible to perform joint variable selection and GP optimization and provides strong performance guarantees for the algorithm.

Abstract

Maximizing high-dimensional, non-convex functions through noisy observations\nis a notoriously hard problem, but one that arises in many applications. In\nthis paper, we tackle this challenge by modeling the unknown function as a\nsample from a high-dimensional Gaussian process (GP) distribution. Assuming\nthat the unknown function only depends on few relevant variables, we show that\nit is possible to perform joint variable selection and GP optimization. We\nprovide strong performance guarantees for our algorithm, bounding the sample\ncomplexity of variable selection, and as well as providing cumulative regret\nbounds. We further provide empirical evidence on the effectiveness of our\nalgorithm on several benchmark optimization problems.\n

Keywords

Computer ScienceDecision Sciences

Journal of the Royal Statistical Society Series B (Statistical Methodology)Regression Shrinkage and Selection Via the Lasso

51,790 Citations1996Robert Tibshirani

A new method for estimation in linear models called the lasso, which minimizes the residual sum of squares subject to the sum of the absolute value of the coefficients being less than a constant, is proposed.

Neural ComputationBayesian Interpolation

4,377 Citations1992David Mackay

The Bayesian approach to regularization and model-comparison is demonstrated by studying the inference problem of interpolating noisy data by examining the posterior probability distribution of regularizing constants and noise levels.

View PDF

arXiv (Cornell University)A Tutorial on Bayesian Optimization of Expensive Cost Functions, with Application to Active User Modeling and Hierarchical Reinforcement Learning

2,133 Citations2010Eric Brochu, Vlad M. Cora +1 more

A tutorial on Bayesian optimization, a method of finding the maximum of expensive cost functions using the Bayesian technique of setting a prior over the objective function and combining it with evidence to get a posterior function.

View PDF

Infoscience (Ecole Polytechnique Fédérale de Lausanne)Gaussian Process Optimization in the Bandit Setting: No Regret and Experimental Design

1,048 Citations2009Niranjan Srinivas, Andreas Krause +2 more

View PDF

Automatic gait optimization with Gaussian process regression

247 Citations2007Daniel J. Lizotte, Tao Wang +2 more

A Bayesian approach based on Gaussian process regression that addresses all three drawbacks of local optimization procedures, using a global search strategy based on a posterior model inferred from all of the individual noisy evaluations.

Nonmyopic active learning of Gaussian processes

172 Citations2007Andreas Krause, Carlos Guestrin

An analysis and efficient algorithms are presented that address the question of when an active learning, or sequential design, strategy will perform significantly better than sensing at an a priori specified set of locations for Gaussian Processes.

arXiv (Cornell University)Bandit Theory meets Compressed Sensing for high dimensional Stochastic\n Linear Bandit

76 Citations2012Alexandra Carpentier, Rémi Munos

This paper combines ideas from Compressed Sensing and Bandit Theory and derive an algorithm with a regret bound in O(S p n), an application to the problem of optimizing a function that depends on many variables but among which only a small number of them are relevant.

View PDF