A Survey of Some Results in Stochastic Adaptive Control
SIAM Journal on Control and OptimizationPublished 1 May 1985
P. R. Kumar
Citations224
SJR quartileQ1
SJR score1.32
SNIP1.49
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
Some results in discrete-time stochastic adaptive control are surveyed and Bayesian and non-Bayesian adaptive control issues are dealt with.
Abstract
Some results in discrete-time stochastic adaptive control are surveyed. The survey divides itself into two parts—Bayesian and non-Bayesian adaptive control. In the former area, the problems of converting an incompletely observed system into a completely observed one, multi-armed bandit processes, Bayesian adaptive control of Markov chains and Bayesian adaptive control of linear systems are exposed and surveyed. In the latter area, non-Bayesian adaptive control of Markov chains and the self-tuning regulator are dealt with. Proofs are given, where appropriate, to illustrate the methods involved.
Keywords
Computer ScienceDecision Sciences
Bulletin of the American Mathematical SocietySome aspects of the sequential design of experiments
2,184 Citations1952Herbert Robbins
DSpace@MIT (Massachusetts Institute of Technology)Stochastic optimal control : the discrete time case
1,694 Citations2007Dimitri P. Bertsekas, Steven E. Shreve
This research monograph is the authoritative and comprehensive treatment of the mathematical foundations of stochastic optimal control of discrete-time systems, including thetreatment of the intricate measure-theoretic issues.
Journal of the Royal Statistical Society Series B (Statistical Methodology)Bandit Processes and Dynamic Allocation Indices
1,532 Citations1979John Gittins
Applied mathematical sciencesStochastic Approximation Methods for Constrained and Unconstrained Systems
1,482 Citations1978Harold J. Kushner, Dean S. Clark
The Robbins-Monro and Kiefer-Wolfowitz Algorithm for Inequality Constraints and the Weak Convergence of Probability Measures, a simple example, and the Convergence Theorem, a proof of the Main Theorem.
IEEE Transactions on Automatic ControlAnalysis of recursive stochastic algorithms
1,468 Citations1977Lennart Ljung
It is shown how a deterministic differential equation can be associated with the algorithm and examples of applications of the results to problems in identification and adaptive control.
AutomaticaImplementation of self-tuning regulators with variable forgetting factors
843 Citations1981T.R. Fortescue, L.S. Kershenbaum +1 more
A modified version of the self-tuning regulator having limited adaptability has been successfully implemented on a large-scale chemical pilot plant and the use of a variable forgetting factor with correct choice of information bound can avoid one of the major difficulties associated with constant exponential weighting of past data.
AutomaticaTheory and applications of self-tuning regulators
649 Citations1977Karl Johan Åström, Ulf Borisson +2 more
The regulator algorithms, their theory and industrial applications are reviewed and the major ideas are covered but detailed analysis is given elsewhere.
The Annals of StatisticsLeast Squares Estimates in Stochastic Regression Models with Applications to Identification and Control of Dynamic Systems
637 Citations1982Tze Leung Lai, Ching Zong Wei
Journal of Economic TheoryA two-armed bandit theory of market pricing
635 Citations1974Michael Rothschild
Journal of the American Statistical AssociationApproximation and Weak Convergence Methods for Random Processes, With Applications to Stochastic Systems Theory.
570 Citations1985Ioannis Karatzas, Harold J. Kushner
Control and communications engineers, physicists, and probability theorists, among others, will find this book unique, which contains a detailed development of approximation and limit theorems and methods for random processes and applies them to numerous problems of practical importance.
Journal of the Royal Statistical Society Series B (Statistical Methodology)Multi-Armed Bandits and the Gittins Index
530 Citations1980Peter Whittle
Foundations of Non-stationary Dynamic Programming with Discrete Time Parameter
484 Citations1970Karl Hinderer
SIAM Journal on Control and OptimizationDiscrete Time Stochastic Adaptive Control
433 Citations1981Graham C. Goodwin, Peter J. Ramadge +1 more
IEEE Transactions on Automatic ControlOn positive real transfer functions and the convergence of some recursive schemes
404 Citations1977Lennart Ljung
The convergence with probability one of a recently suggested recursive identification method by Landau is investigated and the positive realness of a certain transfer function is shown to play a crucial role, both for the proof of convergence and for convergence itself.
IEEE Transactions on Automatic ControlDual effect, certainty equivalence, and separation in stochastic control
381 Citations1974Yaakov Bar‐Shalom, Edison Tse
IEE Proceedings D Control Theory and ApplicationsSelf-tuning controllers based on pole-zero placement
359 Citations1980Karl Johan Åström, B. Wittenmark
The paper gives a review of pole-placement design for systems with known parameters, then focuses entirely on the servos, leading to the formulation of explicit and implicit schemes.
AutomaticaSelf-tuning regulators for a class of multivariable systems
296 Citations1979Ulf Borison
A multivariable self-tuning regulator based on the minimum variance strategy, which uses a recursive least squares estimator and a linear controller obtained directly from the current estimates, gives insight into the case when several single-input single-output self- Tuning regulators are operating in cascade mode.
AutomaticaA multivariable self-tuning controller
296 Citations1980H.N. Koivo
A multivariable self-tuning controller is derived extending the scalar version of Clarke and Gawthrop (1975) because the control terms are penalized in the cost function, fluctuations and peaking in control signals are reduced.
International Journal of ControlNecessary and sufficient condition for robust stability of linear distributed feedback systems†
292 Citations1982M. J. Chen, C. Desoer
Considering linear time-invariant distributed feedback systems, a necessary and sufficient condition for robust stability is derived with respect to plant perturbations belonging to a specified ball with design limitations imposed by plant uncertainties.
IEEE Transactions on Automatic ControlThe convergence of AML
253 Citations1979Victor Solo
It is shown that provided a certain positive real condition is satisfied, the AML recursion for the parameters of a scalar ARMAX time series model converges with probability one without the need of monitoring.
Operations ResearchMarkovian Decision Processes with Uncertain Transition Probabilities
227 Citations1973Jay Satia, Roy E. Lave
International Journal of ControlStochastic adaptive control methods: a survey
184 Citations1975Björn Wittenmark
The main part of this paper will cover stochastic adaptive controllers and the problem of dual control is given particular attention.
Journal of Mathematical Analysis and ApplicationsSufficient statistics in the optimum control of stochastic systems
183 Citations1965Charlotte Striebel
Only certain formal properties of the loss function will be required, they will be kept to a minimum, and their significance will be discussed as they are introduced.
International Journal of ControlSelf-tuning pole/zero assignment regulators
183 Citations1979P.E. Wellstead, J. M. EDMUNDS +2 more
Advances in Applied ProbabilityEstimation and control in Markov chains
180 Citations1974Petr Mandl
Asymptotic properties of the estimate and of the criterion function follow from the law of large numbers and from the central limit theorem for controlled Markov chains derived with the aid of martingales.
IEEE Transactions on Automatic ControlWide-sense adaptive dual control for nonlinear stochastic systems
171 Citations1973Edison Tse, Yaakov Bar‐Shalom +1 more
The control problem is formulated such that the resulting control sequence exhibits the closed-loop property, i.e., it takes into account the past observations and also the future observation program, which means that this control also plans its future learning according to the control objective.
IEEE Transactions on Information TheoryApplications of a Kushner and Clark lemma to general classes of stochastic algorithms
158 Citations1984Michel Métivier, P. Priouret
It is shown how one can apply a lemma of Kushner and Clark to obtain properties of stochastic algorithms, including algorithms considered by Ljung as well as algorithms of the form \theta_{n+1} = \theTA_{n} - \gamma_{n-1} V_{n +1}(\theta_n, Z) , where Z is a stationary ergodic process.
International Journal of ControlExtended self-tuning algorithm
145 Citations1981P.E. Wellstead, S.P. Sanoff
IEEE Transactions on Automatic ControlStochastic dynamic programming: Caution and probing
131 Citations1981Yaakov Bar‐Shalom
IEEE Transactions on Automatic ControlAdaptive control of Markov chains, I: Finite parameter set
122 Citations1979Vivek S. Borkar, Pravin Varaiya
AutomaticaStochastic adaptive control using a modified least squares algorithm
121 Citations1982Kwai Sin, Graham C. Goodwin
It is shown that, with probability one, the algorithm will ensure that the system inputs and outputs are sample mean square bounded and the mean square output tracking error achieves its global minimum possible value for linear feedback control.
IEEE Transactions on Automatic ControlAdaptive control with the stochastic approximation algorithm: Geometry and convergence
121 Citations1985Arthur H. Becker, P. Rajesh Kumar +1 more
IEEE Transactions on Automatic ControlAn actively adaptive control for linear systems with random parameters via the dual control approach
115 Citations1973Edison Tse, Yaakov Bar‐Shalom
Proceedings of the IEEEPartitioning: A unifying framework for adaptive systems, II: Control
108 Citations1976D.G. Lainiotis
Using the partitioning framework and a related general class of PAC's the adaptive control problem is treated from a global viewpoint that unifies seemingly unrelated previously obtained suboptimal adaptive controls, and yields fundamentally new adaptive control algorithms.
IEE Proceedings D Control Theory and ApplicationsGeneralised self-tuning controller with pole assignment
107 Citations1980A.Y. Allidina, F.M. Hughes
AutomaticaAdaptive control of linear stochastic systems
99 Citations1973Jayant G. Deshpande, T.N. Upadhyay +1 more
An adaptive control algorithm for linear systems with unknown constant parameters and quadratic performance criterion has been obtained and is given as the weighted integral of the model conditional optimal controls with the a-posteriori probabilities as weights.
The Annals of StatisticsIncomplete Information in Markovian Decision Models
92 Citations1974Detlef Rhenius
Advances in Applied ProbabilityBayesian dynamic programming
91 Citations1975Ulrich Rieder
It is shown that this model can be reduced to a non-Markovian (resp. Markovian) decision model with completely known transition probabilities and the restriction on deterministic generalized Markov policies, the criteria of optimality and the existence of Bayes policies are presented.
SIAM Journal on Control and OptimizationOptimal Adaptive Control of Linear-Quadratic-Gaussian Systems
78 Citations1983Pappu Kumar
A new adaptive control law, based on biasing the usual least-squares parameter estimation criterion with a term favoring parameters associated with lower optimal costs, is introduced, which proves its imperviousness to the closed-loop identification problem.
IEEE Transactions on Automatic ControlA new family of optimal adaptive controllers for Markov chains
78 Citations1982P. Rajesh Kumar, Arthur H. Becker
A new family of adaptive controllers is exhibited which achieves a performance precisely equalling the optimal performance achievable if the transition probabilities were known instead of the model or dynamics of the system.
Journal of Applied ProbabilityOn Bayesian models in stochastic scheduling
76 Citations1977John Gittins, K. D. Glazebrook
IEEE Transactions on Automatic ControlOptimal adaptive controllers for unknown Markov chains
70 Citations1982P. Ramakanth Kumar, Woei Lin
The goal is to design an adaptive controller to adequately control the unknown system when its performance is measured by the average cost incurred over a long operating time period, and the adaptive controllers proposed here are truly optimal, even when operating on an unknown system.
International Journal of ControlOn the stability and convergence of a self-tuning controller
69 Citations1980P.J. Gawthrop
International Journal of ControlA control weighted minimum-variance controller for non-minimum phase systems
65 Citations1981M.J. Grimble
IEEE Transactions on Automatic ControlStochastic adaptive control and prediction--The general delay-colored noise case
64 Citations1980Graham C. Goodwin, Kwai Sin +1 more
The general delay-white noise case and the unit delay-colored noise case are treated and a slightly different proof technique is used which relies only upon elementary analysis and is thus of pedagogical interest.
Journal of the Franklin InstituteOptimal control of partially observable Markovian systems
60 Citations1965Masanao Aoki
This paper discusses the discrete-time Bayesian optimal control of stochastic dynamic systems where some vectors, which augment the system state vectors and the observed state vectors by additional variables, constitute multi-dimensional Markov chains.
IEEE Transactions on Automatic ControlConvergence of adaptive minimum variance algorithms via weighting coefficient selection
59 Citations1982R. Kumar, J.B. Moore
Strong global convergence of the parameter estimates, state estimates, and prediction or tracking errors is not only guaranteed under the appropriate noise, passivity, and stability or minimum phase conditions, but the convergence is also as fast as it appears reasonable to achieve given the simplicity of the adaptive scheme.
International Journal of ControlCaution and probing in stochastic control†
58 Citations1972O.L.R. Jacobs, John W. Patchell
SIAM Journal on Control and OptimizationAn Invariant Measure Approach to the Convergence of Stochastic Approximations with State Dependent Noise
57 Citations1984Harold J. Kushner, Adam Shwartz
SIAM Journal on Control and OptimizationIdentification and Adaptive Control of Markov Chains
55 Citations1982Vivek S. Borkar, Pravin Varaiya
AutomaticaAdaptive control and identification of the dissolved oxygen process
55 Citations1982Kay Yuh-Ju Ko, B. McInnis +1 more
This paper suggests how nonlinear adaptive control might lead to improved control of the dissolved oxygen (DO) concentration in the aerator of a wastewater treatment plant.
IEEE Transactions on Automatic ControlA multiple model adaptive dual control algorithm for stochastic systems with unknown parameters
53 Citations1980C. Wenk, Yaakov Bar‐Shalom
IEEE Transactions on Automatic ControlConvergence of recursive adaptive and identification procedures via weak convergence theory
49 Citations1977Harold J. Kushner
The Annals of StatisticsMulti-Armed Bandits with Discount Factor Near One: The Bernoulli Case
45 Citations1981F. P. Kelly
International Journal of ControlServo self-tuners
43 Citations1979P.E. Wellstead, P. Zanker
The problem of adding reference inputs to self-tuning systems is considered using the classical device of set-point feedforward and enough additional information is generated to formulate a novel approximate maximum likelihood estimator for use with self- Tuned systems.
AutomaticaConvergence analysis of recursive identification algorithms with forgetting factor
41 Citations1983Luis Lozano
A convergence analysis of a modified version of the least-squares recursive identification algorithm with forgetting factor is given and it is shown that the parametric distance converges to a zero mean random variable.
IEEE Transactions on Automatic ControlGeneralized certainty equivalence and dual effect in stochastic control
41 Citations1975Edison Tse, Yaakov Bar‐Shalom
IEE Proceedings D Control Theory and ApplicationsExplicit self-tuning methods
36 Citations1980J.-J. Fuchs
A general scheme, which allows the analysis of self-tuning controllers of linear systems with constant but unkonwn parameters, is presented and it is shown that any identification algorithm satisfying the previously obtained properties leads to an overall adaptive system stable in some satisfactory stochastic sense.
AutomaticaAn adaptive automaton controller for discrete-time markov processes
35 Citations1969J. S. Riordon
An existing method for dual control of a repetitive single-stage stochastic process may be extended to apply to the present case and a method is introduced of calculating successive policy estimates recursively, so that the task of updating the estimated optimal feedback policy at each stage of the process is rendered feasible.
Extension of the multi-armed bandit problem
35 Citations1983Pravin Varaiya, Jean Walrand +1 more
AutomaticaSelf-tuning control of a chip refiner motor load
34 Citations1982Guy A. Dumont
Results of plant trials on an industrial thermomechanical pulp production unit show that this self-tuning regulator is capable of tracking the slow gain drift and of taking proper action in case a pulp pad collapse occurs.
Journal of Applied ProbabilityStrong consistency of a modified maximum likelihood estimator for controlled Markov chains
31 Citations1980Bharat T. Doshi, Steven E. Shreve
IEEE Transactions on Automatic ControlSimultaneous identification and adaptive control of unknown systems over finite parameter sets
30 Citations1983P. R. Kumar
IEEE Transactions on Automatic ControlLearning control of finite Markov chains with unknown transition probabilities
30 Citations1982Mitsuo Satõ, Kenichi Abe +1 more
It is shown that the algorithms are asymptotically optimal in the sense that the probability of selecting an optimal policy converges to unity.
IEEE Transactions on Automatic ControlIndirect stochastic adaptive control: The general delay-white noise case
29 Citations1982J.-J. Fuchs
This work shows that in the white noise case a single adaptation algorithm suffices to establish that, with probability one, the systems input, output and the output tracking error are sample mean-square bounded.
Journal of the Royal Statistical Society Series B (Statistical Methodology)On Randomized Dynamic Allocation Indices for the Sequential Design of Experiments
25 Citations1980K. D. Glazebrook
Journal of Applied ProbabilityOn a sufficient condition for superprocesses due to whittle
24 Citations1982K. D. Glazebrook
Advances in Applied ProbabilityRandomised allocation of treatments in sequential trials
23 Citations1980J. A. Bather
Journal of Applied ProbabilityOn the evaluation of suboptimal strategies for families of alternative bandit processes
22 Citations1982K. D. Glazebrook
AutomaticaVariable forgetting factors in parameter estimation
22 Citations1983M. B. Zarrop
Based on Ljung's analysis of the off-line case, it is shown that the use of forgetting factors can give rise to identifiability problems, unless the behaviour of these factors over time satisfied certain conditions.
International Journal of ControlWeighted minimum-variance self-tuning control
20 Citations1982M.J. Grimble
Mathematische Operationsforschung und Statistik Series OptimizationThe Average-Optimal Adaptive Control of a Markov Renewal Model in Presence of an Unknown Parameter
20 Citations1982Michael Kolonko
SIAM Journal on Control and OptimizationAdaptive Control with a Compact Parameter Set
18 Citations1982P. R. Kumar
AutomaticaOn adaptive minimum variance regulation for non-minimum phase plants
18 Citations1983R. Kumar, J.Β. Moore
This paper presents only the discrete-time adaptive regulator version of the adaptive pole assignment scheme, which has interpretations as an adaptive quadratic index minimizing procedure and also as an Adaptive pole assignment algorithm.
Journal of Applied ProbabilityStrongly consistent estimation in a controlled Markov renewal model
17 Citations1982Michael Kolonko
Advances in Applied ProbabilityGradient approach for recursive estimation and control in finite Markov chains
17 Citations1981Y. M. El-Fattah
A recursive stochastic gradient algorithm is presented for the adaptation of the control policy at consecutive times using a Taylor-like expansion formula for the expected reward in terms of policy variations.
IEEE Transactions on Systems Man and CyberneticsRecursive Algorithms for Adaptive Control of Finite Markov Chains
17 Citations1981Y. M. El-Fattah
A recursive stochastic gradient algorithm is presented for the adaptation of the control policy at consecutive times using the estimated transition parameter which is also recursively updated using the gradient of the likelihood function.
SIAM Journal on Control and OptimizationAn Exact Formula for a Linear Quadratic Adaptive Stochastic Optimal Control Law
16 Citations1986Raymond Rishel
IEEE Transactions on Automatic ControlIndirect stochastic adaptive control: The general delay-colored noise case
15 Citations1982J.-J. Fuchs
IEEE Transactions on Automatic ControlStochastic adaptive control and prediction based on a modified least squares--The general delay-colored noise case
15 Citations1982Youhong Zhang
AutomaticaSimulations of adaptive controllers for a paper machine headbox
15 Citations1983Frederik D'Hulster, R.M.C. De Keyser +1 more
Different adaptive control methods are used on the simulated process and compared to the classical PI-controls, particularly, a single-output self-tuning controller with feedforward action and a multivariable adaptive deadbeat controller are promising.
IEEE Transactions on Automatic ControlOptimal strategies for families of alternative bandit processes
15 Citations1983K. D. Glazebrook
IEEE Transactions on Automatic ControlOn the optimal solution of the one-armed bandit adaptive control problem
15 Citations1981P. R. Kumar, Thomas I. Seidman
AutomaticaComments on: ‘Implementation of self-tuning regulators with variable forgetting factors’
14 Citations1983S.P. Sanoff, P.E. Wellstead
It is shown that the method of Fortescue, Kershenbaum and Ydstie can be made significantly more efficient, computationally, than is indicated.
IEEE Transactions on Automatic ControlSome properties of the dual adaptive stochastic control algorithm
14 Citations1981Pierre Dersin, Michael Athans +1 more
Comparing analytically the properties of the suboptimal dual adaptive stochastic control with those of the optimal control, when plant dynamic contain multiplicative white noise parameters is compared.
IEEE Transactions on Automatic ControlConvergence and rate of convergence of a recursive identification and adaptive control method which uses truncated estimators
12 Citations1982Harold J. Kushner, Rajendra Kumar
IEEE Transactions on Automatic ControlSimultaneous adaptive control and identification via the weighted least-square algorithm
11 Citations1984R. Kumar
It is proved that the regularity and persistency conditions are satisfied, under mild mixing conditions on the noise and no pole zero cancellation in the plant input-output transfer function, by a random perturbation of the reference trajectory.
The Annals of StatisticsAdaptive Policies for Markov Renewal Programs
11 Citations1973Bennett L. Fox, John E. Rolph
AutomaticaOn low frequency and long-run effects in self-tuning control
10 Citations1983Krzysztof J. Latawiec, M. Chyra
An 'extended self-tuning controller' is developed and its superiority over classical ones is demonstrated.
AutomaticaAdaptive control to a dry etch process by microcomputer
10 Citations1982H Boehm
The aim of this paper is to show, using a pilot installation, that adaptive control is able to solve development demands concerning working point variations during one run and to help the process engineer to investigate a new tool.
IEEE Transactions on Automatic ControlAdaptive control and parameter estimation in Markov chains: A linear case
9 Citations1982B. Sagalovsky
Some Results in the Adaptive Control of Linear Systems
8 Citations1977Petr Mandl
AutomaticaThe convergence of an instrumental-variable-like recursion
8 Citations1981Victor Solo
Here it is shown that the recursion converges without being monitored and the result is disappointing because the positive real condition depends on the transfer function characteristic function.
The strong consistency of the stochastic gradient algorithm of adaptive control
6 Citations1984H. Chen, Peter E. Caines
Recursive system identification and adaptive control by use of the modified least squares algorithm
5 Citations1983H. Chen
A weaker than persistent excitation condition is used to establish the strong consistency of MLS for MIMO stochastic systems without monitoring and a comparison between various conditions for consistency is given.
CyberneticsRecursive algorithms of adaptive control in stochastic systems
4 Citations1981Владимир В. Баранов
A recursive procedure is constructed that has a number of fairly useful properties and makes it possible to construct an optimal true stationary strategy after finitely many steps of adaptive control and to solve problems of fairly large dimension.
…