Maxent, Mathematics, and Information Theory
Published 1 January 1996
Imre Csiszár
Citations67
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This is a mathematically oriented survey about the method of maximum entropy or minimum I-divergence, with a critical treatment of its various justifications and relation to Bayesian statistics.
Abstract
This is a mathematically oriented survey about the method of maximum entropy or minimum I-divergence, with a critical treatment of its various justifications and relation to Bayesian statistics. Information theoretic ideas are given substantial attention, including "information geometry". The axiomatic approach is considered as the best justification of maxent, as well as of alternate methods of minimizing some Bregman distance or f-divergence other than I-divergence. The possible interpretation of such alternate methods within the original maxent paradigm is also considered.
Keywords
MathematicsNeurosciencePhysics and Astronomy
REAL-EOD (Library of the Hungarian Academy of Sciences and the Information Center Oriental Collection)Information Theory: Coding Theorems for Discrete Memoryless Systems
3,338 Citations2014Imre Csiszár, János Körner
This new edition presents unique discussions of information theoretic secrecy and of zero-error information theory, including the deep connections of the latter with extremal combinatorics.
USSR Computational Mathematics and Mathematical PhysicsThe relaxation method of finding the common point of convex sets and its application to the solution of problems in convex programming
2,659 Citations1967L.M. Bregman
This method can be regarded as a generalization of the methods discussed in [1–4] and applied to the approximate solution of problems in linear and convex programming.
IEEE Transactions on Information TheoryAxiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy
1,799 Citations1980John E. Shore, Ryan F. Johnson
Jaynes's principle of maximum entropy and Kullbacks principle of minimum cross-entropy (minimum directed divergence) are shown to be uniquely correct methods for inductive inference when new information is given in the form of expected values.
The Annals of Probability$I$-Divergence Geometry of Probability Distributions and Minimization Problems
1,648 Citations1975Imre Csiszár
Journal of the Royal Statistical Society Series B (Statistical Methodology)A General Class of Coefficients of Divergence of One Distribution from Another
1,230 Citations1966Syed Mustafa Ali, S. D. Silvey
Journal of the Royal Statistical Society Series B (Statistical Methodology)Reference Posterior Distributions for Bayesian Inference
1,103 Citations1979José M. Bernardo
The Annals of StatisticsWhy Least Squares and Maximum Entropy? An Axiomatic Approach to Inference for Linear Inverse Problems
804 Citations1991Imre Csiszár
The Annals of ProbabilitySanov Property, Generalized $I$-Projection and a Conditional Limit Theorem
343 Citations1984Imre Csiszár
E. T. Jaynes: Papers on Probability, Statistics and Statistical Physics
304 Citations1983Jaynes, Edwin T, Rosenkrantz, R D
Journal of Statistical Planning and InferenceJeffreys' prior is asymptotically least favorable under entropy risk
296 Citations1994Bertrand Clarke, Andrew R. Barron
It is shown that, among continuous positive priors, Jeffreys’ prior uniquely achieves the asymptotic maximin value and, in the discrete parameter case, the Bayes risk reduces to the entropy of the prior so that the reference prior is seen to be the maximum entropy prior.
IEEE Transactions on Information TheoryMaximum entropy and conditional probability
210 Citations1981Jan Van Campenhout, Thomas M. Cover
The following theorem is offered, which states that the conditional distribution of a given random variable X is the (normalized) product of the maximum entropy distribution and the initial distribution.
IEEE Transactions on Information TheoryCross entropy, dissimilarity measures, and characterizations of quadratic entropy
131 Citations1985C. Radhakrishna Rao, Tapan K. Nayak
It is shown that the square root of the Jensen difference (dissimilarity measure) arising out of a quadratic entropy provides a metric on a probability space.
SIAM Journal on OptimizationPartially-Finite Programming in $L_1 $ and the Existence of Maximum Entropy Estimates
106 Citations1993Jonathan M. Borwein, Adrian S. Lewis
A complete duality and existence theory is developed for this problem and for an associated extended problem which allows singular, measure-theoretic solutions.
IEEE Transactions on Information TheoryA source matching approach to finding minimax codes
95 Citations1980L. Davisson, Alberto Leon‐Garcia
A source matching approach to the problem of finding minimax cedes for classes of memoryless sources leads to a channel capacity problem so that Blahut's algorithm can be used to find approximations to the minimax code.
The Annals of StatisticsBayesian methods and maximum entropy for ill-posed inverse problems
86 Citations1997Fabrice Gamboa, Élisabeth Gassiat
The Annals of StatisticsA Geometric Interpretation of Darroch and Ratcliff's Generalized Iterative Scaling
77 Citations1989Imre Csiszár
French digital mathematics library (Numdam)Maximum d'entropie et problème des moments
66 Citations1990Didier Dacunha‐Castelle, Fabrice Gamboa
Acta Crystallographica Section A Foundations of CrystallographyThe use of non-local constraints in maximum-entropy electron density reconstruction
52 Citations1986Jorge Navaza
Statistics in MedicineSample sizes for phase ii clinical trials derived from Bayesian decision theory
42 Citations1994Hazel Brunier, John Whitehead
A decision theoretic approach is proposed to help determine the number of patients that should be treated in early phase clinical trials of a new medical treatment, and extends work of Sylvester and Staquet, and adopts a Bayesian formulation.
The Maximum Entropy on the Mean Method, Noise and Sensitivity
15 Citations1996Jean‐François Bercher, Guy Le Besnerais +1 more
This paper addresses the problem of building convenient criteria to solve linear and noisy inverse problems of the form y = Ax + n, and gives a criterion of the forms x = arg min x F(x) + G(y - Ax), where F and G are convex, without constraints.
Inverse ProblemsComputationally feasible high-resolution minimum-distance procedures which extend the maximum-entropy method
13 Citations1990Lee K. Jones, V Trutzer
A class of directed distances is introduced leading to numerically efficient reconstruction algorithms with high-resolution capability, and several examples of reconstruction with higher resolution than that for the maximum-entropy method are presented.
Inverse ProblemsComputationally feasible high-resolution minimum-distance procedures which extend the maximum-entropy method
9 Citations1989Lee K. Jones, V Trutzer
A class of directed distances is introduced leading to numerically efficient reconstruction algorithms with high-resolution capability, and several examples of reconstruction with higher resolution than that for the maximum-entropy method are presented.
“Barycenter” of a Set of Probability Measures and its Application in Statistical Decision
6 Citations1984Amalia Pérez
An algorithm for constructing the barycenter in some important cases of a set S of probability measures as a probability measure from a given projection family minimizing its maximal discrepancy with respect to the probability measures of the set S is given.