The Entropy as a Measure of Uncertainty
Foundations in signal processing, communications and networkingPublished 1 January 2014
Rudolf Ahlswede, Alexander Ahlswede, Ingo Althöfer, Christian Deppe, Ulrich Tamm
Citations3
Generate an AI Snapshot to get a quick, structured summary of this paper.
Study Snapshot
ObjectiveStudy objective
MethodsResearch methodology
PopulationPopulation studied
Sample sizeSample sizes
OutcomesStudy outcomes here
ResultsStudy results comes here
LimitationsResearch study limitations comes here
A concise AI-generated summary of the paper will appear here once you click Generate AI Snapshot.
TL;DR
This lecture will take a different approach and interpret the entropy as a measure of uncertainty of an experiment with \(n\) possible outcomes, where each outcome will take place with a certain probability.
Abstract
Up to now we had an operational access to the entropy function, i.e., the entropy was involved into the solution of a mathematical problem. More specifically, the entropy turned out to be a measure for data compression. In this lecture we will take a different approach and interpret the entropy as a measure of uncertainty of an experiment with \(n\) possible outcomes, where each outcome will take place with a certain probability. The approach will be axiomatic, i.e., some "reasonable" conditions which a measure of uncertainty should possess are postulated.
Keywords
Computer ScienceNeurosciencePhysics and Astronomy
Bell System Technical JournalA Mathematical Theory of Communication
80,675 Citations1948Claude E. Shannon
Journal of the American Statistical AssociationInequalities: Theory of Majorization and its Applications.
4,984 Citations1981Barry C. Arnold, Albert W. Marshall +1 more
IEEE Transactions on Information TheoryAxiomatic derivation of the principle of maximum entropy and the principle of minimum cross-entropy
1,799 Citations1980John E. Shore, Ryan F. Johnson
Jaynes's principle of maximum entropy and Kullbacks principle of minimum cross-entropy (minimum directed divergence) are shown to be uniquely correct methods for inductive inference when new information is given in the form of expected values.
The Annals of Probability$I$-Divergence Geometry of Probability Distributions and Minimization Problems
1,648 Citations1975Imre Csiszár
The Annals of StatisticsWhy Least Squares and Maximum Entropy? An Axiomatic Approach to Inference for Linear Inverse Problems
804 Citations1991Imre Csiszár
Information and ControlSome inequalities satisfied by the quantities of information of Fisher and Shannon
700 Citations1959A. J. Stam
A certain analogy is found to exist between a special case of Fisher's quantity of information I and the inverse of the “entropy power” of Shannon and this constitutes a sharpening of the uncertainty relation of quantum mechanics for canonically conjugated variables.
The Annals of ProbabilityThe Capacity Region of a Channel with Two Senders and Two Receivers
347 Citations1974Rudolf Ahlswede
IEEE Transactions on Information TheoryThe convolution inequality for entropy powers
342 Citations1965Nelson M. Blachman
An improved version of Stam's proof of Shannon's convolution inequality for entropy power is presented, which is obtained by mathematical induction from the one-dimensional case.
Information and ControlA coding theorem and Rényi's entropy
278 Citations1965L. L. Campbell
It is shown that L(t) has some desirable properties as a measure of typical code length and a coding theorem for a noiseless channel is proved.
IEEE Transactions on Information TheoryGaussian feedback capacity
221 Citations1989Thomas M. Cover, Sandeep Pombra
An asymptotic equipartition theorem for nonstationary Gaussian processes is proved and it is proved that the feedback capacity C/sub FB/ in bits per transmission and the nonfeedback capacity C satisfy C > C >.
CERN Document Server (European Organization for Nuclear Research)Mathematical foundations of information theory
108 Citations2013Khinchin, Aleksandr Yakovlevich
Proceedings of the IEEEAn information-theoretic proof of Burg's maximum entropy spectrum
54 Citations1984B. S. Choi, Thomas M. Cover
This work poses a somewhat broader entropy maximization problem, in which stationarity and normality are not assumed, and shifts the burden of proof from the previous focus on the calculus of variations and time series techniques to a string of information-theoretic inequalities.
Mathematical Proceedings of the Cambridge Philosophical SocietySome inequalities concerning positive-definite Hermitian matrices
39 Citations1955Ky Fan
IEEE Transactions on Information TheoryAn information - theoretic proof of Hadamard's inequality (Corresp.)
19 Citations1983Thomas M. Cover, Abbas El Gamal
Hadamard's inequality follows immediately from inspection of both sides of the entropy inequality h(X_{1}, X_{2},, X_{n})\leq h( X_{i}) when (X_{l}, X_2},\cdots, X_n}) is multivariate normal.
American Mathematical MonthlyA Convexity Proof of Hadamard's Inequality
18 Citations1982Albert W. Marshall, Ingram Olkin