Next:Entropy of a ContinuousUp:Entropy Rate and Entropy

## Entropy Series

This subsection deals with a measure of entropy commonly referred as entropy series, where in the probability distribution we have . This quantity is given by where with and .

Let and be two sequences. Then it can easily be checked that the series diverges and the series converges. Let be the sum of the series. Consider , then In view of the fact that the series diverges, we get as infinite. In order that the entropy series converges, we need some restrictions. If there is a convergent series of positive terms such that also converges. Then by use of the inequality (1.9), we get the following bound: The following properties give better restrictions to bound the entropy series .

Property 1.64. If for some , then Property 1.65. For each nondecreasing probability sequence i.e., , the entropy series converges iff the series converges.

Property 1.66. For each nondecreasing sequence , the following bound on the entropy series holds: where is a nonnegative, nondecreasing sequence of numbers with the property that for some  and Property 1.67. For each nondecreasing probability sequence , the following bound on the entropy series hold: where is as given in property 1.65.

Property 1.68. (Maximum-Entropy Principle). The entropy series under the constraints

(a) (b) is maximized when and the maximum value is given by where are normalizing constants.

Note 1.6. Using the condition , we get (1.12)

By applying the conditions , we can calculate the constants . The probability distribution (1.12) is famous in the literature as Gibbs distribution''.

Particular cases: The property 1.69 admits the following interesting particular cases.

(i) The probability distribution  that maximizes the corresponding entropy is the uniform distribution given by .
(ii) The probability distribution  that maximizes the corresponding entropy series subject to the constraint is the geometric distribution given by , where b can be obtained by using the given constraints.
(iii) The probability distribution  (integers) that maximizes the corresponding entropy subject to the constraints and is the discrete normal distribution given by , where can be obtained by using the given constraints.
Property 1.69. Let be a sequence of real numbers with the property that for some , then the following bound on the entropy series hold: Property 1.70. The following bound on the entropy series holds: where is as given in (1.8).

The following properties are also worth emphasizing, whose details can be seen in Capocelli et al. (1988a;b) .

Property 1.71. The following bound on the entropy series hold: where is the Riemann zeta function and is the unique solution of the equation and is defined as when is a power of a prime p; 0 otherwise.

Property 1.72. The following bound on the entropy series hold: where is a bounded function of , with Property 1.73. For each nonincreasing probability distribution and for each , the following bound on the entropy series holds: where is a constant independent of the probability distribution and for  , we have and Property 1.74. For each nonincreasing probability distribution , the following bound on the entropy series holds: where Property 1.75. For each nonincreasing probability distribution and for each , the following limits hold:

(i) (ii) (iii) (iv) where   and Note 1.7. The good references for the sections 1.4, 1.5, 1.6 and 1.7 are the book by Csiszár and Körner (1981) , Guiasu (1977) , McEliece (1977) , etc..
21-06-2001
Inder Jeet Taneja
Departamento de Matemática - UFSC
88.040-900 Florianópolis, SC - Brazil