Next:Multivariate EntropiesUp:Shannon's Entropy

# Properties of Shannon's Entropy

The measure of uncertainty  given by (1.7) satisfy many interesting properties. For simplicity, we shall take  or  instead of . Unless otherwise specified, it is understood that  and . Also it is understood that  and all the logarithms are with base 2.

Property 1.1. (Nonnegativity) with equality iff , where.

Property 1.2. (Continuity) is a continuous function of P.

Property 1.3. (Symmetry) is a symmetric function of its arguments, i.e.,

where  is any permutation from 1 to n.

Property 1.4. (Expansible). We have

Property 1.5. (Decisive). We have

Property 1.6. (Normality). We have

Property 1.7. (Sum Representation). We can write

where

Property 1.8. (Recursivity). We have

Property 1.9. (Additivity). We have

Property 1.10. (Strongly Additive). We have

Property 1.11. (Grouping). We have

Property 1.12. (Generalized Grouping). We have

Property 1.13. (Binary-Entropic). Let

 (1.8)

Then

(i)
(ii)
(iii)
(iv)
(v) there exists a K such that .
(vi) the function  is non-decreasing on the interval
(vii) for every , we have
(viii)
(ix)
Property 1.14. (Shannon-Gibbs Inequality). For all  and , we have
 (1.9)

with equality iff  or iff  (whenever for some , the corresponding  is also zero).

Property 1.15. (Maximality). is maximum when all the probabilities are equal i.e.,

with equality iff

Property 1.16. (Uniform distribution). Let

Then
(i)
(ii)
(iii)
(iv)
Property 1.17. (Sub-additivity). We have

Property 1.18. (Independence Inequality). If

and
then

Property 1.19. (Concavity). is a concave function of  in .

Property 1.20. (Schur-concavity). is a Shur-concave function of  in .

Property 1.21. Let  be a probability distribution such that . Let us define  such that  Then .

Property 1.22. (Difference among two entropies). If

then
for all .

Property 1.23. Bounds on H(P). For , we have

(i)
(ii)
Property 1.24. (Relative to maximum probability). Let . Then
(i)
(ii)
(iii)  where  is a positive integer satisfying
(iv)
Property 1.25. Let  and be two probability distributions. If , then , where  is as given in (1.8).

Property 1.26. If  is strongly additive (property 1.10), then it is additive (property 1.9).

Property 1.27. If  is additive (property 1.9) for  and is expansible (property 1.4), then it is also additive (property 1.9).

Property 1.28. If  is expansible (property 1.4) and strongly additive (property 1.10), then it is recursive (property 1.8).

Property 1.29. If  is recursive (property 1.8) for  and symmetric (property 1.3) for , then it is symmetric (property 1.3) for  and decisive (property 1.5).

Property 1.30. If  is recursive (property 1.8), symmetric (property 1.3) for , then it is symmetric (property 1.3) and expansible (property 1.4).

Property 1.31. If  is symmetric (property 1.3) and recursive (property 1.8), then

Property 1.32. If  is recursive (property 1.8) and symmetric (property 1.3) for , then it is also strongly additive (property 1.10).

Property 1.33. If  is expansible (property 1.4) and subadditive (property 1.17) for , then it is nonnegative (property 1.1).

Property 1.34. If  is branching of the form

where
then

Property 1.35. If  is recursive (property 1.8), and , then

with the convention that

Property 1.36. If  is normalized (property 1.6), symmetric (for ) (property 1.3) and recursive (for ) (property 1.8), then the function  satisfies the property 1.13(i)-(iv).

Property 1.37. Binary entropic properties given by 1.13(i)-(iv) implies that  is symmetric (property 1.3), normalized (property 1.6), expansible (property 1.4), decisive (property 1.5), recursive (property 1.8), strongly additive (property 1.9) and additive (property 1.10).

Note 1.1. Some of the properties given above can be seen in Aczél and Daróczy (1975) [2] and Mathai and Rathie (1975) [71]

21-06-2001
Inder Jeet Taneja
Departamento de Matemática - UFSC
88.040-900 Florianópolis, SC - Brazil