Next:Properties of Shannon's EntropyUp:Shannon's Entropy

# Measures of Uncertainty: Shannon's Entropy

Let be a discrete random variable taking a finite number of possible values with probabilities respectively such that  . We attempt to arrive at a number that will measure the amount of uncertainty. Let be a function defined on the interval and be interpreted as the uncertainty associated with the event , or the information conveyed by revealing that has taken on the value in a given performance of the experiment. For each n, we shall define a function of the n variables . The function is to be interpreted as the average uncertainty associated with the event given by (1.1)

Thus is the average uncertainty removed by revealing the value of . For simplicity we shall denote We shall now present some axiomatic characterizations of the measure of uncertainty  to arrive at its exact expression. For that, let and be two independent experiments with n and m values respectively. Let be a probability distribution associated with and be a probability distribution associated with . This lead us to write that (1.2)

for all  and  . Replacing by  we get (1.3)

Based on (1.2) and (1.3) we present the following theorem.

Theorem 1.1. Let be a function satisfying (1.2) and (1.3), where is real valued continuous function defined over [0,1]. Then is given by (1.4)

where with .

The proof is based on the following Lemma (ref. Chaundy and McLeod, 1961 ).

Lemma 1.1. Let be a continuous function satisfying (1.5)

for all Then (1.6)

for all with .

Alternatively the measure (1.4) can be characterized as follows (ref. Shannon,1948 ; Feinstein, 1958 ).

Theorem 1.2. Let be a function satisfying the following axioms:

(i) is a continuous function of .
(ii) be a symmetric function of its arguments.
(iii)  Then is given by (1.4).

A third way to characterize the measure (1.4) is as follows (ref. Aczél and Daróczy, 1975 ).

Theorem 1.3. Let be a function satisfying the following axioms:

(i) is a continuous and symmetric function with respect to its arguments.
(ii) (iii) (iv) For we have  Then is given by (1.4).

The following is a different way to characterize the measure (1.4). It is based on the functional equation famous as fundamental equation of information.

Theorem 1.4. Let be a function satisfying where satisfies the following functional equation with for all . Then is given by (1.4).

For simplicity, let us take in (1.4). If we put the restriction in the above theorems with , we get . This yields (1.7)
The expression (1.7) is famous as Shannon's entropy or measure of uncertainty.

For more characterizations of the measure (1.4) or (1.7) refer to Aczél and Daróczy (1975)  and Mathai and Rathie (1975) .

21-06-2001
Inder Jeet Taneja
Departamento de Matemática - UFSC
88.040-900 Florianópolis, SC - Brazil