Next:Mutual InformationUp:Shannon's Entropy

# Multivariate Entropies

Let  and  be two discrete finite random variables with joint and individual probability distributions given by

and
The conditional probability of  given  is given by
with
The conditional probability of  given is given by
with
The following relations are well known in the literature:

and
for each  When  and  are independent, we have

and
for each  Based on the above notations, we give now the joint, individual and conditional measures of uncertainty. The joint measure of uncertainty of  is given by
The individual measures of uncertainty of  and  are given by

and
respectively The conditional uncertainty of Y given  is given by
for each. The conditional uncertainty of  given  is the average uncertainty of  with the probabilities  is given by
Similarly, we can write the conditional uncertainty of  given  as
In case of three random variables  and  with their respective probability distributions, we have the following measures of uncertainty

etc..

The following properties hold for the above uncertainty measures.

Property 1.38. We have

(i)
(ii)
Property 1.39. We have
(i)
(ii)
Property 1.40. We have
(i) , with equality iff  and  are independent i.e.,
(ii) , with equality iff  and  are conditionally independent given  i.e.,  and each .
(iii) , with equality iff  and  are independent i.e.,
Note 1.2. Since the random variables  and  are symmetric among them, then from the property 1.40(ii), we can write

Property 1.41. We have
(i)
(ii)
(iii)
(iv)
Property 1.42. We have
(i) , with equality iff  and  are independent i.e.,
(ii) , with equality iff  and  are independent i.e., iff
(iii) , with equality iff  and  are conditionally independent given  i.e., iff
Property 1.43. We have
(i)
(ii) If , then
Property 1.44. For each k, define
Then
Property 1.45. Let . Then

Note 1.3. The property 1.45 is famous as "Fano-inequality".

For four discrete random variables  the following property holds.

Property 1.46. We have

(i)
(ii)
(iii)
(iv)
(v)
(vi)
(vii)
(viii)

21-06-2001
Inder Jeet Taneja
Departamento de Matemática - UFSC
88.040-900 Florianópolis, SC - Brazil