Next:Mutual
InformationUp:Shannon's
Entropy
Previous:Properties
of Shannon's Entropy Go to:Table
of Contents
Multivariate Entropies
Let
and
be two discrete finite random variables with joint and individual probability
distributions given by
and
The conditional probability of
given
is given by
with
The conditional probability of
given
is given by
with
The following relations are well known in the literature:
and
for each
When
and
are independent, we have
and
for each
Based on the above notations, we give now the joint, individual
and conditional measures of uncertainty. The joint measure
of uncertainty of
is given by
The individual measures of uncertainty of
and
are given by
and
respectively The conditional uncertainty of Y given
is given by
for each.
The conditional uncertainty of
given
is the average uncertainty of
with the probabilities
is given by
Similarly, we can write the conditional uncertainty of
given
as
In case of three random variables
and
with their respective probability distributions, we have the following
measures of uncertainty
etc..
The following properties hold for the above uncertainty measures.
Property 1.38. We have

(i)

(ii)
Property 1.39. We have

(i)

(ii)
Property 1.40. We have

(i) ,
with equality iff
and
are independent i.e.,

(ii) ,
with equality iff
and
are conditionally independent given
i.e.,
and each .

(iii) ,
with equality iff
and
are independent i.e.,
Note 1.2. Since the random variables
and
are symmetric among them, then from the property 1.40(ii), we can write
Property 1.41. We have

(i)

(ii)

(iii)

(iv)
Property 1.42. We have

(i) ,
with equality iff
and
are independent i.e.,

(ii) ,
with equality iff ,
and
are independent i.e., iff

(iii) ,
with equality iff
and
are conditionally independent given
i.e., iff
Property 1.43. We have

(i)

(ii) If ,
then
Property 1.44. For each k, define
Then
Property 1.45. Let .
Then
Note 1.3. The property 1.45 is famous as "Fanoinequality".
For four discrete random variables
the following property holds.
Property 1.46. We have

(i)

(ii)

(iii)

(iv)

(v)

(vi)

(vii)

(viii)
21062001
Inder Jeet Taneja
Departamento de Matemática  UFSC
88.040900 Florianópolis, SC  Brazil