Go to:Table of Contents

Unified (r,s)-Mutual Information

In this section, we present six different ways to define the unified $(r,s)-$mutual information.

Let us consider

$$^\alpha{\ensuremath{\boldsymbol{\mathscr{F}}}}^s_r(X\wedge Y)={\e......lpha{\ensuremath{\boldsymbol{\mathscr{H}}}}^s_r(X\vert Y),\,\, \alpha=1,2,3\,\, \,\, 4,$$ (6.6)

and

$$^\alpha{\ensuremath{\boldsymbol{\mathscr{F}}}}^s_r(X\wedge Y\vert......ha{\ensuremath{\boldsymbol{\mathscr{H}}}}^s_r(X\vert Y,Z), \,\, \alpha=1,2,\,\, \,\, 3,$$ (6.7)

where

$$^\alpha{\ensuremath{\boldsymbol{\mathscr{F}}}}^s_r=\left\{\begin{...... ^\alpha{\ensuremath{\boldsymbol{\mathscr{F}}}},& r=1,\,s=1\end{array}\right.$$ (6.8)

and  $$^\alpha{\ensuremath{\boldsymbol{\mathscr{F}}}}^s_r(.\wedge.)={\en......H}}}}^s_r(.)-\ ^\alpha{\ensuremath{\boldsymbol{\mathscr{H}}}}^s_r(.\vert.),\$$    etc.$$$$

 We call the measures $^\alpha{\ensuremath{\boldsymbol{\mathscr{F}}}}^s_r(X\wedge Y)$ ($\alpha =1,2,3$ and $4$) the unified $(r,s)-$mutual information among the random variables $X$ and $Y$. The measures $^\alpha{\ensuremath{\boldsymbol{\mathscr{F}}}}^s_r(X\wedge Y\vert Z)$ ($\alpha=1,2$ and $3$) we call the unified $(r,s)-$mutual information among the random variables $X$ and $Y$ given $Z$.

Property 6.14. We have

(i) $^\alpha{\ensuremath{\boldsymbol{\mathscr{F}}}}^s_r(X\wedge Y)\geq 0$(resp. $\leq 0$) ($\alpha =1,2,3$ and $4$), with equality iff $X$ and $Y$ are independent.

(ii) $^\alpha{\ensuremath{\boldsymbol{\mathscr{F}}}}^s_r(X\wedge Y\vert Z)\geq 0$(resp. $\leq 0$) ($\alpha=1,2$ and $3$), with equality iff $X$ and $Y$ are independent given $Z$.

The above property 6.12 holds under the following conditions:

($c_1$) For $\alpha =1$, $r>0$ with $s\geq r$ or $s \geq 2-\frac{1}{r}$ (resp. $r<0$ with$s\leq 1$ or $1<s\leq 2-\frac{1}{r}$);

($c_2$) For $\alpha=2$ and 3, $r>0,\ s \in (-\infty ,\infty)$ (resp. $r<0$ $s \in (-\infty,\infty)$);

($c_3$) For $\alpha=4$, $r\geq s\geq 2-1/r\geq 1$ (resp. $s \leq r < 0$) (only for part(i)).

Note 6.4. Similar to Shannon's entropy (property 1.51) we left for the readers to check the validity of the following inequality:

$$^\alpha{\ensuremath{\boldsymbol{\mathscr{F}}}}_r^s(X,Y \wedge Z) ......a{\ensuremath{\boldsymbol{\mathscr{F}}}}_r^s(Y,Z)\,\,(\alpha=1,2 \, and \,3),$$

i.e., to find the conditions on the parameters $r$ and $s$ for the validity of the above expression.

Property 6.15. We have

$$^\alpha{\ensuremath{\boldsymbol{\mathscr{F}}}}^s_r(X\wedge Z)+\ ^......suremath{\boldsymbol{\mathscr{F}}}}^s_r(X\wedgeZ\vert Y),\,\, \alpha = 1,2\,\, \,\,3.$$ (6.9)

Note 6.5. The relation (6.9) is famous as "additive property". It also hold when $\alpha=4$, but at moment, we are not aware of the conditons on $r$ and $s$ for which$^4{\ensuremath{\boldsymbol{\mathscr{F}}}}_r^s(X \wedge Y\vert Z)$ is nonnegative, but the particular case when $r=s$ is discussed in section 6.4

For Simplicity, let us write 

$$P_{XY} =\{p(x_i,y_j)\} \,\,$$and$$\,\,P_X \times P_Y = \{p(x_i)p(y_j)\}$$ forall $i=1,2,...,n; j=1,2,...,m$.

We have 

$$F(X \wedge Y) = H(X) - H(X\vert Y)$$ $$=\sum_{i=1}^n \sum_{j=1}^m p(x_i,y_j)\log\left(\frac{p(x_i,y_j)}{p(x_i) p(y_j)}\right) = D\left(P_{XY}\vert\vert P_X \wedge P_Y\right),$$

where $D(P\vert\vert Q)$ is as given in (2.1)

We can write 

$$F(X \wedge Y) = \sum_{i=1}^m p(y_j)D(P_{X\vert Y=y_j}\vert\vert P_X),$$ where  $$D(P_{X\vert Y=y_j}\vert\vert P_X) = \sum_{i=1}^np(x_i\vert y_j) \log\left(\frac{p(x_i\vert y_j)}{p(x_i)}\right),$$ for all$j=1,2,\cdots,m.$

Thus 

$$F(X \wedge Y) = D(P_{X\vert Y=y_j}\vert\vert P_X) =\sum_{i=1}^n p(y_j) D\left(P_{X\vert Y=j}\vert\vert P_X\right).$$ The two more ways to define the unified $(r,s)-$mutual information are given by

$$^5F_r^s(X \wedge Y) = \sum_{i=1}^n p(y_j) {\ensuremath{\boldsymbol{\mathscr{D}}}}_r^s(P_{X\vert Y=y_j}\vert\vert P_X),$$ (6.10)

$$^6F_r^s(X \wedge Y) = {\ensuremath{\boldsymbol{\mathscr{D}}}}_r^s(P_{XY}\vert\vert P_X \times P_Y),$$ (6.11)

where $D_r^s(P\vert\vert Q)$ is as given by (4.1).

Property 6.16. For $\alpha = 5$ and $6$, we have

(i) $^\alpha{\ensuremath{\boldsymbol{\mathscr{F}}}}^s_r(X\wedge Y) = \, ^\alpha{\ensuremath{\boldsymbol{\mathscr{F}}}}^s_r(Y\wedge X);$

(ii) $^\alpha{\ensuremath{\boldsymbol{\mathscr{F}}}}^s_r(X\wedge Y)\left\{\begin{array}{ll}\geq 0, & r>0\\ \leq 0, & r<0\end{array}\right.$ with equality iff $X$ and $Y$ are independent.

(iii) $^5{\ensuremath{\boldsymbol{\mathscr{F}}}}^s_r(X\wedge Y)\left\{\begin{array}{l......math{\boldsymbol{\mathscr{F}}}}^s_r(X\wedge Y), & s\leq r\end{array}\right.$

Note 6.6. If we consider the fifth and sixth way of unified $(r,s)-$conditional entropies in the following way

$$^\alpha{\ensuremath{\boldsymbol{\mathscr{H}}}}_r^s(X\vert Y) = {\......th{\boldsymbol{\mathscr{F}}}}_r^s(X \wedge Y), \,\, (\alpha = 5 \,\,and\,\, 6),$$ (6.12)

then, we don't know for what values of $r$ and $s$ the measures given in (6.12) turn out to be nonnegative.

Note 6.7. Some of the above generalized mutual information measures can be connected to unified$(r,s)-$divergence measures given in Chapter 5. These connections are as follows:

In (5.12), take $\lambda_j = p(y_j)$ and $p_{ij}= p(x_i\vert y_j)$, we get 

$$^1I_r^s(P_1,P_2,\cdots,P_M) = \,\,^5{\ensuremath{\boldsymbol{\mathscr{F}}}}_r^s(X \wedge Y).$$

Similarly, we can write

$$^2I_r^s(P_1,P_2,\cdots,P_M) = \,\,^6{\ensuremath{\boldsymbol{\mathscr{F}}}}_r^s(X \wedge Y).$$

$$^3I_r^s(P_1,P_2,\cdots,P_M) = \,\,^1{\ensuremath{\boldsymbol{\mathscr{F}}}}_r^s(X \wedge Y).$$

In the notes 6.4 and 6.5, we have seen that there are difficulties in getting certain values of $r$ and $s$ for which the measures$^\alpha{\ensuremath{\boldsymbol{\mathscr{F}}}}_r^s(X \wedge Y\vert Z)$ or $^\alpha{\ensuremath{\boldsymbol{\mathscr{H}}}}_r^s(X\vert Y)\,(\alpha=5$ and $6$) become nonnegative. But, in the particular case, when $r=s$, the above difficulties are overcomed. The particular case for $\alpha=4$ is discussed in the following subsection.