Go to:Table of Contents

Properties of Information Measures

Some properties of the measures (2.1) and (2.2) are presented in this subsection. For simplicity, let us denote$D(P\vert\vert Q)=\ ^1{\ensuremath{\boldsymbol{\mathscr{L}}}}(P\vert\vert Q)\$   and$\ H(P\vert\vert Q) =\ ^2{\ensuremath{\boldsymbol{\mathscr{L}}}}(P\vert\vert Q)$.

Property 2.1. (Continuity).$^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(P\vert\vert Q)$$(\alpha=1 \$and$\ 2)$ are continuous functions of the pair $(P,Q)$.

Property 2.2. (Symmetry).$^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(P\vert\vert Q)\ \ (\alpha=1 \$and$\ 2)$ are symmetric functions of their arguments in pair $(P,Q)$, i.e., for$\alpha =1$ and 2, we have 

$$^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}\big(p_{\tau(1)},......suremath{\boldsymbol{\mathscr{L}}}}\big(p_1,...,p_n\vert\vert q_1,...,q_n\big),$$ where$\tau$ is any permutation from $1$ to $n$.

Property 2.3. (Expansibility). For$\alpha =1$ and 2, we have 

$$^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(p_1,...,p_n,0\ve......pha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(p_1,...,p_n\vert\vert q_1,...,q_n).$$

Property 2.4. (Additivity). For $\alpha =1$ and 2, we have 

$$^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(P_1*P_2\vert\ver......\vert Q_1)+^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(P_2\vert\vert Q_2).$$

where $P_1,Q_1\ \in\ \Delta_n,\ P_2,Q_2\ \in \\Delta_m.$

Property 2.5. (Sum representation). We can write 

$$^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(P\vert\vert Q)=\sum_{i=1}^n{f_{\alpha}(p_i,q_i)}, \,(\alpha=1\$$   and$$\ 2)$$ where  $$f_1(p,q)=p\ \log {p\over q},$$ and $$f_2(p,q)=-p\ \log q,$$ for all $p,q\ \in \ [0,1].$

Property 2.6. (Nonnegativity).$^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(P\vert\vert Q) \geq 0\ (\alpha =1 \$   and$\ 2)$, with equality iff $P=Q$ for $\alpha =1$ and$P=Q=P^0=(0,0,...,0,1,0,...,0)$$\in\ \Delta_n$ for $\alpha=2$.

Property 2.7. (Recursivity). For$\alpha =1$ and $2$, we have 

$^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(p_1,...,p_n\vert\vert q_1,.........th{\boldsymbol{\mathscr{L}}}}(p_1+p_2,p_3,...,p_n\vert\vert q_1+q_2,q_3...,q_n)$

$$+(p_1+p_2)\^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}\Big......+p_2},{p_2\over p_1+p_2}\vert\vert{q_1\over q_1+q_2},{q_2\over q_1+q_2}\Big),$$ $p_1+p_2>0,\ \ q_1+q_2>0,\ n\geq 2,\ \alpha=1$ and $2$.

Property 2.8. (Strongly additive). For$\sum_{i=1}^n{\sum_{j=1}^m{ p_{ij}}}$ =$\sum_{i=1}^n{\sum_{j=1}^m{q_{ij}}}=1$, $p_i=\sum_{j=1}^m{p_{ij}}>0,$$\forall \ i$,$q_i=\sum_{j=1}^m{q_{ij}}>0$, $\forall \ i$, we have

$^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(p_{11},...,p_{nm}\vert\vert q......lpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(p_1,...,p_n\vert\vert q_1,...,q_n)$

$$+ \, \sum_{i=1}^n{p_i}\ ^{\alpha}{\ensuremath{\boldsymbol{\mathsc......t {q_{i1}\over q_i},...,{q_{im}\overq_i}\Big)\,\, (\alpha=1 \ \mbox{and}\ 2).$$

Property 2.9. (Functional equation). Let

$$^{\alpha}\psi(x,y)=\ ^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(x,1-x\vert\vert y,1-y),$$ then for$\alpha =1$ and 2, we have

(i) $^{\alpha}\psi(x_1,y_1)+(1-x_1)\ ^{\alpha}\psi\left(\frac{x_2}{1-x_1},\frac{y_2}{1-y_1}\right)=$ 

$$=\^{\alpha}\psi(x_2,y_2)+(1-x_2)\ ^{\alpha}\psi\left(\frac{x_1}{1-x_2},\frac{y_1}{1-y_2}\right),$$ for all$x_1,y_1,x_2,y_2\ \in\ [0,1)$ with $x_1+x_2 \leq 1,\ y_1+y_2 \leq1$.

(ii) $^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(P\vert\vert Q)=\sum_{t=2}^n\e......ta_t}\vert\vert\frac{\zeta_{t-1}}{\zeta_t},\frac{\zeta_{t-1}}{\zeta_t} \big),$

where $\eta_t=p_1+p_2+...+p_t,\ \zeta_t=q_1+q_2+...+q_t,\t=2,3,...,n.$

Property 2.10. (Parallelogram identity). For any $P$,$Q$,$S\ \in$$\Delta_n$, we have

$$D(P\vert\vert S)+D(Q\vert\vert S)=D\big(P\vert\vert{P+Q\over 2}\big)+D\big(Q\vert\vert{P+Q\over2}\big)+ 2D\big({P+Q\over 2}\vert\vert S\big).$$ Property 2.11. (Convexity). We have

(i) $D(P\vert\vert Q)$ is a convex function in a pair of distributions $(P,Q)$ $\ \in\ \Delta_n\times\Delta_n$.

(ii) $H(P\vert\vert Q)$ is a convex function in $Q\ \in\ \Delta_n$.

Property 2.12. (Data processing inequality). Let $$P(B)=\Big(\sum_{i=1}^n{p_ib_{i1}},...,\sum_{i=1}^n{p_ib_{im}}\Big)\\in\ \Delta_m,$$ and $$Q(B)=\Big(\sum_{i=1}^n{q_ib_{i1}},...,\sum_{i=1}^n{q_ib_{im}}\Big)\\in\ \Delta_m,$$ be two probability distributions, where$B=\{b_{ij}\}$, $b_{ij}\geq 0$, $\forall \ i=1,2,...,n$;$j=1,2,...,m$ is a stochastic matrix such that$\sum_{j=1}^n{b_{ij}}=1$, $\forall \ i=1,2,...,n$. Then $$D\big(P(B)\vert\vert Q(B)\big)\leq D(P\vert\vert Q).$$ Property 2.13. (Schur-convexity). We have

(i) $D(P\vert\vert Q)$ is a Schur-convex function in the pair $(P,Q)$.

(ii) $H(P\vert\vert Q)$ is a Schur-convex function in $Q$.

Property 2.14. (Inequality). We have $$\sum_{i=1}^n{\big(\sqrt{p_i}-\sqrt{q_i}\big)^2}\leq{D(P\vert\vert Q)\over \log e}\leq \sum_{i=1}^n{ {(p_i-q_i)^2\over q_i}},$$ with equality iff $P=Q$.