Next:Distance MeasuresUp:Relative Information and Inaccuracy
Previous:Relative Information and Inaccuracy Go to:Table of Contents

Properties of Information Measures


Some properties of the measures (2.1) and (2.2) are presented in this subsection. For simplicity, let us denote$ D(P\vert\vert Q)=\ ^1{\ensuremath{\boldsymbol{\mathscr{L}}}}(P\vert\vert Q)\ $   and$ \ H(P\vert\vert Q) =\ ^2{\ensuremath{\boldsymbol{\mathscr{L}}}}(P\vert\vert Q)$.

Property 2.1. (Continuity).$ ^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(P\vert\vert Q)$$ (\alpha=1 \ $and$ \ 2)$ are continuous functions of the pair $ (P,Q)$.

Property 2.2. (Symmetry).$ ^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(P\vert\vert Q)\ \ (\alpha=1 \ $and$ \ 2)$ are symmetric functions of their arguments in pair $ (P,Q)$, i.e., for$ \alpha =1$ and 2, we have 

$\displaystyle ^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}\big(p_{\tau(1)},......suremath{\boldsymbol{\mathscr{L}}}}\big(p_1,...,p_n\vert\vert q_1,...,q_n\big),$
where$ \tau$ is any permutation from $ 1$ to $ n$.

Property 2.3. (Expansibility). For$ \alpha =1$ and 2, we have 

$\displaystyle ^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(p_1,...,p_n,0\ve......pha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(p_1,...,p_n\vert\vert q_1,...,q_n).$

Property 2.4. (Additivity). For $ \alpha =1$ and 2, we have 

$\displaystyle ^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(P_1*P_2\vert\ver......\vert Q_1)+^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(P_2\vert\vert Q_2).$

where $ P_1,Q_1\ \in\ \Delta_n,\ P_2,Q_2\ \in \\Delta_m.$

Property 2.5. (Sum representation). We can write 

$\displaystyle ^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(P\vert\vert Q)=\sum_{i=1}^n{f_{\alpha}(p_i,q_i)}, \,(\alpha=1\ $   and$\displaystyle \ 2)$
where 
$\displaystyle f_1(p,q)=p\ \log {p\over q},$
and
$\displaystyle f_2(p,q)=-p\ \log q,$
for all $ p,q\ \in \ [0,1].$

Property 2.6. (Nonnegativity).$ ^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(P\vert\vert Q) \geq 0\ (\alpha =1 \ $   and$ \ 2)$, with equality iff $ P=Q$ for $ \alpha =1$ and$ P=Q=P^0=(0,0,...,0,1,0,...,0)$$ \in\ \Delta_n$ for $ \alpha=2$.

Property 2.7. (Recursivity). For$ \alpha =1$ and $ 2$, we have 

$ ^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(p_1,...,p_n\vert\vert q_1,.........th{\boldsymbol{\mathscr{L}}}}(p_1+p_2,p_3,...,p_n\vert\vert q_1+q_2,q_3...,q_n)$

$\displaystyle +(p_1+p_2)\^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}\Big......+p_2},{p_2\over p_1+p_2}\vert\vert{q_1\over q_1+q_2},{q_2\over q_1+q_2}\Big),$
$ p_1+p_2>0,\ \ q_1+q_2>0,\ n\geq 2,\ \alpha=1$ and $ 2$.

Property 2.8. (Strongly additive). For$ \sum_{i=1}^n{\sum_{j=1}^m{ p_{ij}}}$ =$ \sum_{i=1}^n{\sum_{j=1}^m{q_{ij}}}=1$$ p_i=\sum_{j=1}^m{p_{ij}}>0,$$ \forall \ i$,$ q_i=\sum_{j=1}^m{q_{ij}}>0$$ \forall \ i$, we have

$ ^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(p_{11},...,p_{nm}\vert\vert q......lpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(p_1,...,p_n\vert\vert q_1,...,q_n)$

$\displaystyle + \, \sum_{i=1}^n{p_i}\ ^{\alpha}{\ensuremath{\boldsymbol{\mathsc......t {q_{i1}\over q_i},...,{q_{im}\overq_i}\Big)\,\, (\alpha=1 \ \mbox{and}\ 2).$

Property 2.9. (Functional equation). Let

$\displaystyle ^{\alpha}\psi(x,y)=\ ^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(x,1-x\vert\vert y,1-y),$
then for$ \alpha =1$ and 2, we have
(i) $ ^{\alpha}\psi(x_1,y_1)+(1-x_1)\ ^{\alpha}\psi\left(\frac{x_2}{1-x_1},\frac{y_2}{1-y_1}\right)=$ 
$\displaystyle =\^{\alpha}\psi(x_2,y_2)+(1-x_2)\ ^{\alpha}\psi\left(\frac{x_1}{1-x_2},\frac{y_1}{1-y_2}\right),$
for all$ x_1,y_1,x_2,y_2\ \in\ [0,1)$ with $ x_1+x_2 \leq 1,\ y_1+y_2 \leq1$.
(ii) $ ^{\alpha}{\ensuremath{\boldsymbol{\mathscr{L}}}}(P\vert\vert Q)=\sum_{t=2}^n\e......ta_t}\vert\vert\frac{\zeta_{t-1}}{\zeta_t},\frac{\zeta_{t-1}}{\zeta_t} \big),$
where $ \eta_t=p_1+p_2+...+p_t,\ \zeta_t=q_1+q_2+...+q_t,\t=2,3,...,n.$

Property 2.10. (Parallelogram identity). For any $ P$,$ Q$,$ S\ \in$$ \Delta_n$, we have

$\displaystyle D(P\vert\vert S)+D(Q\vert\vert S)=D\big(P\vert\vert{P+Q\over 2}\big)+D\big(Q\vert\vert{P+Q\over2}\big)+ 2D\big({P+Q\over 2}\vert\vert S\big).$
Property 2.11. (Convexity). We have
(i) $ D(P\vert\vert Q)$ is a convex function in a pair of distributions $ (P,Q)$ $ \ \in\ \Delta_n\times\Delta_n$.
(ii) $ H(P\vert\vert Q)$ is a convex function in $ Q\ \in\ \Delta_n$.
Property 2.12. (Data processing inequality). Let
$\displaystyle P(B)=\Big(\sum_{i=1}^n{p_ib_{i1}},...,\sum_{i=1}^n{p_ib_{im}}\Big)\\in\ \Delta_m,$
and
$\displaystyle Q(B)=\Big(\sum_{i=1}^n{q_ib_{i1}},...,\sum_{i=1}^n{q_ib_{im}}\Big)\\in\ \Delta_m,$
be two probability distributions, where$ B=\{b_{ij}\}$$ b_{ij}\geq 0$$ \forall \ i=1,2,...,n$;$ j=1,2,...,m$ is a stochastic matrix such that$ \sum_{j=1}^n{b_{ij}}=1$$ \forall \ i=1,2,...,n$. Then
$\displaystyle D\big(P(B)\vert\vert Q(B)\big)\leq D(P\vert\vert Q).$
Property 2.13. (Schur-convexity). We have
(i) $ D(P\vert\vert Q)$ is a Schur-convex function in the pair $ (P,Q)$.
(ii) $ H(P\vert\vert Q)$ is a Schur-convex function in $ Q$.
Property 2.14. (Inequality). We have
$\displaystyle \sum_{i=1}^n{\big(\sqrt{p_i}-\sqrt{q_i}\big)^2}\leq{D(P\vert\vert Q)\over \log e}\leq \sum_{i=1}^n{ {(p_i-q_i)^2\over q_i}},$
with equality iff $ P=Q$.

21-06-2001
Inder Jeet Taneja
Departamento de Matemática - UFSC
88.040-900 Florianópolis, SC - Brazil