Fano's inequality

In information theory, Fano's inequality (also known as the Fano converse and the Fano lemma) relates the average information lost in a noisy channel to the probability of the categorization error. It was derived by Robert Fano in the early 1950s while teaching a Ph.D. seminar in information theory at MIT, and later recorded in his 1961 textbook.

It is used to find a lower bound on the error probability of any decoder as well as the lower bounds for minimax risks in density estimation.

Let the random variables ${\displaystyle X}$ and ${\displaystyle Y}$ represent input and output messages with a joint probability ${\displaystyle P(x,y)}$. Let ${\displaystyle e}$ represent an occurrence of error; i.e., that ${\displaystyle X\neq {\tilde {X}}}$, with ${\displaystyle {\tilde {X}}=f(Y)}$ being an approximate version of ${\displaystyle X}$. Fano's inequality is

${\displaystyle H(X|Y)\leq H_{b}(e)+P(e)\log(|{\mathcal {X}}|-1),}$

where ${\displaystyle {\mathcal {X}}}$ denotes the support of ${\displaystyle X}$,

${\displaystyle H\left(X|Y\right)=-\sum _{i,j}P(x_{i},y_{j})\log P\left(x_{i}|y_{j}\right)}$

is the conditional entropy,

${\displaystyle P(e)=P(X\neq {\tilde {X}})}$

is the probability of the communication error, and

${\displaystyle H_{b}(e)=-P(e)\log P(e)-(1-P(e))\log(1-P(e))}$

is the corresponding binary entropy.