A Few Comments on Entropy

Entropy $S(x)$ is the measure of randomness of a variable $x$. It is important in the area of information theory, which, on the other hand, shares similarities with the entropy that we have in thermodynamics. We write entropy as

$$S(x) =\sum -p(x) \log p(x),$$

here $p(x)$ is the probability mass distribution of the variable. In quantum information theory (or quantum Shannon theory), we use discrete matrices in the place of mass distribution. We mostly prefer the logarithms in base 2 and the entropy is measured in bits. 

Suppose that Alice has sent a message which contains either $a$ or $b$. There is half-chance probability occurring of either. In this case, the binary entropy looks like the below figure, where when $p=1/2$ and $(1-p)=1/2$ the entropy becomes $1$ bit,

$$S(x)=-p(x)\log p(x) - (1-p) \log (1-p),$$

For more than one variable, we have joint entropy

$$S(x,y) = -\sum_{x}\sum_{y} p(x,y) \log p(x,y)$$

If, for instance, Alice sends a message consisted of strings a and b

$$ababcbcbcba$$

then the messaged received by Bob is given by conditional entropy which is given by conditional probability

$P_{x \mid y}\left(x_{i} \mid y_{j}\right)=\frac{P_{x, y}\left(x_{i}, y_{j}\right)}{P_{y}\left(y_{j}\right)}$

and (we change the notation a bit, calling $X,Y$ random variable)

$$I(X; Y)=\sum_X \sum_Y p(x,y) \log \frac{p(x,y)}{p(x)p(y)} = S_{X}-S_{X Y}+S_{Y}$$

is the mutual information between two variables $X,Y$. The mutual information ($I(X;Y)$) is given by the relative entropy of the joint probability mass function and the product distribution given by $p(x)p(y)$ (I will recommend T. Cover and J. Thomas, Elements Of Information Theory. John Wiley Sons, 2006, for introductory materials.)

A look at general $x\log x$.

This entry was posted in . Bookmark the post. Print it.

3 Responses to A Few Comments on Entropy

  1. Maanu says:

    You are learning information theory? It is great!!!!!!!!

  2. Ashish Kumar says:

    Maanu what is exitcing in information theory?

  3. Ashish, information theory is a wide subject which is used in physics, computer science, engineering, biology and many other subjects. It is eclectic. It is exciting in sense that you could study how information travels, how to make it more efficient and how to make it error-free. Read Thomas and Cover's book on information theory.

Leave a Reply