Entropy $S(x)$ is the measure of randomness of a variable $x$. It is important in the area of information theory, which, on the other hand, shares similarities with the entropy that we have in thermodynamics. We write entropy as
$$S(x) =\sum -p(x) \log p(x),$$
here $p(x)$ is the probability mass distribution of the variable. In quantum information theory (or quantum Shannon theory), we use discrete matrices in the place of mass distribution. We mostly prefer the logarithms in base 2 and the entropy is measured in bits.
Suppose that Alice has sent a message which contains either $a$ or $b$. There
is half-chance probability occurring of either. In this case, the binary
entropy looks like the below figure, where when $p=1/2$ and $(1-p)=1/2$ the
entropy becomes $1$ bit,
$$S(x)=-p(x)\log p(x) - (1-p) \log (1-p),$$
For more than one variable, we have joint entropy
$$S(x,y) = -\sum_{x}\sum_{y} p(x,y) \log p(x,y)$$
If, for instance, Alice sends a message consisted of strings a and b
$$ababcbcbcba$$
then the messaged received by Bob is given by conditional entropy which is given by conditional probability
$P_{x \mid y}\left(x_{i} \mid y_{j}\right)=\frac{P_{x, y}\left(x_{i}, y_{j}\right)}{P_{y}\left(y_{j}\right)}$
and (we change the notation a bit, calling $X,Y$ random variable)
$$I(X; Y)=\sum_X \sum_Y p(x,y) \log \frac{p(x,y)}{p(x)p(y)} = S_{X}-S_{X Y}+S_{Y}$$
is the mutual information between two variables $X,Y$. The mutual information ($I(X;Y)$) is given by the relative entropy of the joint probability mass function and the product distribution given by $p(x)p(y)$ (I will recommend T. Cover and J. Thomas, Elements Of Information Theory. John Wiley Sons, 2006, for introductory materials.)
 |
| A look at general $x\log x$. |
You are learning information theory? It is great!!!!!!!!
Maanu what is exitcing in information theory?
Ashish, information theory is a wide subject which is used in physics, computer science, engineering, biology and many other subjects. It is eclectic. It is exciting in sense that you could study how information travels, how to make it more efficient and how to make it error-free. Read Thomas and Cover's book on information theory.