Mutual information

1. What is the meaning of mutual information?
2. How do you use mutual information?
3. What is the difference between correlation and mutual information?
4. Is mutual information Positive?

What is the meaning of mutual information?

Mutual information is a quantity that measures a relationship between two random variables that are sampled simultaneously. In particular, it measures how much information is communicated, on average, in one random variable about another.

How do you use mutual information?

Mutual information is often used as a general form of a correlation coefficient, e.g. a measure of the dependence between random variables. It is also used as an aspect in some machine learning algorithms.

What is the difference between correlation and mutual information?

Correlation analysis provides a quantitative means of measuring the strength of a linear relationship between two vectors of data. Mutual information is essentially the measure of how much “knowledge” one can gain of a certain variable by knowing the value of another variable.

Is mutual information Positive?

Mutual information is nonnegative, i.e. I(X;Y ) ≥ 0. Equivalently, H(X|Y ) ≤ H(X). Hence conditioning one random variable on another can only decrease entropy. Equality holds if and only if the random variables are independent.