Howtosignalprocessing
- Is Kullback-Leibler a distance?
- What is the range of KL divergence?
- Can KL divergence be used as a distance measure?
- How is Kullback-Leibler calculated?
Is Kullback-Leibler a distance?
The Kullback-Leibler divergence between two probability distributions is a measure of how different the two distributions are. It is sometimes called a distance, but it's not a distance in the usual sense because it's not symmetric.
What is the range of KL divergence?
If two distributions perfectly match, D_KL (p||q) = 0 otherwise it can take values between 0 and ∞. Lower the KL divergence value, the better we have matched the true distribution with our approximation.
Can KL divergence be used as a distance measure?
Although the KL divergence measures the “distance” between two distri- butions, it is not a distance measure. This is because that the KL divergence is not a metric measure. It is not symmetric: the KL from p(x) to q(x) is generally not the same as the KL from q(x) to p(x).
How is Kullback-Leibler calculated?
KL divergence can be calculated as the negative sum of probability of each event in P multiplied by the log of the probability of the event in Q over the probability of the event in P. The value within the sum is the divergence for a given event.
