# Kullback-Leibler Divergence

### Description

Estimate the Kullback-Leibler divergence of several distributions.

### Usage

1 2 3 4 |

### Arguments

`object` |
See Methods section below. |

`method` |
The method to be used; "continuous" determines the Kullback-Leibler divergence between the unweighted theoretical component distributions and the unweighted posterior probabilities at the observed points are used by "discrete". |

`eps` |
Probabilities below this threshold are replaced by this threshold for numerical stability. |

`overlap` |
Logical, do not determine the KL divergence for
those pairs where for each point at least one of the densities has a
value smaller than |

`...` |
Passed to the matrix method. |

### Details

Estimates

*\int f(x) (\log f(x) - \log g(x)) dx*

for distributions with densities *f()* and *g()*.

### Value

A matrix of KL divergences where the rows correspond to using the
respective distribution as *f()* in the formula above.

### Methods

- object = "matrix":
Takes as input a matrix of density values with one row per observation and one column per distribution.

- object = "flexmix":
Returns the Kullback-Leibler divergence of the mixture components.

### Note

The density functions are modified to have equal support.
A weight of at least `eps`

is given to each
observation point for the modified densities.

### Author(s)

Friedrich Leisch and Bettina Gruen

### References

S. Kullback and R. A. Leibler. On information and sufficiency.*The
Annals of Mathematical Statistics*, **22**(1), 79–86, 1951.

Friedrich Leisch. Exploring the structure of mixture model components. In Jaromir Antoch, editor, Compstat 2004–Proceedings in Computational Statistics, 1405–1412. Physika Verlag, Heidelberg, Germany, 2004. ISBN 3-7908-1554-3.

### Examples

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | ```
## Gaussian and Student t are much closer to each other than
## to the uniform:
x <- seq(-3, 3, length = 200)
y <- cbind(u = dunif(x), n = dnorm(x), t = dt(x, df = 10))
matplot(x, y, type = "l")
KLdiv(y)
if (require("mlbench")) {
set.seed(2606)
x <- mlbench.smiley()$x
model1 <- flexmix(x ~ 1, k = 9, model = FLXmclust(diag = FALSE),
control = list(minprior = 0))
plotEll(model1, x)
KLdiv(model1)
}
``` |