Adaptive Mixture of Studentt Distributions
Description
Function which performs the fitting of an adaptive mixture of Studentt distributions to approximate a target density through its kernel function
Usage
1 
Arguments
KERNEL 
kernel function of the target density on which the adaptive mixture is fitted. This
function should be vectorized for speed purposes (i.e., its first
argument should be a matrix and its output a vector). Moreover, the function must contain
the logical argument 
mu0 
initial value in the first stage optimization (for the location of
the first Studentt component) in the adaptive mixture, or
location of the first Studentt component if 
Sigma0 
scale matrix of the first Studentt component (square, symmetric and positive definite). Default:

control 
control parameters (see *Details*). 
... 
further arguments to be passed to 
Details
The argument KERNEL
is the kernel function of the target
density, and it should be vectorized for speed purposes.
As a first example, consider the kernel function proposed by GelmanMeng (1991):
k(x1,x2) = exp( 0.5*[A*x1^2*x2^2 + x1^2 + x2^2  2*B*x1*x2  2*C1*x1  2*C2*x2] )
where commonly used values are A=1, B=0, C1=3 and C2=3.
A vectorized implementation of this function might be:
1 2 3 4 5 6 7 8 9 10 11 
This way, we may supply a point (x1,x2)
for x
and the function will output a single value (i.e., the kernel
estimated at this point). But the function is vectorized, in the sense
that we may supply a Nx2 matrix
of values for x
, where rows of x
are
points (x1,x2) and the output will be a vector of
length N, containing the kernel values for these points.
Since the AdMit
procedure evaluates KERNEL
for a
large number of points, a vectorized implementation is important. Note
also the additional argument log = TRUE
which is used for
numerical stability.
As a second example, consider the following (simple) econometric model:
y_t ~ i.i.d. N(mu,sigma^2) t=1,...,T
where mu is the mean value and sigma is the standard deviation. Our purpose is to estimate theta=(mu,sigma) within a Bayesian framework, based on a vector y of T observations; the kernel thus consists of the product of the prior and the likelihood function. As previously mentioned, the kernel function should be vectorized, i.e., treat a (Nx2) matrix of points theta for which the kernel will be evaluated. Using the common (Jeffreys) prior p(theta)=1/sigma for sigma>0, a vectorized implementation of the kernel function might be:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28  KERNEL < function(theta, y, log = TRUE)
{
if (is.vector(theta))
theta < matrix(theta, nrow = 1)
## sub function which returns the logkernel for a given
## thetai value (i.e., a given row of theta)
KERNEL_sub < function(thetai)
{
if (thetai[2] > 0) ## check if sigma>0
{ ## if yes, compute the logkernel at thetai
r <  log(thetai[2])
+ sum(dnorm(y, thetai[1], thetai[2], TRUE))
}
else
{ ## if no, returns Infinity
r < Inf
}
as.numeric(r)
}
## 'apply' on the rows of theta (faster than a for loop)
r < apply(theta, 1, KERNEL_sub)
if (!log)
r < exp(r)
as.numeric(r)
}

Since this kernel function also depends on the vector y, it
must be passed to KERNEL
in the AdMit
function. This is
achieved via the argument ..., i.e., AdMit(KERNEL, mu = c(0, 1), y = y)
.
To gain even more speed, implementation of KERNEL
might rely on C or Fortran
code using the functions .C
and .Fortran
. An example is
provided in the file ‘AdMitJSS.R’ in the package's folder.
The argument control
is a list that can supply any of
the following components:
Ns
number of draws used in the evaluation of the importance sampling weights (integer number not smaller than 100). Default:
Ns = 1e5
.Np
number of draws used in the optimization of the mixing probabilities (integer number not smaller than 100 and not larger than
Ns
). Default:Np = 1e3
.Hmax
maximum number of Studentt components in the adaptive mixture (integer number not smaller than one). Default:
Hmax = 10
.df
degrees of freedom parameter of the Studentt components (real number not smaller than one). Default:
df = 1
.CVtol
tolerance for the relative change of the coefficient of variation (real number in [0,1]). The adaptive algorithm stops if the new component leads to a relative change in the coefficient of variation that is smaller or equal than
CVtol
. Default:CVtol = 0.1
, i.e., 10%.weightNC
weight assigned to the new Studentt component of the adaptive mixture as a starting value in the optimization of the mixing probabilities (real number in [0,1]). Default:
weightNC = 0.1
, i.e., 10%.trace
tracing information on the adaptive fitting procedure (logical). Default:
trace = FALSE
, i.e., no tracing information.IS
should importance sampling be used to estimate the mode and the scale matrix of the Studentt components (logical). Default:
IS = FALSE
, i.e., use numerical optimization instead.ISpercent
vector of percentage(s) of largest weights used to estimate the mode and the scale matrix of the Studentt components of the adaptive mixture by importance sampling (real number(s) in [0,1]). Default:
ISpercent = c(0.05, 0.15, 0.3)
, i.e., 5%, 15% and 30%.ISscale
vector of scaling factor(s) used to rescale the scale matrix of the mixture components (real positive numbers). Default:
ISscale = c(1, 0.25, 4)
.trace.mu
Tracing information on the progress in the optimization of the mode of the mixture components (nonnegative integer number). Higher values may produce more tracing information (see the source code of the function
optim
for further details). Default:trace.mu = 0
, i.e., no tracing information.maxit.mu
maximum number of iterations used in the optimization of the modes of the mixture components (positive integer). Default:
maxit.mu = 500
.reltol.mu
relative convergence tolerance used in the optimization of the modes of the mixture components (real number in [0,1]). Default:
reltol.mu = 1e8
.trace.p
,maxit.p
,reltol.p
the same as for the arguments above, but for the optimization of the mixing probabilities of the mixture components.
Value
A list with the following components:
CV
: vector (of length H) of coefficients of variation of
the importance sampling weights.
mit
: list (of length 4) containing information on the fitted mixture of
Studentt distributions, with the following components:
p
: vector (of length H) of mixing probabilities.
mu
: matrix (of size Hxd) containing the
vectors of modes (in row) of the mixture components.
Sigma
: matrix (of size Hxd*d) containing the scale
matrices (in row) of the mixture components.
df
: degrees of freedom parameter of the Studentt components.
where H (>=1) is the number of components in the adaptive
mixture of Studentt distributions and d (>=1) is
the dimension of the first argument in KERNEL
.
summary
: data frame containing information on the optimization
procedures. It returns for each component of the adaptive mixture of
Studentt distribution: 1. the method used to estimate the mode
and the scale matrix of the Studentt component (‘USER’ if Sigma0
is
provided by the user; numerical optimization: ‘BFGS’, ‘NelderMead’;
importance sampling: ‘IS’, with percentage(s) of importance weights
used and scaling factor(s)); 2. the time required for this optimization;
3. the method used to estimate the mixing probabilities
(‘NLMINB’, ‘BFGS’, ‘NelderMead’, ‘NONE’); 4. the time required for this
optimization; 5. the coefficient of variation of the importance
sampling weights.
Note
Further details and examples of the R package AdMit
can be found in Ardia, Hoogerheide, van Dijk (2009a,b). See also
the package vignette by typing vignette("AdMit")
and the
files ‘AdMitJSS.txt’ and ‘AdMitRnews.txt’ in the ‘/doc’ package's folder.
Further details on the core algorithm are given in Hoogerheide (2006), Hoogerheide, Kaashoek, van Dijk (2007) and Hoogerheide, van Dijk (2008).
The adaptive mixture mit
returned by the function AdMit
is used by the
function AdMitIS
to perform importance sampling using
mit
as the importance density or by the function AdMitMH
to perform
independence chain MetropolisHastings sampling using mit
as the
candidate density.
Please cite the package in publications. Use citation("AdMit")
.
Author(s)
David Ardia for the R port,
Lennart F. Hoogerheide and Herman K. van Dijk for the AdMit
algorithm.
References
Ardia, D., Hoogerheide, L.F., van Dijk, H.K. (2009a). AdMit: Adaptive Mixture of Studentt Distributions. The R Journal 1(1), pp.25–30. http://journal.rproject.org/20091/
Ardia, D., Hoogerheide, L.F., van Dijk, H.K. (2009b). Adaptive Mixture of Studentt Distributions as a Flexible Candidate Distribution for Efficient Simulation: The R Package AdMit. Journal of Statistical Software 29(3), pp.1–32. http://www.jstatsoft.org/v29/i03/
Gelman, A., Meng, X.L. (1991). A Note on Bivariate Distributions That Are Conditionally Normal. The American Statistician 45(2), pp.125–126.
Hoogerheide, L.F. (2006). Essays on Neural Network Sampling Methods and Instrumental Variables. PhD thesis, Tinbergen Institute, Erasmus University Rotterdam (NL). ISBN: 9051708261. (Book nr. 379 of the Tinbergen Institute Research Series.)
Hoogerheide, L.F., Kaashoek, J.F., van Dijk, H.K. (2007). On the Shape of Posterior Densities and Credible Sets in Instrumental Variable Regression Models with Reduced Rank: An Application of Flexible Sampling Methods using Neural Networks. Journal of Econometrics 139(1), pp.154–180. doi: 10.1016/j.jeconom.2006.06.009.
Hoogerheide, L.F., van Dijk, H.K. (2008). Possibly IllBehaved Posteriors in Econometric Models: On the Connection between Model Structures, Nonelliptical Credible Sets and Neural Network Simulation Techniques. Tinbergen Institute discussion paper 2008036/4. http://www.tinbergen.nl/discussionpapers/08036.pdf
See Also
AdMitIS
for importance sampling using an
adaptive mixture of Studentt distributions as the importance density,
AdMitMH
for the independence chain MetropolisHastings
algorithm using an adaptive mixture of Studentt distributions as
the candidate density.
Examples
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25  ## NB : Low number of draws for speedup. Consider using more draws!
## Gelman and Meng (1991) kernel function
GelmanMeng < function(x, A = 1, B = 0, C1 = 3, C2 = 3, log = TRUE)
{
if (is.vector(x))
x < matrix(x, nrow = 1)
r < .5 * (A * x[,1]^2 * x[,2]^2 + x[,1]^2 + x[,2]^2
 2 * B * x[,1] * x[,2]  2 * C1 * x[,1]  2 * C2 * x[,2])
if (!log)
r < exp(r)
as.vector(r)
}
## Run AdMit (with default values)
set.seed(1234)
outAdMit < AdMit(GelmanMeng, mu0 = c(0.0, 0.1), control = list(Ns = 1e4))
print(outAdMit)
## Run AdMit (using importance sampling to estimate
## the modes and the scale matrices)
set.seed(1234)
outAdMit < AdMit(KERNEL = GelmanMeng,
mu0 = c(0.0, 0.1),
control = list(IS = TRUE, Ns = 1e4))
print(outAdMit)
