WAIC: WAIC: Widely Applicable Information Criterion

In TreeBUGS: Hierarchical Multinomial Processing Tree Modeling

WAIC: Widely Applicable Information Criterion

Description

Implementation of the WAIC for model comparison.

Usage

WAIC(
  fittedModel,
  n.adapt = 1000,
  n.chains = 3,
  n.iter = 10000,
  n.thin = 1,
  summarize = FALSE
)

## S3 method for class 'waic'
print(x, ...)

## S3 method for class 'waic_difference'
print(x, ...)

## S3 method for class 'waic'
e1 - e2

Arguments

fittedModel	fitted latent-trait or beta MPT model (traitMPT, betaMPT)
n.adapt	number of adaptation samples.
n.chains	number of chains (no parallel computation).
n.iter	number of iterations after burnin.
n.thin	Thinning rate.
summarize	deprecated argument only available for backwards compatibility
x	An object of class waic or waic_difference to be printed.
...	Further arguments that may be passed to print methods.
e1, e2	Two objects of class waic to be compared.

Details

WAIC provides an approximation of predictive accuracy with respect to out-of-sample deviance. The uncertainty of the WAIC for the given number of observed nodes (i.e., number of free categories times the number of participants) is quantified by the standard error of WAIC "se_waic" (cf. Vehtari et al., 2017). In contrast, to assess whether the approximation uncertainty due to MCMC sampling (not sample size) is sufficiently low, it is a good idea to fit each model twice and compute WAIC again to assess the stability of the WAIC values.

For more details, see Vehtari et al. (2017) and the following discussion about the JAGS implementation (which is currently an experimental feature of JAGS 4.3.0):

https://sourceforge.net/p/mcmc-jags/discussion/610036/thread/8211df61/

Value

Function WAIC() returns an object of class waic, which is basically a list containing three vectors p_waic, deviance, and waic, with separate values for each observed node (i.e., for all combinations of persons and free categories).

For these objects, a print() method exists, which also calculates the standard error of the estimate of WAIC.

For backwards compatibility, if WAIC() is called with summarize = TRUE, a vector with values p_waic, deviance, waic, and se_waic is returned.

WAIC values from two models can be compared by using the - operator; the result is an object of class waic_difference.

References

Vehtari, A., Gelman, A., & Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27(5), 1413–1432. doi:10.1007/s11222-016-9696-4

Examples

## Not run: 

#### WAIC for a latent-trait MPT model:
fit <- traitMPT(...)
WAIC(fit)


#### pairwise comparison of two models:

# (1) compute WAIC per model
waic1 <- WAIC(fit1)
waic2 <- WAIC(fit2)

# (2) WAIC difference
waic1 - waic2

## End(Not run)

TreeBUGS documentation built on May 31, 2023, 9:21 p.m.