check.ess: Check whether chains adequately sampled the posterior...

In bstaggmartin/backwards-BM-simulator: Evolving Rate Models of Continuous Trait Evolution

Check whether chains adequately sampled the posterior distribution

Description

This function calculates and uses the bulk/tail effective sample size (ESS) diagnostics to check for adequate sampling of the posterior distribution given a fitted evorates model. Generally, ESS's should at least be greater than 100 and preferably greater than 100 times the number of chains. Notably, this function takes all chains into account to calculate ESS's–diagnostics param_block arrays instead calculate ESS's for each individual chain.

Usage

check.ess(fit, printlen = 6)

Arguments

fit	An object of class "evorates_fit".
printlen	Sets the maximum number of parameter names to report for each category of bulk/tail ESS values: "good", "okay", and "bad". For no maximum, set this to 0, a negative number, or infinity.

Details

ESS's are calculated using rstan::bulk_ess() and rstan::tail_ess(). More details on this diagnostic can be found in the associated documentation: Rhat. Even though 100 times the number of chains or above is ideal, if bulk/tail ESS's for all parameters are around 200 or more, the inferred posterior distribution should be pretty reliable in my experience. In any case, users should ensure make sure all the bulk ESS's are okay to good before trusting parameter point estimates (e.g., means or medians), and that all the tail ESS's are okay to good before trusting parameter uncertainty estimates (e.g., 95% credible intervals).

In the case of high model misspecification (i.e., an evorates model does not fit the data well), one could theoretically get low ess's due to extremely high autocorrelation of posterior samples, and running longer chains would thus yield little benefit. I have yet to see this situation, but it is certainly possible.

This function complements ESS calculation via grapes-diagnostics-grapes since it calculates ESS's for all parameters using all chains. It thus is tailored for diagnosing the "healthiness" of a fit as a whole, rather than the healthiness of chains for individual parameters/chains.

Value

Prints to console how many parameters exhibit "good" (ESS >= 100 times number of chains), "okay" (100 <= ESS < 100 times number of chains), and "bad" (ESS < 100) sampling of both the bulks (i.e., central tendency) and tails (i.e., "bounds" for lack of a better word) of the posterior distribution, along with some general advice depending on the worst observed bulk/tail ESS. If the number of parameters in a particular category is less than or equal to print.len, the function will also print the names of the parameter in that category.

Invisibly, the function also returns a list of all calculated bulk/tail ESS's, named by their corresponding parameters and split into good, okay, and bad categories.

See Also

Rhat for the functions used to calculate bulk/tail ESS's. Also, grapes-diagnostics-grapes for a function tailored towards diagnosing individual chains for specific parameters.

Other evorates diagnostic functions: check.mix()

Examples

#get whale/dolphin evorates fit
data("cet_fit")

#check how well the chains sampled the posterior
ess <- check.ess(cet_fit) #yay!
ess #invisibly returned object

#note the differences between...
ess; cet_fit %diagnostics% list(".", "ess")

#generally you should make sure all the ESS's are okay before trusting your parameter estimates
#especially in the case of bulk ESS!

#fit a new model (poorly--please never set your iterations this low in practice)
#this is just to see what happens with bad sampling
#this will take just a little bit to run...
fit <- fit.evorates(cet_fit$call$tree, cet_fit$call$trait.data,
                    constrain.Rsig2 = TRUE,
                    warmup = 1000, iter = 1040)
ess <- check.ess(fit) #not yay

bstaggmartin/backwards-BM-simulator documentation built on June 3, 2024, 5:51 p.m.