hbmc: hbmc: Check Model Goodness of Fit and Prior Sensitivity

In hbsaems: Hierarchical Bayes Small Area Estimation Model using 'Stan'

hbmc: Check Model Goodness of Fit and Prior Sensitivity

Description

This function computes model fit diagnostics (WAIC, LOO) and performs posterior predictive checks for the primary model. If the 'model' argument is a list containing two or more brmsfit/hbmfit objects, it performs model comparison using selected metrics (LOO, WAIC, Bayes Factor) between the first model and each subsequent model in the list. Prior sensitivity analysis can be run for each model if specified. LOO calculation includes checks for problematic Pareto k values. If k > 0.7 and moment_match was FALSE, a warning is issued. If k > 0.7 and moment_match was TRUE, reloo is attempted. Additional arguments for moment matching and reloo can be passed via moment_match_args and reloo_args.

Usage

hbmc(
  model,
  ndraws_ppc = 100,
  moment_match = FALSE,
  moment_match_args = list(),
  reloo_args = list(),
  plot_types = c("pp_check", "params"),
  comparison_metrics = c("loo", "waic", "bf"),
  run_prior_sensitivity = FALSE,
  sensitivity_vars = NULL
)

Arguments

model	A single brmsfit/hbmfit object, or a list of brmsfit/hbmfit objects.
ndraws_ppc	Number of draws to plot in the posterior predictive check (default: 100).
moment_match	Logical; if TRUE, use moment matching for LOO computation directly. If FALSE (default), and problematic Pareto k values are detected, a warning will suggest re-running with moment_match = TRUE.
moment_match_args	A list of additional arguments to pass to brms::loo when moment_match = TRUE. Default is list().
reloo_args	A list of additional arguments to pass to brms::loo when reloo = TRUE is attempted. Default is list(). Note that k_threshold is already set to 0.7 internally when reloo is attempted.
plot_types	Character vector specifying types of plots to generate for the primary model (default: c("pp_check", "params")).
comparison_metrics	A character vector specifying which metrics to compute for model comparison when multiple models are provided. Options are "loo", "waic", "bf" (Bayes Factor). Default is c("loo", "waic", "bf") to compute all. If NULL or empty, no comparison metrics will be computed.
run_prior_sensitivity	Logical; if TRUE, attempt to run prior sensitivity analysis for each model (default: FALSE).
sensitivity_vars	A character vector of parameter names for which to run sensitivity analysis. Required if run_prior_sensitivity is TRUE.

Details

Check Model Goodness of Fit and Prior Sensitivity

Value

An object of class hbmc_results, which is a list containing:

primary_model_diagnostics	A list containing diagnostics for the primary model: loo: LOO diagnostics. May include notes about Pareto k values or reloo application. waic: WAIC value. pp_check_plot: Posterior predictive check plot. params_plot: Plot of marginal posterior distributions of parameters.
comparison_results	A list where each element contains specified comparison metrics (LOO, WAIC, Bayes Factor) for the primary model vs. another model. LOO results may include notes about Pareto k or reloo. NULL if fewer than two models or comparison_metrics is empty. Each element is named e.g., "model1_vs_model2".
prior_sensitivity_results	A list where each element (named by model name) contains results from prior sensitivity analysis if run_prior_sensitivity = TRUE. NULL otherwise. Each element in the list is itself a list with the following components: result A data frame or list summarizing the sensitivity analysis results, such as changes in posterior estimates or model fit metrics (e.g., WAIC, LOO). plot A plot object (usually a ggplot) visualizing the effect of different priors on model estimates or diagnostics. This plot can be displayed using print(). If no sensitivity was conducted or available, this field may contain a character message or be NULL.
models_compared_count	Integer, the number of models involved in comparison.

Author(s)

Achmad Syahrul Choir, Saniyyah Sri Nurhayati, and Arsyka Laila Oktalia Siregar

References

Buerkner, P. C. (2017). brms: An R package for Bayesian multilevel models using Stan. Journal of Statistical Software, 80(1), 1-28. Kallioinen, N., Paananen, T., Buerkner, PC. et al. Detecting and diagnosing prior and likelihood sensitivity with power-scaling. Statistical Computing, 34, 57 (2024). Gabry,J., Simpson,D., Vehtari, A., Betancourt, M., Gelman, A., Visualization in Bayesian Workflow, Journal of the Royal Statistical Society Series A: Statistics in Society, Volume 182, Issue 2, February 2019, Pages 389-402

Examples

# Load the hbsaems package if not already loaded (for hbm function)
# library(hbsaems) # Assuming hbm is part of this package
# library(brms) # For bf()

# Prepare a sample dataset
set.seed(123)
sim_data <- data.frame(
y = rnorm(100, 50, 10),
x1 = rnorm(100, 5, 2),
x2 = rnorm(100, 10, 3),
grp = factor(sample(1:10, 100, replace = TRUE))
)

# Define formulas and priors
f1 <- bf(y ~ x1 + (1 | grp))
common_priors <- c(prior(normal(0,10), class="b"), prior(cauchy(0,1), class="sd"))

# Fit models using hbm (minimal iterations for example speed)
fit_hbm1 <- try(hbm(f1, data=sim_data, prior=common_priors, family=gaussian(),
chains=1, iter=1000, warmup=500, cores=1, refresh=0, save_model=NULL))

if (!inherits(fit_hbm1, "try-error")) {
# Example: Explicitly request moment_match = TRUE with specific moment_match_args
# (e.g., if you want to pass 'k_threshold' to moment_match itself, though usually
# k_threshold is for reloo)
# For reloo, you might pass other args like 'psis_object' if precomputed.
checks_custom_args <- try(hbmc(
model = fit_hbm1,
moment_match = TRUE,
moment_match_args = list(k_threshold = 0.6), # Example arg for moment_match
reloo_args = list(check = FALSE), # Example arg for reloo
comparison_metrics = "loo"
))
# if (!inherits(checks_custom_args, "try-error")) {
# print(summary(checks_custom_args))
# }
}

hbsaems documentation built on Aug. 8, 2025, 7:28 p.m.