smc_evaluate: Calculates measures to evaluate the performance of the model

In audrey-b/simcases: Perform Simulation Studies Over Multiple Cases/Scenarios

Description

Calculates measures to evaluate the performance of the model The results are compared with the true parameter values by calculating performance measures such as bias, mean square error and coverage probability. R code can be used to customize the performance measures

Usage

smc_evaluate(
  cases,
  setup = NULL,
  measures = c("bias", "mse", "cpQuantile"),
  estimator = mean,
  alpha = 0.05,
  parameters = NULL,
  monitor = ".*",
  custom_funs = list(),
  custom_expr_before = "",
  custom_expr_after = "",
  path = ".",
  progress = FALSE,
  options = furrr::future_options(),
  environment = parent.frame(),
  fun = function(x) read.table(text = gsub(";|,| |:|\t|\\||&|~", "\t",
    readLines(textConnection(x))), header = TRUE),
  ...
)

Arguments

cases	An object that becomes a data frame when fun is applied. The first row of the data frame is a header and each subsequent row defines a case using strings. The strings must refer to objects defined within environment.
setup	An object that becomes a data frame when fun is applied. The first row of the data frame is a header and each subsequent row defines parameters to pass to simanalyse::sma_evaluate for each case using strings. The strings must refer to objects defined within environment.
measures	A vector of strings indicating which performance measures to calculate. Strings may include "bias", "E" (expectation), "cpQuantile" (coverage probability of quantile-based CrIs of level alpha), "LQuantile" (length of quantile-based CrIs of level alpha), "Epvar" (expected posterior variance), "Epsd" (expected posterior standard deviation), "rb" (relative bias), "br" (bias ratio), "var" (variance), "se" (standard error), "mse" (root mean square error), "rmse" (root mean square error), "rrmse" (relative root mean square error), "cv" (coefficient of variation), "all" (all the measures)
estimator	A function, typically mean or median, for the Bayes estimator to use to calculate the performance measures.
alpha	Scalar representing the alpha level used to construct credible intervals. Default is 0.05.
parameters	An nlist object (or list that can be coerced to nlist). True values of parameters to be used to calculate the performance measures.
monitor	A character vector (or regular expression if a string) specifying the names of the stochastic nodes in code to include in the summary. By default all stochastic nodes are included.
custom_funs	A named list of functions to calculate over the mcmc samples. E.g. list(posteriormedian = median).
custom_expr_before	A string of R code to derive custom measures. This code is used BEFORE averaging over all simulations. E.g. "mse = (posteriormedian - parameters)^2". Functions from custom_funs may be used as well as the keywords 'parameters' (the true values of the parameters) and 'estimator' (the estimator defined in estimator).
custom_expr_after	A string of R code to derive additional custom measures. This code is used AFTER averaging over all simulations. E.g. "rmse = sqrt(mse)". Measures calculated from custom_expr_before may be used as well as the keyword 'parameters' (the true values of the parameters).
path	A string specifying the path to the directory.
progress	A flag specifying whether to print a progress bar.
options	The future specific options to use with the workers.
environment	The environment in which the objects described in cases were defined.
fun	A function to convert models and cases to data frames. Default is a function that converts a string to a data frame.
...	Unused

Value

A flag.

Examples

normal <- "a ~ dnorm(0, 1/sigma^2)"
sigma1 <- list(sigma=1)
sigma2 <- list(sigma=2)
models_sims <- "code   parameters
                normal sigma1
                normal sigma2"
smc_simulate(models = models_sims,
                  path = tempdir(), 
                  exists = NA, 
                  ask = FALSE,
                  nsims = 2)
prior1 = "sigma ~ dunif(0, 4)"
prior2 = "sigma ~ dunif(0, 5)"
models_analysis <- "code    code.add
                    normal  prior1
                    normal  prior2"
cases <- "sims analyse
          1    1
          2    2"
smc_analyse(models = models_analysis,
                     cases = cases,
                     path = tempdir())
setup = "parameters
         sigma1
         sigma2"
smc_evaluate(cases, setup, path = tempdir(), monitor="sigma")

audrey-b/simcases documentation built on July 22, 2020, 6:20 p.m.