seq_b_hierarchical_test: Sequential Bayesian hierarchical correlated t-test

In RebeccaGroh/seqbtests: Sequential Bayesian Comparison of ML Algorithm Performances

Description

This function implements a sequential approach of the Bayesian hierarchical test to compare the performance of machine learning algorithms to one another. Sample size is not fixed in advance, data are evaluated as they are collected. Further sampling is stopped in accordance with a pre-defined stopping rule.

Usage

seq_b_hierarchical_test(
  baseline,
  algorithm = NULL,
  measure = NULL,
  compare = NULL,
  rho = 0.1,
  max_repls = 20,
  rope = c(-0.01, 0.01),
  std.upper = 1000,
  d0.lower = NULL,
  d0.upper = NULL,
  alpha.lower = 0.5,
  alpha.upper = 5,
  beta.lower = 0.05,
  beta.upper = 0.15,
  nsim = 2000,
  nchains = 8,
  parallel = TRUE,
  stan.output.file = NULL,
  prob = 0.95,
  seed = as.numeric(Sys.time()),
  min_repls = 5,
  adapt_delta = 0.8,
  max_treedepth = 10,
  ...
)

Arguments

baseline	('character') First algorithm. Value in 'algorithm' column.
algorithm	('character') Second algorithm. Value in 'algorithm' column. If not defined, the baseline is tested against all algorithms in the data frame.
measure	('character') Name of the 'measure' column. If not defined, the first 'measure' column in the data frame is used.
compare	('character') Defines if one algorithm needs to perform better (better) for decisions based on the posterior distribution or whether it is sufficient to perform not worse (equal).
rho	('double') Correlation factor. Default is 0.1.
max_repls	('double') Maximum number of replications. If a complete data frame is passed (max_repls) should correspond to the maximum number of replications that are built. Default is 20.
rope	('double') Region of practical equivalence. Default is c(-0.01, 0.01).
std.upper	('double') Factor to set the upper bound for both sigma_i and sigma_0. Default is 1000.
d0.lower	('any') Lower bound for the prior for mu_0. If not provided, the smallest observed difference is used.
d0.upper	('any') Upper bound for the prior for mu_0. If not provided, the biggest observed difference is used.
alpha.lower	('double') Lower bound for the (uniform) prior for the alpha hyperparameter. Default is 0.5.
alpha.upper	('double') Upper bound for the (uniform) prior for the alpha hyperparameter. Default is 0.5.
beta.lower	('double') Lower bound for the (uniform) prior for the beta hyperparameter. Default is 0.5.
beta.upper	('double') Upper bound for the (uniform) prior for the beta hyperparameter. Default is 0.5.
nsim	('double') Number of samples (per chain) used to estimate the posterior distribution. Note that, by default, half the simulations are used for the burn-in.
nchains	('double') Number of MC chains to be simulated. As half the simulations are used for the warm-up, the total number of simulations will be nchain*nsim/2.
parallel	('logical') If true, Stan code is executed in parallel.
stan.output.file	('character') String containing the base name for the output files produced by Stan. If NULL, no files are stored.
prob	('double') Threshold probability that decision rely on. Default is 0.95.
seed	('double') Optional parameter used to fix the random seed.
min_repls	('double') Minimum number of replications that is used/generated before an optional stopping rule is activated. Default is 5.
adapt_delta	('double') Average proposal acceptance probability during Stan’s adaptation period.
max_treedepth	('double') Maximum treedepth parameter.
...	(any) Additional arguments for (get_replication). To pass a complete data frame, set (df) ('character').

Details

The basis for this test has first been implemented in scmamp.

Value

('list')
A list containing the following components:

• measure ('character')
  A string with the name of the measure column.

• method ('character')
  A string with the name of the method.

• baseline ('character')
  A string with the name of the first algorithm. Value in 'algorithm' column.

• data_frame ('list')
  A list containing the following components:

• algorithm ('character')
  Second algorithm. Value in 'algorithm' column. If not defined, the baseline is tested against all algorithms in the data frame.

• left ('double')
  Left probability.

• rope ('double')
  Rope probability.

• right ('double')
  Right probability.

• repls ('double')
  Number of evaluated replications.

• probabilities ('character')
  Decisions based on posterior probabilities and threshold probability.

References

https://github.com/b0rxa/scmamp

Examples

    ## Not run: 
    results <- seq_b_hierarchical_test(df = test_benchmark_small, 
      baseline = "algo_1", algorithm = "algo_3", max_repls = 10, 
      min_repls = 8)
    
## End(Not run)

RebeccaGroh/seqbtests documentation built on Nov. 17, 2021, 8:50 a.m.