b_hierarchical_test: Bayesian hierarchical correlated t-test

In RebeccaGroh/seqbtests: Sequential Bayesian Comparison of ML Algorithm Performances

Description

This function implements a Bayesian hierarchical test. The performance of one baseline algorithm on multiple data set is compared to either one or multiple algorithms.

Usage

b_hierarchical_test(
  df,
  baseline,
  algorithm = NULL,
  measure = NULL,
  rho = 0.1,
  compare = NULL,
  std.upper = 1000,
  d0.lower = NULL,
  d0.upper = NULL,
  prob = 0.95,
  alpha.lower = 0.5,
  alpha.upper = 5,
  beta.lower = 0.05,
  beta.upper = 0.15,
  rope = c(-0.01, 0.01),
  nsim = 2000,
  parallel = TRUE,
  stan.output.file = NULL,
  nchains = 8,
  seed = as.numeric(Sys.time()),
  adapt_delta = 0.8,
  max_treedepth = 10
)

Arguments

df	('list') Data frame containing the performane measure.
baseline	('character') First algorithm. Value in 'algorithm' column.
algorithm	('character') Second algorithm. Value in 'algorithm' column. If not defined, the baseline is tested against all algorithms in the data frame.
measure	('character') Name of the 'measure' column. If not defined, the first 'measure' column in the data frame is used.
rho	('double') Correlation factor. Default is 0.1.
compare	('character') Defines if one algorithm needs to perform better (better) for decisions based on the posterior distribution or whether it is sufficient to perform not worse (equal).
std.upper	('double') Factor to set the upper bound for both sigma_i and sigma_0. Default is 1000.
d0.lower	('any') Lower bound for the prior for mu_0. If not provided, the smallest observed difference is used.
d0.upper	('any') Upper bound for the prior for mu_0. If not provided, the biggest observed difference is used.
prob	('double') Threshold probability that decision rely on. Default is 0.95.
alpha.lower	('double') Lower bound for the (uniform) prior for the alpha hyperparameter. Default is 0.5.
alpha.upper	('double') Upper bound for the (uniform) prior for the alpha hyperparameter. Default is 0.5.
beta.lower	('double') Lower bound for the (uniform) prior for the beta hyperparameter. Default is 0.5.
beta.upper	('double') Upper bound for the (uniform) prior for the beta hyperparameter. Default is 0.5.
rope	('double') Region of practical equivalence. Default is c(-0.01, 0.01).
nsim	('double') Number of samples (per chain) used to estimate the posterior distribution. Note that, by default, half the simulations are used for the burn-in.
parallel	('logical') If true, Stan code is executed in parallel.
stan.output.file	('character') String containing the base name for the output files produced by Stan. If NULL, no files are stored.
nchains	('double') Number of MC chains to be simulated. As half the simulations are used for the warm-up, the total number of simulations will be nchain*nsim/2.
seed	('double') Optional parameter used to fix the random seed.
adapt_delta	('double') Average proposal acceptance probability during Stan’s adaptation period.
max_treedepth	('double') Maximum treedepth parameter.

Details

The test has first been implemented in scmamp. Note that if no measure column is defined per default the first column defined as measure_* in the data frame is used. The default of rho is 0.1.

Value

('list')
A list containing the following components:

• measure ('character')
  A string with the name of the measure column.

• method ('character')
  A string with the name of the method.

• baseline ('character')
  A string with the name of the first algorithm. Value in 'algorithm' column.

• data_frame ('list')
  A list containing the following components:

• algorithm ('character')
  Second algorithm. Value in 'algorithm' column. If not defined, the baseline is tested against all algorithms in the data frame.

• left ('double')
  Left probability.

• rope ('double')
  Rope probability.

• right ('double')
  Right probability.

• probabilities ('character')
  Decisions based on posterior probabilities and threshold probability.

References

https://github.com/b0rxa/scmamp

Examples

    ## Not run: 
      results <- b_hierarchical_test(df= test_benchmark_small, 
        baseline = "algo_1", algorithm = "algo_3",  rho=0.1, 
        rope=c(-0.01, 0.01))
    
## End(Not run)

RebeccaGroh/seqbtests documentation built on Nov. 17, 2021, 8:50 a.m.