seq_b_corr_t_test: Sequential Bayesian correlated t-test

In RebeccaGroh/seqbtests: Sequential Bayesian Comparison of ML Algorithm Performances

Description

This function implements a sequential approach of the Bayesian correlated t-test to compare the performance of machine learning algorithms to one another. Sample size is not fixed in advance, data are evaluated as they are collected. Further sampling is stopped in accordance with a pre-defined stopping rule.

Usage

seq_b_corr_t_test(
  problem,
  baseline,
  algorithm = NULL,
  measure = NULL,
  compare = NULL,
  rho = 0.1,
  rope = c(-0.01, 0.01),
  max_repls = 20,
  prob = 0.95,
  min_repls = 5,
  ...
)

Arguments

problem	('character') Problem set used to evaluate the algorithms performance. Value in 'problem' column.
baseline	('character') First algorithm. Value in 'algorithm' column.
algorithm	('character') Second algorithm. Value in 'algorithm' column. If not defined, the baseline is tested against all algorithms in the data frame.
measure	('character') Name of the 'measure' column. If not defined, the first 'measure' column in the data frame is used.
compare	('character') Defines if one algorithm needs to perform better (better) for decisions based on the posterior distribution or whether it is sufficient to perform not worse (equal).
rho	('double') Correlation factor. Default is 0.1.
rope	('double') Region of practical equivalence. Default is c(-0.01, 0.01).
max_repls	('double') Maximum number of replications. If a complete data frame is passed (max_repls) should correspond to the maximum number of replications that are built. Default is 20.
prob	('double') Threshold probability that decision rely on. Default is 0.95.
min_repls	('double') Minimum number of replications that is used/generated before an optional stopping rule is activated. Default is 5.
...	(any) Additional arguments for (get_replication). To pass a complete data frame, set (df) ('character').

Details

The basis for this test has first been implemented in scmamp.

Value

('list')
A list containing the following components:

• measure ('character')
  A string with the name of the measure column.

• method ('character')
  A string with the name of the method.

• baseline ('character')
  A string with the name of the first algorithm. Value in 'algorithm' column.

• data_frame ('list')
  A list containing the following components:

• algorithm ('character')
  Second algorithm. Value in 'algorithm' column. If not defined, the baseline is tested against all algorithms in the data frame.

• left ('double')
  Left probability.

• rope ('double')
  Rope probability.

• right ('double')
  Right probability.

• repls ('double')
  Number of evaluated replications.

• probabilities ('character')
  Decisions based on posterior probabilities and threshold probability.

References

https://github.com/b0rxa/scmamp

Examples

    results <- seq_b_corr_t_test(df = test_benchmark_small, rho=0.1,
    problem = "problem_a", baseline = "algo_1", 
    compare = "equal", max_repls = 10)

RebeccaGroh/seqbtests documentation built on Nov. 17, 2021, 8:50 a.m.