b_corr_t_test: Bayesian correlated t test

In RebeccaGroh/seqbtests: Sequential Bayesian Comparison of ML Algorithm Performances

Description

This function implements the Bayesian version of the correlated t-test. The performance of one baseline algorithm on one data set is compared to either one or multiple algorithms.

Usage

b_corr_t_test(
  df,
  problem,
  baseline,
  algorithm = NULL,
  measure = NULL,
  compare = NULL,
  rho = 0.1,
  rope = c(-0.01, 0.01),
  prob = 0.95
)

Arguments

df	('list') Data frame containing the performane measure.
problem	('character') Problem set used to evaluate the algorithms performance. Value in 'problem' column.
baseline	('character') First algorithm. Value in 'algorithm' column.
algorithm	('character') Second algorithm. Value in 'algorithm' column. If not defined, the baseline is tested against all algorithms in the data frame.
measure	('character') Name of the 'measure' column. If not defined, the first 'measure' column in the data frame is used.
compare	('character') Defines if one algorithm needs to perform better (better) for decisions based on the posterior distribution or whether it is sufficient to perform not worse (equal).
rho	('double') Correlation factor. Default is 0.1.
rope	('double') Region of practical equivalence. Default is c(-0.01, 0.01).
prob	('double') Threshold probability that decision rely on. Default is 0.95.

Details

The test has first been implemented in scmamp. If rho equals 0 this converts the test in the equivalent of the standard t test.

Value

('list')
A list containing the following components:

• measure ('character')
  A string with the name of the measure column.

• method ('character')
  A string with the name of the method.

• baseline ('character')
  A string with the name of the first algorithm. Value in 'algorithm' column.

• data_frame ('list')
  A list containing the following components:

• algorithm ('character')
  Second algorithm. Value in 'algorithm' column. If not defined, the baseline is tested against all algorithms in the data frame.

• left ('double')
  Left probability.

• rope ('double')
  Rope probability.

• right ('double')
  Right probability.

• probabilities ('character')
  Decisions based on posterior probabilities and threshold probability.

References

https://github.com/b0rxa/scmamp

Examples

results <- b_corr_t_test(df= test_benchmark_small, problem = "problem_a", 
                         baseline = "algo_1", algorithm = "algo_2")

RebeccaGroh/seqbtests documentation built on Nov. 17, 2021, 8:50 a.m.