seq_b_sign_test: Sequential Bayesian Sign test

Description Usage Arguments Details Value References Examples

View source: R/bayesian_sequential_tests.R

Description

This function implements a sequential approach of the Bayesian version of the sign test to compare the performance of machine learning algorithms to one another. Sample size is not fixed in advance, data are evaluated as they are collected. Further sampling is stopped in accordance with a pre-defined stopping rule.

Usage

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
seq_b_sign_test(
  problem = NULL,
  baseline,
  algorithm = NULL,
  measure = NULL,
  compare = NULL,
  s = 0.5,
  z_0 = 0,
  rope = c(-0.01, 0.01),
  weights = c(s/2, rep(1, length(x))),
  mc_samples = 1e+05,
  max_repls = 20,
  prob = 0.95,
  min_repls = 5,
  ...
)

Arguments

problem

('character')
Problem set used to evaluate the algorithms performance. Value in 'problem' column.

baseline

('character')
First algorithm. Value in 'algorithm' column.

algorithm

('character')
Second algorithm. Value in 'algorithm' column. If not defined, the baseline is tested against all algorithms in the data frame.

measure

('character')
Name of the 'measure' column. If not defined, the first 'measure' column in the data frame is used.

compare

('character')
Defines if one algorithm needs to perform better (better) for decisions based on the posterior distribution or whether it is sufficient to perform not worse (equal).

s

('double')
Scale parameter of the prior Dirichlet Process. Default is 0.5

z_0

('double')
Position of the pseudo-observation associated to the prior Dirichlet Process. Default is 0.

rope

('double')
Region of practical equivalence. Default is c(-0.01, 0.01).

weights

('any')
A prior weights.

mc_samples

('double')
Number of samples used to estimate the posterior probability distribution.

max_repls

('double')
Maximum number of replications. If a complete data frame is passed (max_repls) should correspond to the maximum number of replications that are built. Default is 20.

prob

('double')
Threshold probability that decision rely on. Default is 0.95.

min_repls

('double')
Minimum number of replications that is used/generated before an optional stopping rule is activated. Default is 5.

...

(any)
Additional arguments for (get_replication). To pass a complete data frame, set (df) ('character').

Details

The basis for this test has first been implemented in rNPBST. For testing over multiple datasets, donĀ“t specify the problem set argument in the function.

Value

('list')
A list containing the following components:

References

https://github.com/JacintoCC/rNPBST

Examples

1
2
3
    
    results <- seq_b_sign_test(df = test_benchmark_small, 
    baseline = "algo_1", algorithm = "algo_2", max_repls = 10)

RebeccaGroh/seqbtests documentation built on Nov. 17, 2021, 8:50 a.m.