b_signed_rank_test: Bayesian Signed Rank test

In RebeccaGroh/seqbtests: Sequential Bayesian Comparison of ML Algorithm Performances

Description

This function implements the Bayesian version of the signed rank test. The performance of one baseline algorithm on one or multiple data sets is compared to either one or multiple algorithms.

Usage

b_signed_rank_test(
  df,
  problem = NULL,
  baseline,
  compare = NULL,
  algorithm = NULL,
  measure = NULL,
  prob = 0.95,
  s = 0.5,
  z_0 = 0,
  weights = NULL,
  mc_samples = 1e+05,
  rope = c(-0.01, 0.01)
)

Arguments

df	('list') Data frame containing the performane measure.
problem	('character') Problem set used to evaluate the algorithms performance. Value in 'problem' column.
baseline	('character') First algorithm. Value in 'algorithm' column.
compare	('character') Defines if one algorithm needs to perform better (better) for decisions based on the posterior distribution or whether it is sufficient to perform not worse (equal).
algorithm	('character') Second algorithm. Value in 'algorithm' column. If not defined, the baseline is tested against all algorithms in the data frame.
measure	('character') Name of the 'measure' column. If not defined, the first 'measure' column in the data frame is used.
prob	('double') Threshold probability that decision rely on. Default is 0.95.
s	('double') Scale parameter of the prior Dirichlet Process. Default is 0.5
z_0	('double') Position of the pseudo-observation associated to the prior Dirichlet Process. Default is 0.
weights	('any') A prior weights.
mc_samples	('double') Number of samples used to estimate the posterior probability distribution.
rope	('double') Region of practical equivalence. Default is c(-0.01, 0.01).

Details

The test has first been implemented in rNPBST.

Value

('list')
A list containing the following components:

• measure ('character')
  A string with the name of the measure column.

• method ('character')
  A string with the name of the method.

• baseline ('character')
  A string with the name of the first algorithm. Value in 'algorithm' column.

• data_frame ('list')
  A list containing the following components:

• algorithm ('character')
  Second algorithm. Value in 'algorithm' column. If not defined, the baseline is tested against all algorithms in the data frame.

• left ('double')
  Left probability.

• rope ('double')
  Rope probability.

• right ('double')
  Right probability.

• probabilities ('character')
  Decisions based on posterior probabilities and threshold probability.

References

https://github.com/JacintoCC/rNPBST

Examples

    ## Not run: 
    results <- b_signed_rank_test(df= test_benchmark_small,
    baseline = "algo_1", algorithm = "algo_2")
    
## End(Not run)

RebeccaGroh/seqbtests documentation built on Nov. 17, 2021, 8:50 a.m.