bCorrelatedTtest: Bayesian equivalent to the correlated t-test

In b0rxa/scmamp: Statistical Comparison of Multiple Algorithms in Multiple Problems

Bayesian equivalent to the correlated t-test

Description

Implementation of the Bayesian version of the correlated t-test as presented in Benavoli et al. 2017

Usage

bCorrelatedTtest(x, y = NULL, rho = 0, rope = c(-0.01, 0.01))

Arguments

x	First sample
y	Second sample (if not provided, x is assumed to be the difference)
rho	Correlation factor (see details)
rope	Interval for the difference considered as "irrelevant"

Details

Note that the default value for rho is 0, wich converts the test in the equivalent of the standard t-test. To correct due to correlation you need to set the rho parameter. In the case of classifiers compared using any validation scheme the heuristic typically used is to set rho to num. test instances / total num. of instance The function has been implemented to be used in the comparison of classifiers and, in such situation, the heuristic used to fix the correlation factor is the size of the training set divided by the total size of the data. For the same reason, the reference value (the midpoint for the rope) is 0. However, this may be changed (though it matches the prior for the paremter, which follows a Gaussian distribution of 0 mean). The results includes the typicall information relative to the three areas of the posterior density (left, right and rope probabilities), but also the basic functions (density, cummulative and quantile), as well as the parameters of the posterior density function, which is a Student's t.

Value

A list with the following elements:

method	a string with the name of the method used
posterior.probabilities	a vector with the left, rope and right probabilities
approximated	a logical value, TRUE if the posterior distribution is approximated (sampled) and FALSE if it is exact
parameters	parameters used by the method
posterior	posterior density function if the method is exact and the obtained sample if the method is approximated
additional	a list that contains the posterior cumulative function (pposterior), the posterior quantile function (qposterior) and the parameters of the posterior density function (posterior.df, posterior.mean, posterior.sd)

References

A. Benavoli, G. Corani, J. Demsar, M. Zaffalon (2017) Time for a Change: a Tutorial for Comparing Multiple Classifiers Through Bayesian Analysis. Journal of Machine Learning Research, 18, 1-36.

Examples

sample1 <- rnorm(25, 1, 1)
sample2 <- rnorm(25, 1.2, 1)
correlatedTtest (x=sample1, y=sample2, rho=0.1, alternative="less")

b0rxa/scmamp documentation built on Jan. 17, 2024, 10:49 a.m.