resampled_ttest: Compute correlated t-statistic and p-value for resampled data

In correctR: Corrected Test Statistics for Comparing Machine Learning Models on Correlated Samples

Compute correlated t-statistic and p-value for resampled data

Description

Compute correlated t-statistic and p-value for resampled data

Usage

resampled_ttest(x, y, n, n1, n2, tailed = c("two", "one"), greater = NULL)

Arguments

x	numeric vector of values for model A
y	numeric vector of values for model B
n	integer denoting number of repeat samples. Defaults to length(x)
n1	integer denoting train set size
n2	integer denoting test set size
tailed	character denoting whether to perform a two-tailed or one-tailed test. Can be one of "two" or "one". Defaults to "two"
greater	character specifying whether "x" or "y" is greater for the one-tailed test if tailed = "one". Defaults to NULL

Value

data.frame containing the test statistic and p-value

Author(s)

Trent Henderson

References

Nadeau, C., and Bengio, Y. Inference for the Generalization Error. Machine Learning 52, (2003).

Bouckaert, R. R., and Frank, E. Evaluating the Replicability of Significance Tests for Comparing Learning Algorithms. Advances in Knowledge Discovery and Data Mining. PAKDD 2004. Lecture Notes in Computer Science, 3056, (2004).

Examples

x <- rnorm(100, mean = 95, sd = 0.5)
y <- rnorm(100, mean = 90, sd = 1)
resampled_ttest(x = x, y = y, n = 100, n1 = 80, n2 = 20, tailed = "two")

correctR documentation built on April 12, 2025, 1:57 a.m.