pairwise.test: Performs a pairwise Kolmogorov-Smirnov test on the...

In IOHanalyzer: Data Analysis Part of 'IOHprofiler'

Performs a pairwise Kolmogorov-Smirnov test on the bootstrapped running times among a data set

Description

This function performs a Kolmogorov-Smirnov test on each pair of algorithms in the input x to determine which algorithm gives a significantly smaller running time. The resulting p-values are arranged in a matrix, where each cell (i, j) contains a p-value from the test with alternative hypothesis: the running time of algorithm i is smaller (thus better) than that of j.

Usage

pairwise.test(x, ...)

## S3 method for class 'list'
pairwise.test(x, max_eval, bootstrap.size = 30, ...)

## S3 method for class 'DataSetList'
pairwise.test(x, ftarget, bootstrap.size = 0, which = "by_FV", ...)

Arguments

x	either a list that contains running time sample for each algorithm as sub-lists, or a DataSetList object
...	all other options
max_eval	list that contains the maximal running time for each algorithm as sub-lists
bootstrap.size	integer, the size of the bootstrapped sample. Set to 0 to disable bootstrapping
ftarget	float, the target value used to determine the running / hitting
which	wheter to do fixed-target ('by_FV') or fixed-budget ('by_RT') comparison time

Value

A matrix containing p-values of the test

Examples

pairwise.test(subset(dsl, funcId == 1), 16)

IOHanalyzer documentation built on Sept. 20, 2023, 5:07 p.m.