perf_wilcox: Compute column-wise statistics in a performance matrix

Description Usage Arguments Details Value Examples

View source: R/perf_wilcox.R

Description

Function perf_wilcox compares all the columns of a matrix through a wilcox.test. The columns are assumed to be performance measures (e.g. AUROC) whereas the rows are instances.

Usage

1
2
3
4
5
6
7
8
perf_wilcox(
    perf_mat,
    adjust = function(p) stats::p.adjust(p, method = "fdr"),
    ci = 0.95,
    digits_ci = 2,
    digits_p = 3,
    ...
)

Arguments

perf_mat

Numeric matrix whose columns contain performance metrics of different methods.

adjust

Function to adjust the p-values for multiple testing. By default, p.adjust with its default parameters is used.

ci

Numeric, confidence interval (defaults to 0.95)

digits_ci

Integer, digits to display in the confidence interval

digits_p

Integer, digits to display in the p-value

...

further arguments for format

Details

The statistical comparison of the columns is intended to ease comparisons between methods in a rigorous way. Methods are compared pairwise and a p-value for difference in performance. The function perf_wilcox returns a character matrix so that (1) the upper triangular matrix contains confidence intervals on the estimate of the difference between performances, and (2) the lower triangular matrix contains the two-tailed p-value that tests difference in performance, with multiple testing correction. The comparison takes place between row and column in that precise order: a positive difference favours the row and a negative one, the column.

Value

Character matrix. The upper triangular matrix contains a confidence interval and the estimate of the pairwise difference in performance. The lower triangular matrix shows the associated two-tailed p-value, with multiple testing correction.

Examples

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
# Dummy data frame to test 
n <- 100
perf_mat <- cbind(
    good = runif(n = n, min = 0.5, max = 1), 
    so_so = runif(n = n, min = 0.2, max = 0.7), 
    bad = runif(n = n, min = 0, max = 0.5)
)
wilcox_mat <- perf_wilcox(perf_mat)

# See how the methods in the rows compare to those
# in the columns, confidence interval 
# (upper) and p-value (lower)
wilcox_mat

diffuStats documentation built on Feb. 22, 2021, 10 a.m.