testMultipleControl: Multiple Comparison Statistical Test (Friedman + Control Holm...

In exreport: Fast, Reliable and Elegant Reproducible Research

Description

This function perfoms a multiple comparison statistical test for the given experiment. First of all it performs a Friedman Test over all methods. In the case this test is rejected, meaning that significant differences are present among the methods a post-hoc test is then executed. For that, a comparison using the best method as a control is performed for each other method, finally a Holm familywise error correction is applied to the resulting p-values.

Usage

testMultipleControl(e, output, rankOrder = "max", alpha = 0.05)

Arguments

e	Input experiment
output	The output for which the tet will be performed.
rankOrder	The optimization strategy, can be either maximizing "max" or minimizing "min" the target output variable.
alpha	The significance level used for the whole testing procedure.

Value

an testMultipleControl object

Examples

# First we create an experiment from the wekaExperiment problem and prepare
# it to apply the test:
experiment <- expCreate(wekaExperiment, name="test", parameter="fold")
experiment <- expReduce(experiment, "fold", mean)
experiment <- expSubset(experiment, list(featureSelection = "yes"))
experiment <- expInstantiate(experiment, removeUnary=TRUE)

# Then we perform a testMultiplePairwise test procedure
test <- testMultipleControl(experiment, "accuracy", "max")

summary(test)

Example output

---------------------------------------------------------------------
Friedman test, objetive maximize output variable accuracy. Obtained p-value: 3.3072e-04
Chi squared with 3 degrees of freedom statistic: 18.6000
Test rejected: p-value: 3.3072e-04 < 0.0500
---------------------------------------------------------------------
Control post hoc test for output accuracy
Adjust method: Holm

Control method: RandomForest
p-values:
            J48	0.2579
     NaiveBayes	0.0324
           OneR	0.0001
---------------------------------------------------------------------

exreport documentation built on May 1, 2019, 10:13 p.m.