postHocTest: Post hoc tests for multiple comparison analises
In scmamp: Statistical Comparison of Multiple Algorithms in Multiple Problems

Description Usage Arguments Value References See Also Examples

This function is a wrapper to run the post hoc tests. It can run both all vs. control and all vs. all post hoc tests.

1
2
3

postHocTest(data, algorithms = NULL, group.by = NULL, test = "friedman",
  control = NULL, use.rank = FALSE, sum.fun = mean, correct = "finner",
  alpha = 0.05, ...)

`data`	A matrix or data frame containing the results obtained by the algorithms (columns) in each problem (rows). It can contain additional columns, but if any of the column has to be discarderd (not used neither to group the problems nor to be part of the comparison), then it is mandatory to indicate, in the `algorithms` parameter, which columns contain the algorithm information.
`algorithms`	Vector with either the names or the indices of the columns that contain the values to be tested. If not provided, the function assumes that all the columns except those indicated in `group.by` represent the results obtained by an algorithm.
`group.by`	Vector with either the names or the indices of the columns to be used to group the data. Each group is tested independently. If `NULL`, all the data is used for a single comparison.
`test`	Parameter that indicates the statistical test to be used. It can be either a string indicating one of the available test or a function. As a string, it can take the following values: `'wilcoxon'` - Wilcoxon Signed Rank test, as in Demsar (2006) `'t-test'` - t-test (R's t.test function with paired option set at `TRUE`) `'friedman'` - Friedman post hoc test, as in Demsar (2006) `'aligned ranks'` Friedman's Aligned Ranks post hoc test, as in Garcia and Herrera (2010) `'quade'` - Quade post hoc test, as in Garcia and Herrera (2010) `'tukey'` - Tukey's ANOVA post hoc test, as in Test 28 in Kanji (2006). If a function is provided, then it has to have as first argument a matrix containing the columns to be compared. The function has to return a list with, at least, an element named `p.value` (as the `htest` objects that are usually returned by R's test implementations).
`control`	Either the name or the index of a column in the dataset (one of those in the `algorithms` vector), to be used as control. Alternatively, this argument can be `'min'`, to select the algorithm with the minimum value, `'max'`, to select the algorithm with the maximum value as control. If the argument is not provided (or is `NULL`), all the pairwise comparisons are performed instead of all vs. control comparisons.
`use.rank`	If `TRUE`, then the summarization of the data is based on the ranks, rather than on the actual values. The selecion of the algorithm with the maximum or minimum value is also done in terms of the summarized ranking.
`sum.fun`	Function to be used to summarize the data. By default, average is used.
`correct`	Either string indicating the type of correction that has to be applied or a function to correct the p-values for multiple testing; This parameter is only need in case the data is grouped. As a string, the valid values are: `shaffer` - Shaffer's (static) procedure, as in Garcia and Herrera (2008) `bergmann` - Bergman and Hommel's procedure (similar to Shaffer dynamic), as in Garcia and Herrera (2008) `holland` - Holland's procedure, as in Garcia and Herrera (2010) `finner` - Finner's procedure, as in Garcia and Herrera (2010) `rom` - Rom's procedure, as in Garcia and Herrera (2010) `li` - Li's procedure, as in Garcia and Herrera (2010) Any of the methods implemented in the `p.adjust` function. For a list of options, type `p.adjust.methods` . If a function is provided, the it has to recieve, as first argument, a vector of pvalues to be corrected and has to return a verctor with the corrected p-values in the same order as the input vector.
`alpha`	Alpha value used in Rom's correction. By default, it is set at 0.05.
`...`	Special argument used to pass additional parameters to the statistical test and the correction method.

In all cases the function returns a list with three elements, the summarization of the data (a row per group), the raw p-values and the corrected p-values. When the data is grouped and all the pairwise comparisons are performed (no control is provided), the p-values are in three dimensional arrays where the last dimension is corresponds to the group. In any other cases the result is a matrix with one or more rows.

Note that Shaffer and Bergmann and Hommel's correction can only be applied when all the pairwise tests are conducted, due to their assumptions. Moreover, its use when the data is grouped (multiple pairwise comparsions) is not trivial and, thus, it is not possible to use it when the data is grouped.

S. Garcia and F. Herrera (2010) Advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and ata mining: Experimental analysis of power. Information Sciences, 180, 2044-2064.

Garcia S. and Herrera, F. (2008) An Extension on "Statistical Comparisons of Classifiers over Multiple Data Sets" for All Pairwise Comparisons. Journal of Machine Learning Research, 9, 2677-2694.

Kanji, G. K. (2006) 100 Statistical Tests. SAGE Publications Ltd, 3rd edition.

Demsar, J. (2006) Statistical Comparisons of Classifiers over Multiple Data Sets. Journal of Machine Learning Research, 7, 1-30.

friedmanPost, friedmanAlignedRanksPost, quadePost, tukeyPost, adjustShaffer, adjustBergmannHommel, adjustHolland, adjustFinner, adjustRom, adjustLi

# Grouped data, all pairwise
data(data_blum_2015)
res <- postHocTest (data=data.blum.2015, algorithms=c("FrogCOL", "FrogMIS", "FruitFly"), 
                    use.rank=TRUE, group.by=c("Size"), test="quade", correct="finner")
                   
# Data summarization
res$summary

# Corrected pvalues for the first group
res$corrected.pval[, , 1]

# Grouped data, all vs. control
res <- postHocTest (data=data.blum.2015, control="max", use.rank=FALSE, 
                    group.by=c("Size","Radius"), test="wilcoxon", correct="finner")
                   
# Data summarization
res$summary

# Corrected pvalues
res$corrected.pval
                                       
# Not grouped data
data(data_gh_2008)
postHocTest (data=data.gh.2008, test="aligned ranks", correct="bergmann")