Description Usage Arguments Details Value Author(s) References Examples
simple function to create permutation testing of a classifier
1 2 |
Data |
(dataframe) dataframe of the data |
classCol |
(numeric or string) column number that contains the variable to be predicted |
selectedCols |
(optional) (numeric or string) all the columns of data that would be used either as predictor or as feature |
classifierFun |
(optional) (function) classifier function |
nSims |
(optional) (numeric) number of simulations |
plot |
(optional) (logical) whether to plot null accuracy distribution |
silent |
(optional) (logical) whether to print messages or not |
... |
(optional) additional arguments for the function |
The function implements Permutation tests for classification. Permutation tests are a set of non-parametric methods for hypothesis testing without assuming a particular distribution (Good, 2005). In case of classification analysis, this requires shuffling the labels of the dataset (i.e. randomly shuffling classes/conditions between observations) and calculating accuracies obtained.
Returns actualAcc
of the classification analysis,
p-value
from permutation testing, nullAcc
distribution of the permutation figure
containing null distribution
Atesh Koul, C'MON unit, Istituto Italiano di Tecnologia
Phipson, B., & Smyth, G. K. (2010). Permutation P-values Should Never Be Zero: Calculating Exact P-values When Permutations Are Randomly Drawn. Statistical Applications in Genetics and Molecular Biology, 9(1), 1544-6115.
Ojala, M. & Garriga, G. C. Permutation Tests for Studying Classifier Performance. J. Mach. Learn. Res. 11, 1833-1863 (2010).
Good, P. (2005). Permutation, Parametric and Bootstrap Tests of Hypotheses. New York: Springer-Verlag.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 | # perform a permutation testing for 10% of the kinematics movement data#'
# not run
# PermutationResult <- ClassPerm(Data = KinData, classCol = 1,
# selectedCols = c(1,2,12,22,32,42,52,62,72,82,92,102,112), nSims = 1000,cvType = "holdout")
# Output:
# Performing Permutation Analysis for Classification
#
# Performing Cross-validation
#
# Performing holdout Cross-validation
# genclassifier was not specified,
# Using default value of Classifier.svm (genclassifier = Classifier.svm)
#
# cvFraction was not specified,
# Using default value of 0.8 (cvFraction = 0.8)
#
# Proportion of Test/Train Data was : 0.2470588
# [1] "Test holdout Accuracy is 0.65"
# holdout classification Analysis:
# cvFraction : 0.8
# Test Accuracy 0.65
# *Legend:
# cvFraction = Fraction of data to keep for training data
# Test Accuracy = Accuracy from the Testing dataset
#
# Performing permutation testing...
# Performing 1000 simulations
# |=======================================================
# ==================================================================|100%
# Completed after 2 m
# The p-value of the permutation testing is 0.001
# p-value generated using the approximate method for p-value calculation.
# See Phipson, B. & Gordon K., S. (2010) for details
# Using LinearDA instead as function
# not run
# PermutationResult <- ClassPerm(Data = KinData, classCol = 1,
# selectedCols = c(1,2,12,22,32,42,52,62,72,82,92,102,112), nSims = 1000,classifierFun = LinearDA)
# Any minimalistic function can be used
# The ClassPerm function sends the dataframe Data, classCol,
# selectedCols as arguments
# not run
# myMinimalFun <- function(...){
# ***Calculate Error function as you want***
# return(accTest)
# }
# Use the function for permutation testing e.g.
# Results <- ClassPerm(Data = KinData, classCol=1,
# selectedCols = c(1,2,12,22,32,42,52,62,72,82,92,102,112),
# nSims = 1000,classifierFun = myMinimalFun)
|
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.