ClassPerm: Permutation Analysis for classification

In ateshkoul/PredPsych: Predictive Approaches in Psychology

Description

simple function to create permutation testing of a classifier

Usage

ClassPerm(Data, classCol, selectedCols, classifierFun, nSims = 1000,
  plot = TRUE, silent = FALSE, ...)

Arguments

Data	(dataframe) dataframe of the data
classCol	(numeric or string) column number that contains the variable to be predicted
selectedCols	(optional) (numeric or string) all the columns of data that would be used either as predictor or as feature
classifierFun	(optional) (function) classifier function
nSims	(optional) (numeric) number of simulations
plot	(optional) (logical) whether to plot null accuracy distribution
silent	(optional) (logical) whether to print messages or not
...	(optional) additional arguments for the function

Details

The function implements Permutation tests for classification. Permutation tests are a set of non-parametric methods for hypothesis testing without assuming a particular distribution (Good, 2005). In case of classification analysis, this requires shuffling the labels of the dataset (i.e. randomly shuffling classes/conditions between observations) and calculating accuracies obtained.

Value

Returns actualAcc of the classification analysis, p-value from permutation testing, nullAcc distribution of the permutation figure containing null distribution

Author(s)

Atesh Koul, C'MON unit, Istituto Italiano di Tecnologia

[e-mail]

References

Phipson, B., & Smyth, G. K. (2010). Permutation P-values Should Never Be Zero: Calculating Exact P-values When Permutations Are Randomly Drawn. Statistical Applications in Genetics and Molecular Biology, 9(1), 1544-6115.

Ojala, M. & Garriga, G. C. Permutation Tests for Studying Classifier Performance. J. Mach. Learn. Res. 11, 1833-1863 (2010).

Good, P. (2005). Permutation, Parametric and Bootstrap Tests of Hypotheses. New York: Springer-Verlag.

Examples

# perform a permutation testing for 10% of the kinematics movement data#' 
# not run
# PermutationResult <- ClassPerm(Data = KinData, classCol = 1,
# selectedCols = c(1,2,12,22,32,42,52,62,72,82,92,102,112), nSims = 1000,cvType = "holdout")
# Output:
# Performing Permutation Analysis for Classification
#
# Performing Cross-validation
#
# Performing holdout Cross-validation 
# genclassifier was not specified, 
#  Using default value of Classifier.svm (genclassifier = Classifier.svm)
#
# cvFraction was not specified, 
#  Using default value of 0.8 (cvFraction = 0.8)
#
# Proportion of Test/Train Data was :  0.2470588 
# [1] "Test holdout Accuracy is  0.65"
# holdout classification Analysis: 
# cvFraction : 0.8 
# Test Accuracy 0.65
# *Legend:
# cvFraction = Fraction of data to keep for training data 
# Test Accuracy = Accuracy from the Testing dataset
# 
# Performing permutation testing...
# Performing 1000 simulations 
# |=======================================================
# ==================================================================|100%
#                      Completed after 2 m 
# The p-value of the permutation testing is 0.001
# p-value generated using the approximate method for p-value calculation. 
# See Phipson, B. & Gordon K., S. (2010) for details


# Using LinearDA instead as function
# not run
# PermutationResult <- ClassPerm(Data = KinData, classCol = 1,
# selectedCols = c(1,2,12,22,32,42,52,62,72,82,92,102,112), nSims = 1000,classifierFun = LinearDA)


# Any minimalistic function can be used 
# The ClassPerm function sends the dataframe Data, classCol, 
# selectedCols as arguments
# not run
# myMinimalFun <- function(...){
# ***Calculate Error function as you want***
# return(accTest)
# } 
# Use the function for permutation testing e.g.
# Results <- ClassPerm(Data = KinData, classCol=1,
# selectedCols = c(1,2,12,22,32,42,52,62,72,82,92,102,112), 
# nSims = 1000,classifierFun = myMinimalFun)

ateshkoul/PredPsych documentation built on Aug. 1, 2020, 5:27 p.m.