predefinedObjectiveFunctions: Predefined objective functions for parameter tuning

In TunePareto: Multi-Objective Parameter Tuning for Classifiers

Predefined objective functions for parameter tuning

Description

Predefined objective functions that calculate several performance criteria of reclassification or cross-validation experiments.

Usage

reclassAccuracy(saveModel = FALSE)
reclassError(saveModel = FALSE)
reclassWeightedError(saveModel = FALSE)
reclassSensitivity(caseClass, saveModel = FALSE)
reclassRecall(caseClass, saveModel = FALSE)
reclassTruePositive(caseClass, saveModel = FALSE)
reclassSpecificity(caseClass, saveModel = FALSE)
reclassTrueNegative(caseClass, saveModel = FALSE)
reclassFallout(caseClass, saveModel = FALSE)
reclassFalsePositive(caseClass, saveModel = FALSE)
reclassMiss(caseClass, saveModel = FALSE)
reclassFalseNegative(caseClass, saveModel = FALSE)
reclassPrecision(caseClass, saveModel = FALSE)
reclassPPV(caseClass, saveModel = FALSE)
reclassNPV(caseClass, saveModel = FALSE)
reclassConfusion(trueClass, predictedClass, saveModel = FALSE)

cvAccuracy(nfold = 10, ntimes = 10,
           leaveOneOut = FALSE, stratified = FALSE,
           foldList = NULL,
           saveModel = FALSE) 
cvError(nfold = 10, ntimes = 10, 
        leaveOneOut = FALSE, stratified=FALSE, 
        foldList=NULL,
        saveModel = FALSE)
cvErrorVariance(nfold = 10, ntimes = 10, 
                leaveOneOut = FALSE, stratified=FALSE, 
                foldList=NULL,
                saveModel = FALSE)
cvWeightedError(nfold = 10, ntimes = 10, 
                leaveOneOut = FALSE, stratified=FALSE, 
                foldList=NULL,
                saveModel = FALSE)
cvSensitivity(nfold = 10, ntimes = 10, 
              leaveOneOut = FALSE, stratified=FALSE, 
              foldList=NULL, caseClass,
              saveModel = FALSE)
cvRecall(nfold = 10, ntimes = 10,
         leaveOneOut = FALSE, stratified=FALSE,
         foldList=NULL, caseClass,
         saveModel = FALSE)
cvTruePositive(nfold = 10, ntimes = 10, 
               leaveOneOut = FALSE, stratified=FALSE, 
               foldList=NULL, caseClass,
               saveModel = FALSE)
cvSpecificity(nfold = 10, ntimes = 10, 
              leaveOneOut = FALSE, stratified=FALSE, 
              foldList=NULL, caseClass,
              saveModel = FALSE)
cvTrueNegative(nfold = 10, ntimes = 10, 
               leaveOneOut = FALSE, stratified=FALSE, 
               foldList=NULL, caseClass,
               saveModel = FALSE)
cvFallout(nfold = 10, ntimes = 10, 
          leaveOneOut = FALSE, stratified=FALSE, 
          foldList=NULL, caseClass,
          saveModel = FALSE)
cvFalsePositive(nfold = 10, ntimes = 10, 
                leaveOneOut = FALSE, stratified=FALSE, 
                foldList=NULL, caseClass,
                saveModel = FALSE)          
cvMiss(nfold = 10, ntimes = 10, 
       leaveOneOut = FALSE, stratified=FALSE, 
       foldList=NULL, caseClass,
       saveModel = FALSE)
cvFalseNegative(nfold = 10, ntimes = 10,
                leaveOneOut = FALSE, stratified=FALSE,
                foldList=NULL, caseClass,
                saveModel = FALSE)
cvPrecision(nfold = 10, ntimes = 10,
            leaveOneOut = FALSE, stratified=FALSE,
            foldList=NULL, caseClass,
            saveModel = FALSE)
cvPPV(nfold = 10, ntimes = 10,
      leaveOneOut = FALSE, stratified=FALSE,
      foldList=NULL, caseClass,
      saveModel = FALSE)
cvNPV(nfold = 10, ntimes = 10,
      leaveOneOut = FALSE, stratified=FALSE,
      foldList=NULL, caseClass,
      saveModel = FALSE)
cvConfusion(nfold = 10, ntimes = 10, 
            leaveOneOut = FALSE, stratified=FALSE,
            foldList=NULL, trueClass, predictedClass,
            saveModel = FALSE)

Arguments

nfold	The number of groups of the cross-validation. Ignored if leaveOneOut=TRUE.
ntimes	The number of repeated runs of the cross-validation. Ignored if leaveOneOut=TRUE.
leaveOneOut	If this is true, a leave-one-out cross-validation is performed, i.e. each sample is left out once in the training phase and used as a test sample
stratified	If set to true, a stratified cross-validation is carried out. That is, the percentage of samples from different classes in the cross-validation folds corresponds to the class sizes in the complete data set. If set to false, the folds may be unbalanced.
foldList	If this parameter is set, the other cross-validation parameters (ntimes, nfold, leaveOneOut, stratified) are ignored. Instead, the precalculated cross-validation partition supplied in foldList is used. This allows for using the same cross-validation experiment in multiple tunePareto calls. Partitions can be generated using generateCVRuns.
caseClass	The class containing the positive samples for the calculation of specificity and sensitivity. All samples with different class labels are regarded as controls (negative samples).
trueClass	When calculating the confusion of two classes, the class to which a sample truly belongs.
predictedClass	When calculating the confusion of two classes, the class to which a sample is erreneously assigned.
saveModel	If set to true, the trained model(s) are stored to the additionalData component of the resulting TuneParetoResult objects (see tunePareto for details). In case of a reclassification, a single model is stored. In case of a cross-validation, a list of length nruns, each containing a sub-list of nfold models, is stored. If the size of a model is large, setting saveModel = TRUE can result in a high memory consumption. As the model information is the same for all reclassification objectives or for cross-validation objectives with the same parameters, it is usually sufficient to set saveModel=TRUE for only one of the objective functions.

Details

The functions do not calculate the objectives directly, but return a structure of class TuneParetoObjectives that provides all information on the objective function for later use in tunePareto.

The behaviour of the functions in tunePareto is as follows:

The reclassification functions train the classifiers with the full data set. Afterwards, the classifiers are applied to the same data set. reclassAccuracy measures the fraction of correctly classified samples, while reclassError calculates the fraction of misclassified samples. reclassWeightedError calculates the sum of fractions of misclassified samples in each class weighted by the class size. reclassSensitivity measures the sensitivity, and reclassSpecificity measures the specificity of the reclassification experiment. reclassTruePositive and reclassRecall are aliases for reclassSensitivity, and reclassTrueNegative is an alias for reclassSpecificity. reclassFallout and its equivalent alias reclassFalsePositive give the ratio of false positives to all negative samples, and reclassMiss and its alias reclassFalseNegative measure the ratio of false negatives to all positive samples. reclassPrecision calculates the precision of the reclassification experiment, i.e. the ratio of true positives to all samples classified as positive. This is equivalent to the positive predictive value (reclassPPV). reclassNPV measures the negative predictive value, i.e. the ratio of true negatives to all samples classified as negative. reclassConfusion calculates the fraction of samples in trueClass that have been confused with predictedClass.

reclassError, reclassWeightedError, reclassFallout, reclassFalsePositive, reclassMiss, reclassFalsePositive and reclassConfusion are minimization objectives, whereas reclassAccuracy, reclassSensitivity, reclassTruePositive, reclassRecall, reclassSpecificity, reclassTrueNegative reclassPrecision, reclassPPV and reclassNPV are maximization objectives.

The cross-validation functions partition the samples in the data set into a number of groups (depending on nfold and leaveOneOut). Each of these groups is left out once in the training phase and used for prediction. The whole procedure is usually repeated several times (as specified in ntimes), and the results are averaged. Similar to the reclassification functions, cvAccuracy calculates the fraction of correctly classified samples over the runs, cvError calculates the average fraction of misclassified samples over the runs, and cvWeightedError calculates the mean sum of fractions of misclassified samples in each class weighted by the class size. cvErrorVariance calculates the variance of the cross-validation error. cvSensitivity, cvRecall and cvTruePositive calculate the average sensitivity, and cvSpecificity and cvTrueNegative calculate the average specificity. cvFallout and cvFalsePositive calculate the average false positive rate over the runs. cvMiss and cvFalseNegative calculate the average false negative rate over the runs. cvPrecision and cvPPV calculate the average precision/positive predictive rate. cvNPV gives the average negative predictive rate over all runs. cvConfusion calculates the average fraction of samples in trueClass that have been confused with predictedClass.

cvError, cvWeightedError, cvErrorVariance, cvFallout, cvFalsePositive, cvMiss, cvFalseNegative and cvConfusion are minimization objectives, and cvAccuracy, cvSensitivity, cvRecall, cvTruePositive, cvSpecificity, cvTrueNegative, cvPrecision, cvPPV and cvNPV are maximization objectives.

Value

An object of class TuneParetoObjective representing the objective function. For more details, see createObjective.

See Also

createObjective, tunePareto, generateCVRuns

Examples

# build a list of objective functions
objectiveFunctions <- list(cvError(10, 10),
                           reclassSpecificity(caseClass="setosa"), 
                           reclassSensitivity(caseClass="setosa"))

# pass them to tunePareto
print(tunePareto(data = iris[, -ncol(iris)], 
                 labels = iris[, ncol(iris)],
                 classifier = tunePareto.knn(),
                 k = c(3,5,7,9),
                 objectiveFunctions = objectiveFunctions))

TunePareto documentation built on Oct. 2, 2023, 5:06 p.m.