modTest: Train a model and store 'ROCit' tests on different datasets

In jknowles/EWStools: Tools for automating the testing and evaluation of education early warning system models

Description

This function wraps the train function in the caret package with model accuracy reports. It also allows for errors in fitting models to be caught to make it easier to use in a loop.

Usage

modTest(method, datatype = c("train", "test"), traindata, testdata = NULL,
  modelKeep = FALSE, length = NULL, fitControl = NULL, metric = NULL,
  cores = NULL, ...)

Arguments

method	a a string specifying which classification or regression model to use. Possible values are found using names(getModelInfo()).
datatype	a named character representing the accuracy object be built on either "train" or "test" data, user can include both
traindata	a list of length two containing a named slot for the matrix of predictors (pred) and the vector of classes (class)
testdata	a list of length two containing a named slot for the matrix of predictors (pred) and the vector of classes (class)
modelKeep	a logical indicating whether the original model object should be stored
length	an integer denoting the number of levels of each tuning parameter that should be generated to be passed to tuneLength in the train call
fitControl	an object generated by trainControl to control the behavior of train. If none is given a default is selected.
metric	a character string passed to train. a string that specifies what summary metric will be used to select the optimal model. By default, possible values are "RMSE" and "Rsquared" for regression and "Accuracy" and "Kappa" for classification. If custom performance metrics are used (via the summaryFunction argument in trainControl, the value of metric should match one of the arguments. If it does not, a warning is issued and the first metric given by the summaryFunction is used.
cores	An integer representing the number of cores to use on Windows. If not on windows, a warning is issued.
...	Additional arguments to be passed to train

Value

A character string with an error if unsuccessful. The result of the modAcc call if successful:

• method - the train method used to fit the model

• summaryTr - the ROCit for the training data

• summaryTe - the ROCit for the test data

• time - the time reported for the model to run, taken from the train object if available

Note

The values presented are for the optimal threshold as computed by the roc function. For some model types linear combos of predictors may be omitted.

jknowles/EWStools documentation built on May 19, 2019, 11:42 a.m.