evaluate: Evaluate a caret model

In correapvf/caretSDM: Species Distribution Models using caret

Evaluate a caret model

Description

Evaluate a caret model using the metrics of summaryFunction.

Usage

evaluate(x, ...)

## S3 method for class 'train'
evaluate(
  x,
  testdata = NULL,
  testy = NULL,
  testindex = NULL,
  summaryFunction = NULL,
  calc.train = TRUE,
  errorFunction = ci_95,
  ...
)

## S3 method for class 'evaluate.train'
plot(x, plot_errorbar = TRUE, ...)

## S3 method for class 'evaluate.train'
dot_plot(x, data = NULL, metric = "all", ...)

## S3 method for class 'evaluate.train'
pairs_plot(x, data = NULL, metric = NULL, fixed_axis = TRUE, ...)

Arguments

x	An object returned by evaluate.
...	ignored
testdata	A data.frame with test values to be evaluated. If NULL, training values will be used.
testy	A factor of response variable of testdata. If NULL, it will be guessed from testdata data.frame.
testindex	A list with rows index of testdata for each resample, preferably an output of create.test.index or create.test.index.blockCV. If NULL, indexes are create based on the same methods provided by trainControl used in the model. Only used if error_bar = TRUE.
summaryFunction	A Summary function (one of defaultSummary) that calculate the metrics. If NULL, the summaryFunction provided in the model will be used.
calc.train	logical. Evaluate training data? If FALSE, only test data is evaluated.
errorFunction	A function used to calculate errors across resamples. Default is 95% confidence interval. If NULL, errors are not calculated.
plot_errorbar	logical. Should plot error bars?
data	Which data type to plot? Should be either 'train' or 'test'. When NULL, it defaults to test data, if present.
metric	A character, indicating which metric to plot. If NULL, only the first metric is plotted. If 'all', all metrics are plotted.
fixed_axis	logical. Should axis in all plots be fixed to the same limits?

Value

An S3 object of class 'evaluate.train', including:

• eval - A data.table in the long format with data type, metrics, values and error across resamples. If errorFunction != NULL, values are means across resamples.

• resample - A data.table with metrics in each resample.

Note

This function is somewhat similar to resamples, however this function supports evaluation using test data, a custom threshold (use setThreshold), or a different summaryFunction.

See Also

confusionMatrix2 ROCcurve

Examples

## Not run: 
evaluate(model)

# evaluate test data only
testindex <- create.test.index(testdata$response) # get response of testdata
evaluate(model, testdata, testindex = testindex, calc.train = FALSE)

# for multiple models
models <- list(model1, model2, model3)
e <- evaluate(models, summaryFunction = twoClassSummary)
plot(e)
dot_plot(e, data.type = "test", metric = "ROC")
pairs_plot(e)
pairs_plot(e, fixed_axis = FALSE)

## End(Not run)

correapvf/caretSDM documentation built on June 2, 2022, 8:29 a.m.