performance: Performance estimation

In fdm2id: Data Mining and R Programming for Beginners

Performance estimation

Description

Estimate the performance of classification or regression methods using bootstrap or crossvalidation (accuracy, ROC curves, confusion matrices, ...)

Usage

performance(
  methods,
  train.x,
  train.y,
  test.x = NULL,
  test.y = NULL,
  train.size = round(0.7 * nrow(train.x)),
  type = c("evaluation", "confusion", "roc", "cost", "scatter", "avsp"),
  protocol = c("bootstrap", "crossvalidation", "loocv", "holdout", "train"),
  eval = ifelse(is.factor(train.y), "accuracy", "r2"),
  nruns = 10,
  nfolds = 10,
  new = TRUE,
  lty = 1,
  seed = NULL,
  methodparameters = NULL,
  names = NULL,
  fuzzy = FALSE,
  positive = NULL,
  stratify = TRUE,
  ...
)

Arguments

methods	The classification or regression methods to be evaluated.
train.x	The dataset (description/predictors), a matrix or data.frame.
train.y	The target (class labels or numeric values), a factor or vector.
test.x	The test dataset (description/predictors), a matrix or data.frame. Giving test.x and test.y is the simplest use of this function: each method is fitted on (train.x, train.y), used to predict test.x, and its predictions are compared with test.y – no resampling at all. protocol then defaults to "holdout", and any other protocol raises an error, since it would ignore the test set.
test.y	The (test) target (class labels or numeric values), a factor or vector.
train.size	The size of the training set, for protocol = "holdout" without an explicit test set: either a number of observations, or a proportion between 0 and 1.
type	The type of evaluation (confusion matrix, ROC curve, ...)
protocol	How the performance is estimated. "bootstrap" (default) nruns bootstrap samples of the training set, each model evaluated on the observations left out of its sample. "crossvalidation" nruns repetitions of a nfolds-fold cross-validation of the training set. "loocv" leave-one-out cross-validation. "holdout" a single train/test split. Uses test.x/test.y when they are given – see test.x – and otherwise draws a training set of train.size observations and evaluates on the rest. "train" evaluates each model on the very data it was fitted on. Optimistic by construction; useful to show students exactly that.
eval	The evaluation functions.
nruns	The number of bootstrap runs.
nfolds	The number of folds (crossvalidation estimation).
new	A logical value indicating whether a new plot should be created or not (cost curves or ROC curves).
lty	The line type (and color) specified as an integer (cost curves or ROC curves).
seed	A specified seed for random number generation (useful for testing different method with the same bootstap samplings).
methodparameters	Method parameters (if null tuning is done by cross-validation).
names	Method names.
fuzzy	Used by type = "roc" and type = "cost" only. FALSE by default: the curves are built from the hard class labels, which reduces them to three points. Pass fuzzy = TRUE to build them from the estimated probabilities of the positive class, which is what a ROC curve is meant to show. See roc.curves.
positive	The label of the positive class. Used by type = "roc" and type = "cost" to orient the curves, and passed on to evaluation for the criteria that are defined on one class – precision, recall, the F-measure and the other measures taking an average argument – so that a two-class problem can be scored on either of its classes. Defaults to the first level of the target, which is worth setting explicitly whenever the class of interest is not the first one.
stratify	Whether the splits should preserve the proportions of the classes (TRUE, the default), for protocol = "crossvalidation" and for protocol = "holdout" when it draws its own split. Ignored for a numeric target, and for the bootstrap and leave-one-out protocols, which have no split to stratify.
...	Other specific parameters for the leaning method.

Value

The evaluation of the predictions (numeric value).

See Also

confusion, evaluation, cost.curves, roc.curves

Examples

## Not run: 
require ("datasets")
data (iris)
# The simplest use: a training set, a test set, and the score of the model fitted on the
# first and evaluated on the second. Same thing as
# evaluation.accuracy (predict (NB (d$train.x, d$train.y), d$test.x), d$test.y).
d = splitdata (iris, 5, seed = 0)
performance (NB, d$train.x, d$train.y, d$test.x, d$test.y)
# Several methods and criteria at once
performance (c (NB, LDA, CART), d$train.x, d$train.y, d$test.x, d$test.y,
             eval = c ("accuracy", "kappa"))
# One method, one evaluation criterion, bootstrap estimation
performance (NB, iris [, -5], iris [, 5], seed = 0)
# One method, two evaluation criteria, train set estimation
performance (NB, iris [, -5], iris [, 5], eval = c ("accuracy", "kappa"),
             protocol = "train", seed = 0)
# Three methods, ROC curves, LOOCV estimation
data (linsep)
performance (c (NB, LDA, LR), linsep [, -3], linsep [, 3], type = "roc",
             protocol = "loocv", seed = 0)
# Same curves, read from the hard predicted labels instead of the class-membership
# scores: each method collapses to a single operating point.
performance (c (NB, LDA, LR), linsep [, -3], linsep [, 3], type = "roc",
             protocol = "loocv", seed = 0, fuzzy = FALSE)
# Choosing the positive class explicitly
performance (NB, linsep [, -3], linsep [, 3], type = "roc", protocol = "loocv",
             seed = 0, positive = levels (linsep [, 3]) [2])
# List of methods in a variable, confusion matrix, hodout estimation
classif = c (NB, LDA, LR)
performance (classif, iris [, -5], iris [, 5], type = "confusion",
             protocol = "holdout", seed = 0, names = c ("NB", "LDA", "LR"))
# List of strings (method names), scatterplot evaluation, crossvalidation estimation
classif = c ("NB", "LDA", "LR")
performance (classif, iris [, -5], iris [, 5], type = "scatter",
             protocol = "crossvalidation", seed = 0)
# Actual vs. predicted
data (trees)
performance (LINREG, trees [, -3], trees [, 3], type = "avsp")

## End(Not run)

fdm2id documentation built on Aug. 28, 2026, 9:07 a.m.