compare: Compare predictions and labels

In maxwestphal/SEPM: Statistical Evaluation of Prediction Models

Compare predictions and labels

Description

This function may be used to compare predictions and labels of multiple prediction models.

Usage

compare(
  hypothesis,
  comparison,
  predictions,
  labels,
  features,
  models,
  model.names,
  pos.label,
  ...
)

Arguments

hypothesis	object of class SEPM.hypothesis, see ?define_hypothesis.
comparison	numeric matrix containing n observations of (dis)similarities between labels and predictions (rows) for S candidate models (columns). If objective$co.primary is TRUE, comparison needs to be a (named) list of two such matrices.
predictions	numeric matrix with n observations (rows) with predictions from S models (columns).
labels	vector of length n containing the true labels.
features	matrix or data.frame with n observations (rows) of P features (columns).
models	named list of prediction models, i.e. list of R objects for which predict(object, newdata=features, ...) produces the predictions for the given features.
model.names	optional character of length S specifyng model names. By default column names of comparison or predictions or names(models) are used.
pos.label	specify the positive label, only used if hypothesis$co.primary is TRUE. If the labels are 0,1 (FALSE, TRUE), 1 (TRUE) is assumed to represent the positive class by default.
...	further arguments (e.g. type) passed to predict for compare mode (3).

Details

One of the following has to be specified: (1) comparison, (2) predictions and labels or (3) labels and features and models. A message indicates which of these compare modes is used to obtain the result. If the requirements for multiple compare modes are met, the result will be calculated based on the priority (1) > (2) > (3). Compare mode (2) should be the default. Mode (1) basically assumes the output has been computed beforehand which might be useful for simulations.

Value

SEPM.comparison object containing a matrix including the compared (predicted, actual) pairs and the input hypothesis.

Examples

y <- rep(1:0, times=c(3,7))
yhat <- cbind(model1 = rep(1:0, 5),
              model2 = rep(0, 10),
              model3 = rep(1:0, times=c(2,8)))
define_hypothesis("accuracy", threshold = 0.75) %>%
compare(predictions = yhat, labels = y)

maxwestphal/SEPM documentation built on April 12, 2024, 12:09 a.m.