classification.performance: Classification Performance Metrics

In randomForestSRC: Fast Unified Random Forests for Survival, Regression, and Classification (RF-SRC)

Classification Performance Metrics

Description

Evaluate classification predictions using confusion matrices, class-specific misclassification rates, ROC area under the curve (AUC), Brier score, log loss, and binary precision-recall summaries. The helpers take observed responses and predicted classes or probabilities directly. A class-assignment helper is also provided.

Usage

get.confusion(y, class.or.prob)

get.misclass.error(y, yhat)

get.auc(y, prob)

get.brier.error(y, prob, normalized = TRUE, vector = FALSE)

get.logloss(y, prob, robust = TRUE)

get.pr.auc(truth, yhat)

get.pr.curve(truth, yhat)

get.bayes.rule(prob, class.relfrq = NULL)

Arguments

y	Observed class labels, normally a factor. Keep the same factor levels and ordering as the probability columns. For get.auc(), a nonfactor response is also accepted, with probability columns in the order of sort(unique(y)). get.misclass.error() also accepts vectors of class labels.
prob	Numeric matrix with one row per observation and one column per class. Arrange columns in response-level order and name them with the corresponding class labels. get.auc() uses column positions even when names are present. get.logloss() and get.bayes.rule() require class column names; get.brier.error() can supply missing names from levels(y).
class.or.prob	For get.confusion(), either a factor of predicted labels with the same levels as y, or a matrix of class probabilities as described for prob. Missing matrix column names are supplied from levels(y).
yhat	For get.misclass.error(), predicted class labels aligned with y. For the precision-recall helpers, a numeric score vector, a one-column matrix or data frame, or a two-column matrix or data frame. A vector or single column contains scores for the positive class; larger scores favor that class. Scores need not lie in [0,1]. See Details for the class and column conventions.
truth	Binary observed responses for precision-recall summaries: a vector coded 0/1, or a factor with two levels. Code 1 is positive. For factors with labels other than 0 and 1, the less frequent class is positive.
normalized	If TRUE, use the normalized Brier scale described below, on which equal probabilities for all classes give score one. If FALSE, average the squared probability errors over classes.
vector	If TRUE, return the scaled Brier contribution for each class, rather than their sum. These are class-level contributions, not observation-level losses.
robust	For get.logloss(), remove infinite log losses before averaging. The default is TRUE. This does not clip probabilities or modify confusion counts.
class.relfrq	Optional vector of class relative frequencies in probability-column order, for a two-class problem. With NULL, get.bayes.rule() selects the largest class probability. Otherwise it applies the minority-prevalence cutoff described below.

Details

Supplying predictions

These helpers score the predictions supplied to them; they do not fit a forest or automatically choose OOB predictions. For a grow object, use o$yvar with o$predicted.oob for OOB probability scores, or o$class.oob for its stored class assignments. For new-data evaluation, use the prediction object's responses and predicted or class components.

Keep responses and predictions aligned by row. For a common scoring sample, restrict to observed responses and finite probabilities before calling the helpers. In particular, get.logloss() and get.misclass.error() require nonmissing observed labels. The examples illustrate this preparation for OOB predictions.

For a combined binary summary with sensitivity, specificity, F1, G-mean, and random-reference comparisons, use get.imbalanced.performance.

Class assignments and misclassification

With class.relfrq = NULL, get.bayes.rule() assigns each row to a class with the largest probability, sampling among tied maximizers. A row whose probabilities are all missing receives NA. With two class relative frequencies supplied, it assigns the minority class when that class's probability is at least its supplied relative frequency. Otherwise it assigns the majority class. If the frequencies tie, the first probability column is treated as the minority. Supply finite probabilities for this frequency-based rule.

get.confusion() has observed classes in rows and predicted classes in columns. Its final class.error column is one minus the fraction correctly classified within each observed class, rounded to four decimal places. Missing response/prediction pairs do not enter the counts. When supplied a probability matrix, it uses the largest-probability rule. To evaluate RFQ or another thresholded rule, supply that rule's predicted factor instead, for example o$class.oob.

get.misclass.error() returns one misclassification rate per observed class, in sort(unique(y)) order. It returns class-specific rates rather than an overall error rate. Missing predicted labels make the corresponding class rate unavailable.

ROC area under the curve

get.auc() uses the pairwise multiclass construction of Hand and Till. For each pair of classes, it calculates a rank-based AUC using each class's probability column as its score and averages the two directions. It then averages over available class pairs. For complementary two-class probabilities, the two directions agree. Larger values indicate better discrimination.

Missing observed labels and nonfinite scores are excluded from the relevant rank calculations. Each direction requires at least two finite scores in each of the two classes. Unavailable pairs are omitted; the result is NA when no pair is available.

Brier score

For J classes, let

b_j=\mathrm{mean}_i\left[(I(y_i=j)-p_{ij})^2\right].

The scalar returned by get.brier.error() is

\frac{J}{J-1}\sum_{j=1}^{J} b_j

with normalized = TRUE, and

\frac{1}{J}\sum_{j=1}^{J} b_j

with normalized = FALSE. Smaller values are better. Equal probabilities p_{ij}=1/J give normalized score one. For complementary binary probabilities, the unnormalized score is the usual mean squared error of either class probability, and the normalized score is four times that value.

With vector = TRUE, each returned element is b_j multiplied by the selected scaling constant. Missing losses are omitted separately within each class. The scalar sums available class contributions and is NA when none are available.

Log loss

With every response level represented, get.logloss() averages -\log(p_{i,y_i}), using the natural logarithm. Smaller values are better. With robust = TRUE, infinite losses from zero probabilities are excluded, rather than replaced using a probability floor. Use robust = FALSE to retain these infinite losses. Missing losses are excluded in either case. An unused factor level contributes one zero term to the helper's average.

Precision-recall summaries

These helpers apply to two classes. For truth coded 0/1, including a factor with those two labels, 1 is positive. For other two-level factors, the less frequent observed class is positive; ties select the first factor level. This choice is made before excluding missing scores.

A score vector or single column always refers to the positive class. With two columns, named columns are matched to the original response levels; unnamed columns follow factor-level order, or 0, 1 order for a nonfactor response. The positive-class column is then selected. Rows with a missing response or nonfinite selected score are excluded.

get.pr.auc() returns the precision-recall area and a random reference area equal to the positive-class proportion among the scored observations. The area uses the analytic precision-recall interpolation implemented by the package. get.pr.curve() supplies recall, precision, and threshold values for plotting; its rows run from high to low recall. Both classes must have a usable score. Larger precision-recall areas are better.

Value

get.confusion

A numeric matrix of class counts with an additional class.error column.

get.misclass.error

An unnamed numeric vector of class-specific error rates, in sort(unique(y)) order.

get.auc

A scalar AUC, or NA when unavailable.

get.brier.error

A scalar Brier score by default, or an unnamed length-J vector of scaled class contributions in probability-column order when vector = TRUE.

get.logloss

A scalar mean log loss. It can be infinite with robust = FALSE, or NaN when no losses remain for averaging.

get.pr.auc

An unnamed numeric vector of length two: the model area followed by the random reference area. Both entries are NA when the calculation is unavailable.

get.pr.curve

A numeric matrix with columns recall, precision, and threshold, or NULL when the calculation is unavailable.

get.bayes.rule

A factor of predicted classes with levels given by the probability-column names.

See Also

rfsrc, predict.rfsrc, imbalanced.rfsrc, get.imbalanced.performance, get.brier.survival, get.auct.survival

Examples

## ------------------------------------------------------------
## A basic calculation from observed labels and probabilities
## ------------------------------------------------------------
y <- factor(c("no", "no", "no", "no", "yes", "yes"),
            levels = c("no", "yes"))
p <- c(.10, .20, .65, .35, .40, .85)
prob <- cbind(no = 1 - p, yes = p)
print(get.confusion(y, prob))
print(get.auc(y, prob))
print(get.brier.error(y, prob))


## ------------------------------------------------------------
## Class-specific errors and probability losses
## ------------------------------------------------------------
yhat <- get.bayes.rule(prob)
print(setNames(get.misclass.error(y, yhat), levels(y)))
print(get.brier.error(y, prob, normalized = FALSE))
print(setNames(get.brier.error(y, prob, vector = TRUE), colnames(prob)))
print(get.logloss(y, prob))

## Use supplied class frequencies for the binary RFQ decision rule.
class.frq <- as.numeric(prop.table(table(y)))
yhat.rfq <- get.bayes.rule(prob, class.relfrq = class.frq)
print(get.confusion(y, yhat.rfq))

## ------------------------------------------------------------
## Precision-recall from a score vector
## ------------------------------------------------------------
truth <- as.integer(y == "yes")
pr.auc <- get.pr.auc(truth, p)
print(setNames(pr.auc, c("model", "random")))
pr <- get.pr.curve(truth, p)
plot(pr[, "recall"], pr[, "precision"], type = "l",
     xlim = c(0, 1), ylim = c(0, 1),
     xlab = "Recall", ylab = "Precision")
abline(h = pr.auc[2], lty = 2)

## A single score column is also accepted.
print(get.pr.auc(truth, matrix(p, ncol = 1)))

## ------------------------------------------------------------
## Multiclass forest: OOB and test-data performance
## ------------------------------------------------------------
set.seed(17)
train <- c(1:35, 51:85, 101:135)
o <- rfsrc(Species ~ ., data = iris[train, ], ntree = 100)

## Select one common set of observed responses and finite OOB scores.
p.oob <- o$predicted.oob
keep <- !is.na(o$yvar) & rowSums(!is.finite(p.oob)) == 0
print(get.confusion(o$yvar[keep], o$class.oob[keep]))
print(get.auc(o$yvar[keep], p.oob[keep, , drop = FALSE]))

## Use the current responses and probabilities for test-data scoring.
p.test <- predict(o, newdata = iris[-train, ])
print(c(
  auc = get.auc(p.test$yvar, p.test$predicted),
  brier = get.brier.error(p.test$yvar, p.test$predicted),
  logloss = get.logloss(p.test$yvar, p.test$predicted)
))

randomForestSRC documentation built on Sept. 16, 2026, 5:06 p.m.