review: Review a trained supervised model

In rtemis: Schema-Driven Machine Learning and Visualization

Review a trained supervised model

Description

Assess a trained model: whether its evaluation can be trusted given the sample size and the number of predictors, whether it performs better than a baseline that ignores the predictors, and whether it overfits. The review is deterministic: it draws no random numbers.

Usage

review(x, confidence_level = NULL, min_cases_per_predictor = NULL, ...)

Arguments

x	Supervised or SupervisedRes object: A trained model, as returned by train.
confidence_level	Optional Numeric (0, 1): Confidence level of every interval. NULL uses 0.95.
min_cases_per_predictor	Optional Numeric (0, Inf): Training cases per learner column – minority-class cases for classification – below which the review notes a shortfall. NULL uses 10, an events-per-variable rule of thumb from logistic regression (see References).
...	Not used.

Details

The review holds every value it rests on, so a reader can judge for themselves: ⁠@sample⁠ (sample sizes, input predictors and the columns the learner received), ⁠@class_counts⁠, ⁠@performance⁠ (one row per metric: training, test, training minus test, the spread over resamples, and the test interval or pooled value), ⁠@baseline⁠ (one row per comparison with a reference that ignores the predictors, naming the reference and the method) and ⁠@tuning⁠ (tuned hyperparameters against their search range). Its findings state what it observes. Printing shows a summary: one row per metric – mean (SD) over resamples for a resampled model – the baseline comparisons, the findings and the limitations.

Single split. The test cases are independent of the fitted model, so the review computes confidence intervals and two-sided comparisons at confidence_level. Accuracy is compared with always predicting the most common training class by the exact McNemar test, which pairs the two predictors' results case by case. Balanced accuracy and AUC are compared with their chance levels (1/K, 0.5) through their intervals: per-class binomial variances for balanced accuracy, the DeLong method for AUC. The Brier score, MSE and MAE are compared with a constant baseline (training proportion, training mean) through a paired t interval of the per-case loss reduction; the skill score is reported as a point estimate. An interval that cannot be computed – too few cases, a class absent from the test set, an AUC of 0 or 1 – is left unset and no verdict is made from it. A training value of the headline metric (balanced accuracy, or mean squared error for regression) outside its test interval is reported as a diagnostic sign of possible overfitting; it is not a test of the train-test difference.

Resampled models. Resamples share training cases, so their test results are dependent, and the review makes no interval or test from them. Every metric is reported as its mean and standard deviation over the outer resamples, each baseline is fit to the training data of its resample, and the review counts the resamples in which the model beat its baseline. When every case is tested once, as with k-fold resampling, pooled out-of-sample values are reported as descriptions for metrics that average over cases; AUC is not pooled, since it would rank scores from different fitted models together.

A tuned hyperparameter selected at the edge of the values searched is reported, since a better value may lie beyond it.

The cases-per-predictor check counts the columns the learner received, after preprocessing and decomposition. Its threshold is a rule of thumb from logistic regression (events per variable; see References), reported for context.

Preprocessing, decomposition and tuning inside train() are fitted on the training cases of each split and only applied to its test cases. Steps taken before the data is passed to train() – selecting predictors or transforming cases using all the data, or choosing among models by their test performance – can bias evaluation, and the fitted model cannot show whether they happened. Performance metrics also cannot establish whether a model is useful. Every review states both.

Print the result to read it; write_text writes it as Markdown.

Value

SupervisedReview object, whose tables are data.frames.

Author(s)

EDG

References

Clopper CJ, Pearson ES (1934). The use of confidence or fiducial limits illustrated in the case of the binomial. Biometrika, 26(4), 404-413.

DeLong ER, DeLong DM, Clarke-Pearson DL (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics, 44(3), 837-845.

McNemar Q (1947). Note on the sampling error of the difference between correlated proportions or percentages. Psychometrika, 12(2), 153-157.

Peduzzi P, Concato J, Kemper E, Holford TR, Feinstein AR (1996). A simulation study of the number of events per variable in logistic regression analysis. Journal of Clinical Epidemiology, 49(12), 1373-1379.

Examples

idx <- c(1:40, 51:90, 101:140)
mod <- train(
  iris[idx, ],
  dat_test = iris[-idx, ],
  hyperparameters = setup_CART(),
  verbosity = 0L
)
review(mod)

# Resampled
mod_res <- train(
  iris,
  hyperparameters = setup_CART(),
  outer_resampling_config = setup_KFold(5L),
  verbosity = 0L
)
review(mod_res)

rtemis documentation built on Oct. 20, 2026, 1:06 a.m.