mlr_measures_surv.calib_index: Integrated Calibration Index Survival Measure

In mlr-org/mlr3proba: Probabilistic Supervised Learning for 'mlr3'

Integrated Calibration Index Survival Measure

Description

Calculates the Integrated Calibration Index (ICI), which evaluates point-calibration (i.e. at a specific time point), see Austin et al. (2020).

Details

Each individual i from the test set, has an observed survival outcome (t_i, \delta_i) (time and censoring indicator) and predicted survival function S_i(t). The predicted probability of an event occurring before a specific time point t_0, is defined as \hat{P_i}(t_0) = F_i(t_0) = 1 - S_i(t_0).

Using hazard regression (via the polspline R package), a smoothed calibration curve is estimated by fitting the following model:

log(h(t)) = g(log(− log(1 − \hat{P}_{t_0})), t)

Note that we substitute probabilities \hat{P}_{t_0} = 0 with a small \epsilon number to avoid arithmetic issues (log(0)). Same with \hat{P}_{t_0} = 1, we use 1 - \epsilon. From this model, the smoothed probability of occurrence at t_0 for observation i is obtained as \hat{P}_i^c(t_0).

The Integrated Calibration Index is then computed across the N test set observations as:

ICI = \frac{1}{N} \sum_{i=1}^N | \hat{P}_i^c(t_0) - \hat{P}_i(t_0) |

Therefore, a perfect calibration (smoothed probabilities match predicted probabilities for all observations) yields ICI = 0, while the worst possible score is ICI = 1.

Dictionary

This Measure can be instantiated via the dictionary mlr_measures or with the associated sugar function msr():

MeasureSurvICI$new()
mlr_measures$get("surv.calib_index")
msr("surv.calib_index")

Parameters

Id	Type	Default	Levels	Range
time	numeric	-		[0, \infty)
eps	numeric	1e-04		[0, 1]
method	character	ICI	ICI, E50, E90, Emax	-
na.rm	logical	TRUE	TRUE, FALSE	-

Meta Information

• Type: "surv"

• Range: [0, 1]

• Minimize: TRUE

• Required prediction: distr

Parameter details

• eps (numeric(1))
  Very small number to substitute zero values in order to prevent errors in e.g. log(0) and/or division-by-zero calculations. Default value is 1e-04.

• time (numeric(1))
  The specific time point t_0 at which calibration is evaluated. If NULL, the median observed time from the test set is used.

• method (character(1))
  Specifies the summary statistic used to calculate the final calibration score.

  • "ICI" (default): Uses the mean of absolute differences | \hat{P}_i^c(t_0) - \hat{P}_i(t_0) | across all observations.

  • "E50": Uses the median of absolute differences instead of the mean.

  • "E90": Uses the 90th percentile of absolute differences, emphasizing higher deviations.

  • "Emax": Uses the maximum absolute difference, capturing the largest discrepancy between predicted and smoothed probabilities.

• na.rm (logical(1))
  If TRUE (default) then removes any NAs/NaNs in the smoothed probabilities \hat{P}_i^c(t_0) that may arise. A warning is issued nonetheless in such cases.

Super classes

mlr3::Measure -> mlr3proba::MeasureSurv -> MeasureSurvICI

Methods

Public methods

• MeasureSurvICI$new()

• MeasureSurvICI$clone()

Inherited methods

• mlr3::Measure$aggregate()
• mlr3::Measure$format()
• mlr3::Measure$help()
• mlr3::Measure$print()
• mlr3::Measure$score()

---

Method new()

Creates a new instance of this R6 class.

Usage

MeasureSurvICI$new()

---

Method clone()

The objects of this class are cloneable with this method.

Usage

MeasureSurvICI$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

References

Austin, C. P, Harrell, E. F, van Klaveren, David (2020). “Graphical calibration curves and the integrated calibration index (ICI) for survival models.” Statistics in Medicine, 39(21), 2714. ISSN 10970258, \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1002/SIM.8570")}, https://pmc.ncbi.nlm.nih.gov/articles/PMC7497089/.

See Also

Other survival measures: mlr_measures_surv.calib_alpha, mlr_measures_surv.calib_beta, mlr_measures_surv.chambless_auc, mlr_measures_surv.cindex, mlr_measures_surv.dcalib, mlr_measures_surv.graf, mlr_measures_surv.hung_auc, mlr_measures_surv.intlogloss, mlr_measures_surv.logloss, mlr_measures_surv.mae, mlr_measures_surv.mse, mlr_measures_surv.nagelk_r2, mlr_measures_surv.oquigley_r2, mlr_measures_surv.rcll, mlr_measures_surv.rmse, mlr_measures_surv.schmid, mlr_measures_surv.song_auc, mlr_measures_surv.song_tnr, mlr_measures_surv.song_tpr, mlr_measures_surv.uno_auc, mlr_measures_surv.uno_tnr, mlr_measures_surv.uno_tpr, mlr_measures_surv.xu_r2

Other calibration survival measures: mlr_measures_surv.calib_alpha, mlr_measures_surv.calib_beta, mlr_measures_surv.dcalib

Other distr survival measures: mlr_measures_surv.calib_alpha, mlr_measures_surv.dcalib, mlr_measures_surv.graf, mlr_measures_surv.intlogloss, mlr_measures_surv.logloss, mlr_measures_surv.rcll, mlr_measures_surv.schmid

Examples

library(mlr3)

# Define a survival Task
task = tsk("lung")

# Create train and test set
part = partition(task)

# Train Cox learner on the train set
cox = lrn("surv.coxph")
cox$train(task, row_ids = part$train)

# Make predictions for the test set
p = cox$predict(task, row_ids = part$test)

# ICI at median test set time
p$score(msr("surv.calib_index"))

# ICI at specific time point
p$score(msr("surv.calib_index", time = 365))

# E50 at specific time point
p$score(msr("surv.calib_index", method = "E50", time = 365))

mlr-org/mlr3proba documentation built on April 12, 2025, 4:38 p.m.