dx_brier: Calculate Brier Score

In overdodactyl/diagnosticSummary: Diagnostic Summaries

Calculate Brier Score

Description

The Brier score is a proper score function that measures the accuracy of probabilistic predictions. It is applicable to tasks in which predictions must assign probabilities to a set of mutually exclusive discrete outcomes. For binary classification, the Brier score is a measure of how far the predicted probabilities are from the actual outcomes.

Usage

dx_brier(predprob, truth, detail = "full")

Arguments

predprob	Numeric vector of predicted probabilities associated with the positive class.
truth	Numeric vector of true binary outcomes, typically 0 or 1, with the same length as predprob.
detail	Character specifying the level of detail in the output: "simple" for raw estimate, "full" for detailed estimate including 95% confidence intervals.

Details

The formula for the Brier score in a binary classification is:

BS = \frac{1}{N} \sum_{i=1}^{N} (f_i - o_i)^2

where:

• \(N\) is the number of predictions,

• \(f_i\) is the predicted probability of the occurrence of the positive class for the ith prediction,

• \(o_i\) is the actual outcome for the ith prediction, 0 or 1.

The Brier score ranges from 0 to 1, where 0 represents a perfect model and 1 represents the worst model. It is equivalent to the mean squared error used in regression and can be decomposed into calibration loss, refinement loss, and uncertainty. This makes it a very informative metric for probabilistic forecasts, providing a nuanced view of the model's predictive performance.

Value

Depending on the detail parameter, returns a numeric value representing the calculated metric or a data frame/tibble with detailed diagnostics including confidence intervals and possibly other metrics relevant to understanding the metric.

Examples

predprob <- dx_heart_failure$predicted
truth <- dx_heart_failure$truth
simple <- dx_brier(predprob, truth, detail = "simple")
detailed <- dx_brier(predprob, truth)
print(simple)
print(detailed)

overdodactyl/diagnosticSummary documentation built on Jan. 28, 2024, 10:07 a.m.