compute_metrics: Calculate and format metrics from tuning functions

In tune: Tidy Tuning Tools

Calculate and format metrics from tuning functions

Description

This function computes metrics from tuning results. The arguments and output formats are closely related to those from collect_metrics(), but this function additionally takes a metrics argument with a metric set for new metrics to compute. This allows for computing new performance metrics without requiring users to re-evaluate models against resamples.

Note that the control option save_pred = TRUE must have been supplied when generating x.

Usage

compute_metrics(x, metrics, summarize, event_level, ...)

## Default S3 method:
compute_metrics(x, metrics, summarize = TRUE, event_level = "first", ...)

## S3 method for class 'tune_results'
compute_metrics(x, metrics, ..., summarize = TRUE, event_level = "first")

Arguments

x	The results of a tuning function like tune_grid() or fit_resamples(), generated with the control option save_pred = TRUE.
metrics	A metric set of new metrics to compute. See the "Details" section below for more information.
summarize	A single logical value indicating whether metrics should be summarized over resamples (TRUE) or return the values for each individual resample. See collect_metrics() for more details on how metrics are summarized.
event_level	A single string containing either "first" or "second". This argument is passed on to yardstick metric functions when any type of class prediction is made, and specifies which level of the outcome is considered the "event".
...	Not currently used.

Details

Each metric in the set supplied to the metrics argument must have a metric type (usually "numeric", "class", or "prob") that matches some metric evaluated when generating x. e.g. For example, if x was generated with only hard "class" metrics, this function can't compute metrics that take in class probabilities ("prob".) By default, the tuning functions used to generate x compute metrics of all needed types.

Value

A tibble. See collect_metrics() for more details on the return value.

Examples

# load needed packages:
library(parsnip)
library(rsample)
library(yardstick)

# evaluate a linear regression against resamples.
# note that we pass `save_pred = TRUE`:
res <-
  fit_resamples(
    linear_reg(),
    mpg ~ cyl + hp,
    bootstraps(mtcars, 5),
    control = control_grid(save_pred = TRUE)
  )

# to return the metrics supplied to `fit_resamples()`:
collect_metrics(res)

# to compute new metrics:
compute_metrics(res, metric_set(mae))

# if `metrics` is the same as that passed to `fit_resamples()`,
# then `collect_metrics()` and `compute_metrics()` give the same
# output, though `compute_metrics()` is quite a bit slower:
all.equal(
  collect_metrics(res),
  compute_metrics(res, metric_set(rmse, rsq))
)

tune documentation built on May 29, 2024, 7:32 a.m.