Measure: Measure Class

In mllg/mlr3: Machine Learning in R - Next Generation

Description

This is the abstract base class for measures like MeasureClassif and MeasureRegr.

Measures are classes around tailored around two functions:

1. A function score which quantifies the performance by comparing true and predicted response.

2. A function aggregator which combines multiple performance scores returned by calculate to a single numeric value.

In addition to these two functions, meta-information about the performance measure is stored.

Predefined measures are stored in the mlr3misc::Dictionary mlr_measures, e.g. classif.auc or time_train. A guide on how to extend mlr3 with custom measures can be found in the mlr3book.

Format

R6::R6Class object.

Construction

Note: This object is typically constructed via a derived classes, e.g. MeasureClassif or MeasureRegr.

m = Measure$new(id, task_type = NA, range = c(-Inf, Inf), minimize = NA, aggregator = NULL, properties = character(), predict_type = "response", predict_sets = "test",
     task_properties = character(), packages = character())

• id :: character(1)
  Identifier for the measure.

• task_type :: character(1)
  Type of the task the measure can operator on. E.g., "classif" or "regr".

• range :: numeric(2)
  Feasible range for this measure as c(lower_bound, upper_bound). Both bounds may be infinite.

• minimize :: logical(1)
  Set to TRUE if good predictions correspond to small values, and to FALSE if good predictions correspond to large values. If set to NA (default), tuning this measure is not possible.

• aggregator :: function(x)
  Function to aggregate individual performance scores x where x is a numeric vector. If NULL, defaults to mean().

• properties :: character()
  Properties of the measure. Must be a subset of mlr_reflections$measure_properties. Supported by mlr3:

  • "requires_task" (requires the complete Task),

  • "requires_learner" (requires the trained Learner),

  • "requires_train_set" (requires the training indices from the Resampling), and

  • "na_score" (the measure is expected to occasionally return NA).

• predict_type :: character(1)
  Required predict type of the Learner. Possible values are stored in mlr_reflections$learner_predict_types.

• predict_sets :: character()
  Prediction sets to operate on, used in aggregate() to extract the matching predict_sets from the ResampleResult. Multiple predict sets are calculated by the respective Learner during resample()/benchmark(). Must be a non-empty subset of c("train", "test"). If multiple sets are provided, these are first combined to a single prediction object. Default is "test".

• task_properties :: character()
  Required task properties, see Task.

• packages :: character()
  Set of required packages. Note that these packages will be loaded via requireNamespace(), and are not attached.

Fields

All variables passed to the constructor.

Methods

• aggregate(rr)
  ResampleResult -> numeric(1)
  Aggregates multiple performance scores into a single score using the aggregator function of the measure. Operates on the Predictions of ResampleResult with matching predict_sets.

• score(prediction, task = NULL, learner = NULL, train_set = NULL)
  ((named list of) Prediction, Task, Learner, integer() | character()) -> numeric(1)
  Takes a Prediction (or a list of Prediction objects named with valid predict_sets) and calculates a numeric score. If the measure if flagged with the properties "requires_task", "requires_learner" or "requires_train_set", you must additionally pass the respective Task, the trained Learner or the training set indices. This is handled internally during resample()/benchmark().

See Also

Other Measure: MeasureClassif, MeasureRegr, mlr_measures

mllg/mlr3 documentation built on Sept. 27, 2019, 9:38 a.m.