eval_function: Functions to evaluate a model
In signaux-faibles/MLsegmentr: Model evaluation on data segments

Description Usage Arguments Details Value Wrapped output and Unnesting behavior

Stores functions and options used to evaluate a model.

1
2
3

eval_function(eval_fun, plot_fun = NULL, cv_summary_fun = NULL,
  wrap_eval_fun = TRUE, unnest = TRUE, compulsory_fields = NULL,
  additional_eval_parameters = NULL, additional_plot_parameters = NULL)

`eval_fun`	:: 'function(1)' Function which takes as input an eval_frame, and returns its evaluation. The way the evaluation is applied to data depends on values of 'wrap_eval_fun' and 'unnest' see below.
`plot_fun`	:: 'function(1)' An optional plotting function that takes the evaluation as input.
`cv_summary_fun`	:: work in progress
`wrap_eval_fun`	:: 'logical(1)' If wrap_eval_fun is TRUE (default), then the evaluation function is applied independently to each (model x target x segment combination), and results are binded into a data.frame (see 'Wrapped output'). This can be used for single output or multi-output evaluation functions (see 'Unnesting behavior').
`unnest`	:: 'logical(1)' Only relevant if wrap_eval_fun is TRUE. If TRUE (default), the results of the evaluation function are unnested, otherwise they are kept as is in a list column. See 'Unnesting behavior' for details.
`compulsory_fields`	:: 'character()' Fields that are needed in the eval_frame for the evaluation function to work properly. When evaluating an eval frame, those fields will be checked. In the Assesser object, these fields are fetched in the test_data, and an error is thrown if they are not found there.
`additional_eval_parameters`	:: 'list(any)' NOT IMPLEMENTED YET. Named list to pass to the evaluation function. Extra parameters can either be saved to the object using this attribute (advised if the parameters do not change once the object is created), or passed to the assess function of the super Assesser object (advised if the parameters may change). Mixing both methods works: some parameters may be saved, and other passed as arguments.
`additional_plot_parameters`	:: 'list(any)' NOT IMPLEMENTED YET. Named list to pass to the plot function. Extra parameters can either be saved to the object using this attribute (advised if the parameters do not change once the object is created), or passed to the assess function of the super Assesser object (advised if the parameters may change). Mixing both methods works: some parameters may be saved, and other passed as arguments.

The evaluation separates two steps. First, the eval_fun evaluation function summarizes different metrics for each combination of (model x target x segment). The result of this evaluation is given as a dataframe, which is in turn optionnaly passed to a plotting function.

wrap_eval_fun is an option to define how eval_fun is applied; unnest is an option to define if the output should be simplified.

An S3-object of class "eval_functions"

If wrap_eval_fun, then the output (before unnesting) will be a dataframe with columns "model", "target_type", "segment", and "evaluation". Each row has a unique combination of (model x target_type x segment). Evaluation is a list column, with for each row the result of the evaluation function.

The list column can automatically unnested if unnest = TRUE. Unnesting works for the following output types of the evaluation function: * If the output is dimension 1 and length 1, the list column is simply replaced by a column of the same type than this output. * If the output is a (named) vector, then the list column is gathered, into an evaluation column of the same type as the vector and an evaluation_type column with the same names. * If the output is a (named) list, then the list column is gathered, into an evaluation column of type list (even if the list items have same type: in this case, prefer a named vector), and an evaluation_type column with the same names. * If the output is a dataframe, then the data.frame columns are spread in several columns, keeping the dataframe column names. Row names are lost (are they ? TODO)

signaux-faibles/MLsegmentr documentation built on Aug. 29, 2019, 2:22 p.m.