compare_two_models: Compare two models based on subset of common forecasts
In epiforecasts/scoringutils: Utilities for Scoring and Assessing Predictions

compare_two_models

R Documentation

Compare two models based on subset of common forecasts

Description

This function compares two models based on the subset of forecasts for which both models have made a prediction. It gets called from pairwise_comparison_one_group(), which handles the comparison of multiple models on a single set of forecasts (there are no subsets of forecasts to be distinguished). pairwise_comparison_one_group() in turn gets called from from get_pairwise_comparisons() which can handle pairwise comparisons for a set of forecasts with multiple subsets, e.g. pairwise comparisons for one set of forecasts, but done separately for two different forecast targets.

Usage

compare_two_models(
  scores,
  name_model1,
  name_model2,
  metric,
  one_sided = FALSE,
  test_type = c("non_parametric", "permutation"),
  n_permutations = 999
)

Arguments

`scores`	An object of class `scores` (a data.table with scores and an additional attribute `metrics` as produced by `score()`).
`name_model1`	Character, name of the first model
`name_model2`	Character, name of the model to compare against
`metric`	A string with the name of the metric for which a relative skill shall be computed. By default this is either "crps", "wis" or "brier_score" if any of these are available.
`one_sided`	Boolean, default is `FALSE`, whether two conduct a one-sided instead of a two-sided test to determine significance in a pairwise comparison.
`test_type`	Character, either "non_parametric" (the default) or "permutation". This determines which kind of test shall be conducted to determine p-values.
`n_permutations`	Numeric, the number of permutations for a permutation test. Default is 999.