compare_two_models: Compare Two Models Based on Subset of Common Forecasts
In scoringutils: Utilities for Scoring and Assessing Predictions

compare_two_models

R Documentation

Compare Two Models Based on Subset of Common Forecasts

Description

This function compares two models based on the subset of forecasts for which both models have made a prediction. It gets called from pairwise_comparison_one_group(), which handles the comparison of multiple models on a single set of forecasts (there are no subsets of forecasts to be distinguished). pairwise_comparison_one_group() in turn gets called from from pairwise_comparison() which can handle pairwise comparisons for a set of forecasts with multiple subsets, e.g. pairwise comparisons for one set of forecasts, but done separately for two different forecast targets.

Usage

compare_two_models(
  scores,
  name_model1,
  name_model2,
  metric,
  one_sided = FALSE,
  test_type = c("non_parametric", "permutation"),
  n_permutations = 999
)

Arguments

`scores`	A data.table of scores as produced by `score()`.
`name_model1`	character, name of the first model
`name_model2`	character, name of the model to compare against
`metric`	A character vector of length one with the metric to do the comparison on. The default is "auto", meaning that either "interval_score", "crps", or "brier_score" will be selected where available. See `available_metrics()` for available metrics.
`one_sided`	Boolean, default is `FALSE`, whether two conduct a one-sided instead of a two-sided test to determine significance in a pairwise comparison.
`test_type`	character, either "non_parametric" (the default) or "permutation". This determines which kind of test shall be conducted to determine p-values.
`n_permutations`	numeric, the number of permutations for a permutation test. Default is 999.