pairwise_comparison: Do Pairwise Comparisons of Scores

View source: R/pairwise-comparisons.R

pairwise_comparisonR Documentation

Do Pairwise Comparisons of Scores


Make pairwise comparisons between models. The code for the pairwise comparisons is inspired by an implementation by Johannes Bracher.

The implementation of the permutation test follows the function permutationTest from the surveillance package by Michael Höhle, Andrea Riebler and Michaela Paul.


  by = c("model"),
  metric = "auto",
  baseline = NULL,



A data.table of scores as produced by score().


character vector with names of columns present in the input data.frame. by determines how pairwise comparisons will be computed. You will get a relative skill score for every grouping level determined in by. If, for example, by = c("model", "location"). Then you will get a separate relative skill score for every model in every location. Internally, the data.frame will be split according by (but removing "model" before splitting) and the pairwise comparisons will be computed separately for the split data.frames.


A character vector of length one with the metric to do the comparison on. The default is "auto", meaning that either "interval_score", "crps", or "brier_score" will be selected where available. See available_metrics() for available metrics.


character vector of length one that denotes the baseline model against which to compare other models.


additional arguments for the comparison between two models. See compare_two_models() for more information.


A ggplot2 object with a coloured table of summarised scores


Nikos Bosse

Johannes Bracher,


df <- data.frame(
  model = rep(c("model1", "model2", "model3"), each = 10),
  date = as.Date("2020-01-01") + rep(1:5, each = 2),
  location = c(1, 2),
  interval_score = (abs(rnorm(30))),
  ae_median = (abs(rnorm(30)))

res <- pairwise_comparison(df,
  baseline = "model1"

eval <- score(example_quantile)
pairwise_comparison(eval, by = c("model"))

scoringutils documentation built on May 14, 2022, 1:06 a.m.