fill_scores: fill scores based on a null

In eco4cast/score4cast: Scoring Utilities for Forecast Competitions

fill scores based on a null

Description

Comparing mean scores across teams requires appropriate treatment of missing values: teams should not be able to improve scores merely by refusing to provide predictions of, e.g. sites or times which are hardest to predict. To avoid this, merely removing missing values when averaging across scores is not sufficient. A simple expedient is to replace missing values with predictions made from a baseline 'null' forecast. This function simply provides this behavior. Original forecast scores with missing values are retained as crps_model and logs_model columns, while crps and logs become filled with baseline scores from the null forecast.

Usage

fill_scores(df, null_model = "EFInull")

Arguments

df	a data frame of forecasts, with column "team" identifying different forecasts.
null_model	the "team" name identifying the baseline (null) forecast used for filling missing values.

Details

Note that this fills implicit NAs, e.g. site/time/variables predicted by the null team but not predicted by the focal team, as well as explicit NAs in the focal team (where the focal team includes each of teams named in the teams column of df). If teams have scores for site/time/variables not forecasted by the "null" team, these rows are removed and thus cannot contribute to the mean score either. If the "null" team contains explicit NAs in scores, (usually but not always due to missing observations), these are not removed.

eco4cast/score4cast documentation built on Nov. 21, 2023, 12:25 p.m.