assess.accuracy: Assess Accuracy

View source: R/output_accuracy.R

assess.accuracyR Documentation

Assess Accuracy

Description

High level handler function to decide which metrics to compute and then call the code to compute them. At present, only continuous and discrete outcomes are supported.

Usage

assess.accuracy(
  predictions.mat,
  observations.vec,
  model.vec,
  locations.vec,
  year.vec,
  forecast.target,
  threshold = "default",
  percentage = "default"
)

Arguments

predictions.mat

A matrix where each row corresponds to a separate forecast, and every column corresponds to a forecast realization. In the case of point forecasts, there will be a single column. In the case of probabilistic forecasts, there will likely be a thousand or more columns. Note that these are realizations, and not the probabilities associated with categories.

observations.vec

A vector where each entry corresponds to the observed outcome for the corresponding row in predictions.mat.

model.vec

A vector indicating which model is being evaluated

locations.vec

A vector giving the location corresponding to each entry in observations.vec

year.vec

A vector giving the year corresponding toeach entry in observations.vec

forecast.target

The quantity being forecast. Forecasts targets are:

  • annual.human.cases

  • human_incidence

  • seasonal.mosquito.MLE

  • peak_mosquito_MLE

  • number_positive_pools

  • human_cases_binary

  • positive_pools_binary

  • peak_timing

threshold

For continuous and discrete forecasts, a threshold of error to be used in classifying the forecast as "accurate".

percentage

For continuous and discrete forecasts, if the prediction is within the specified percentage of the observed value, the forecast is considered accurate.

Value

accuracy.metrics A list containing accuracy information. The list is structured

RMSE Root Mean Squared Error
Scaled_RMSE RMSE scaled by the mean observed value
within_percentage A binary accuracy classification, where a forecast within a specified percentage of the observation is considered accurate, otherwise it is inaccurate.
within_threshold A binary accuracy classification, where a forecast within a specified threshold from the observation is considered accurate, otherwise it is inaccurate
within_threshold_or_percentage A binary accuracy classification, where if it is accurate by either the threshold or percentage approaches, it is considered accurate.
AUC Area Under the Curve from the Receiver Operating Characteristic Plot

akeyel/dfmip documentation built on Sept. 3, 2022, 1:26 a.m.