diagnostic_pass: Automated Diagnostics and Modifications

In hmer: History Matching and Emulation Package

Automated Diagnostics and Modifications

Description

Perform a set of diagnostics on emulators, changing them if needed.

Usage

diagnostic_pass(
  ems,
  targets,
  validation,
  check_output = FALSE,
  verbose = interactive(),
  ...
)

Arguments

ems	The emulators to consider
targets	The output targets to compare implausibility against
validation	The set of validation points (either a single data.frame or one per emulator)
check_output	Whether to check for suitability of outputs re. targets
verbose	Whether messages should be printed while running
...	Other arguments to pass to helper functions

Details

NOTE: Automated diagnostics are currently only supported for deterministic emulators.

There are a number of different characteristics that emulators might possess that give rise to diagnostic flags. This function collects together some of those whose resulting modifications can be automated. The tests, and consequences, are as follows.

Structured Input Space

Looks for errors with dependence on input parameters. If found, the emulator's correlation length is reduced (to a minimum of 1/3);

Structured Output Space

Looks for errors with dependence on output value. If found, the training and validation data is resampled and emulators are retrained, to try to incorporate/remove high leverage points;

Misclassification

Checks agreement between emulator and simulator implausibility classification. If they do not match, emulator uncertainty is inflated;

Comparison

Checks that the emulator predictions agree with the simulator predictions at the validation points, allowing for expected margin of error.

If the automated modifications are not sufficient to remove problems, then offending emulators are removed from the set under consideration. Emulators in this category should be carefully considered and their outputs analyzed: they may require manual determination of the regression surface or additional training points in the neightbourhood of the problematic inputs.

The validation set can also be checked for suitability independent of emulator structure: if check_output = TRUE then the outputs of the validation set will be compared against targets, as well as checking the implausibility of the points with respect to the emulators. If any outputs are consistent under- or over-estimates, or if all points are to be ruled out as implausible, the emulators corresponding to these outputs are removed. This option should be used with care: such a situation could be informative for considering the model structure and whether one should expect a match to observational data.

Value

A collection of modified emulators, potentially a subset of the original collection

Examples

 new_ems <- diagnostic_pass(SIREmulators$ems, SIREmulators$targets, SIRSample$validation)

hmer documentation built on June 22, 2024, 9:22 a.m.