validateEmulator: Validate the Emulator

Description Usage Arguments Details Value Author(s) References See Also


Uses diagnostics described in Bastos and O'Hagan (2009) to validate the emulator.


validateEmulator(emulator, new.outputs, emulator.predictions,
  which.output = 1, plot = TRUE)



Object of class emulatorFit generated by fitEmulator or fitEmulatorData


A matrix outputs used to validate the emulator predictions. This Matrix must have columns equal to the number of outputs.


Output generated from the predict.emulatorFit function.


Defaults to 1. Is an indicative integer that validates the output stated based on column number from the left.


Optionally plot validation plots. Default is set to true.


validateEmulator returns a grid of 4 plots and prints out a value. These are described below:

Top Left plot Plots validation output against the posterior mean generated using the emulator. The error bars are the 95percent confidence intervals of the predicted mean. Ideally, the plot should show all points close to the diagonal line with small error bars.
Top Right plot Shows the QQ plot of cholesky residuals. If the points lie close to the 45 degree line then the normality assumptions for the simulator output is reasonable. If the gradient of the points is greater than 1 (less than 1), it suggests that the predictive variability was underestimated (over estimated). Curvature in the plot indicates nonnormality, and outliers at either end of the plot suggests local fitting problems or nonstationarity.
Bottom Left plot Plot of Pivoted cholesky prediciton errors against the pivoting index. The pivoting index gives the order of the Pivoted cholesky prediciton errors with the largest conditional predictive variance. No patterns are expected. Too many large errors indicate an underestimation of variance and vice versa.Both cases can aslo suggest a nonstationary process. Either large or very small errors at the beginning of the plot (i.e., on the left side) indicates poor estimation of predictive variance or nonstationarity. However, large (or very small) errors at the end of the plot (i.e., on the right side) indicates overestimation (or underestimation) of the correlation length parameters, or that the chosen correlation structure is unsuitable.
Bottom right Plot The red line on the plot is the value of the calculated Mahalanobis distance taking into account correlation among outputs. Ideally it should be between the boundaries of the curve. Extreme values (large or small) indicate a conflict between the emulator and simulator.


The class of 'validateEmulator' is a list containting atleast the following components:

coverage Proportion of validation outputs in 95 percent confidence intervals of the emulator predicted output
RMSE The root mean square error of the predicted values
normRMSE The normalised value of RMSE over the range of the data


Originally written by Jeremy Oakley. Modified by Sajni Malde


Bastos, L. S. and O'Hagan, A. (2009). Diagnostics for gaussian process emulators, Technometrics, 51 (4): 425-438.

See Also

predict.emulatorFit and fitEmulator

OakleyJ/MUCM documentation built on May 7, 2019, 9:01 p.m.