amber-package: Overview of AMBER functions
In amber: Automated Model Benchmarking R Package

Description amber functions Skill scores References

The Canadian Land Surface Scheme Including Biogeochemical Cycles (CLASSIC) is the land surface component of the Canadian Earth System Model (CanESM) (Melton et al, 2020). The Automated Model Benchmarking R package (AMBER) evaluates the ability of CLASSIC to reproduce land surface processes by comparing model outputs against quasi-observational data sets derived from remote sensing products, eddy covariance flux tower measurements, and stream flow measurements. To summarize model performance across different statistical metrics AMBER employs a skill score system that was originally developed by the International Land Model Benchmarking (ILAMB) framework (Collier et al., 2018). AMBER was created to tailor the ILAMB skill score approach for CLASSIC model outputs. While AMBER was tested for CLASSIC only it may also work for other models.

The functions provided by AMBER can be grouped into three categories:

Functions that compute skill scores and other metrics for a single variable. This includes: scores.fluxnet.csv, scores.fluxnet.nc, scores.fluxnet.site, scores.functional.response, scores.grid.notime, scores.grid.time, scores.runoff, and scores.site.notime.
Functions that visualize model output for a single variable. This includes: plotEnsembleHovmoeller, plotEnsembleMean, plotGrid, plotHovmoeller, plotNcIrreg, plotNc, plotZonalMeans, plotZonalMeanStats, seasonalCycleIrreg, seasonalCycle, zonalMeanIrreg, zonalMean, and zonalMeanStats.
Functions that visualize summary statistics across multiple variables. This includes: scores.compare.benchmarks, scores.compare.ensemble, scores.compare, scores.tables, scores.tables.tweak, correlationMatrixDiff, correlationMatrixFluxnet, correlationMatrix, globalSumsTable, metrics.compare, plotBars, and plotFluxnetStats.

The suffix Irreg indicates that this function may be applied to data that is on an irregular grid. In the case of CLASSIC, this applies to high-resolution simulations that are conducted for the Canadian domain. The string ensemble implies that the function is designed to handle multiple model runs.

The performance of a model is expressed through scores that range from zero to one, where increasing values imply better performance. These scores are usually computed in five steps:

(i) computation of a statistical metric,
(ii) nondimensionalization,
(iii) conversion to unit interval,
(iv) spatial integration, and
(v) averaging scores computed from different statistical metrics.

The latter includes the bias, root-mean-square error, phase shift, inter-annual variability, and spatial distribution. The equations for computing the bias score (S_{bias}) are:

(i) \ bias(λ, φ)=\overline{v_{mod}}(λ, φ)-\overline{v_{ref}}(λ, φ)

(ii) \ \varepsilon_{bias}=|bias(λ, φ)|/σ_{ref}(λ, φ)

(iii) \ s_{bias}(λ, φ)=e^{-\varepsilon_{bias}(λ, φ)}

(iv) \ S_{bias}=\overline{\overline{s_{bias}}}

The equations for computing the root mean square error score (S_{rmse}) are:

(i) \ crmse(λ, φ) =√{\frac{1}{t_{f}-t_{0}}\int_{t_{0}}^{t_{f}}[(v_{mod}(t,λ, φ)-\overline{v_{mod}}(λ, φ))-(v_{ref}(t,λ, φ)-\overline{v_{ref}}(λ, φ))]^{2}dt}

(ii) \ \varepsilon_{rmse}(λ, φ)=crmse(λ, φ)/σ_{ref}(λ, φ)

(iii) \ s_{rmse}(λ, φ)=e^{-\varepsilon_{rmse}(λ, φ)}

(iv) \ S_{rmse}=\overline{\overline{s_{rmse}}}

The equations for computing the phase score (S_{phase}) are:

(i) \ θ(λ, φ)=\max(c_{mod}(t,λ, φ))-\max(c_{ref}(t,λ, φ))

(ii) \ \textrm{not applicable, as units are consistent across all variables}

(iii) \ s_{phase}(λ, φ)=\frac{1}{2}[1+\cos(\frac{2πθ(λ, φ)}{365})]

(iv) \ S_{phase}=\overline{\overline{s_{phase}}}

The equations for computing the inter-annual variability score (S_{iav}) are:

(i) \ iav_{ref}(λ, φ)=√{\frac{1}{t_{f}-t_{0}}\int_{t_{0}}^{t_{f}}(v_{ref}(t,λ, φ)-c_{ref}(t,λ, φ))^{2}dt}

(i) \ iav_{mod}(λ, φ)=√{\frac{1}{t_{f}-t_{0}}\int_{t_{0}}^{t_{f}}(v_{mod}(t,λ, φ)-c_{mod}(t,λ, φ))^{2}dt}

(ii) \ \varepsilon_{iav}=|(iav_{mod}(λ, φ)-iav_{ref}(λ, φ))|/iav_{ref}(λ, φ)

(iii) \ s_{iav}(λ, φ)=e^{-\varepsilon_{iav}(λ, φ)}

(iv) \ S_{iav}=\overline{\overline{s_{iav}}}

The equations for computing the spatial distribution score (S_{dist}) are:

(i) \ σ=σ_{\overline{v_{mod}}}/σ_{\overline{v_{ref}}}

(ii) and (iii) \ \textrm{not applicable}

(iv) \ S_{dist}=2(1+R)/(σ+\frac{1}{σ})^{2}

where \overline{v_{mod}}(λ, φ) and \overline{v_{ref}}(λ, φ) are the mean values in time t of a variable v as a function of longitude λ and latitude φ for model and reference data, respectively, t_0 and t_f are the initial and final time step, σ_{\overline{v_{mod}}} and σ_{\overline{v_{ref}}} are the standard deviation of the time mean values from the model and reference data, and R is the spatial correlation coefficient of \overline{v_{ref}}(λ, φ) and \overline{v_{mod}}(λ, φ).

Score values are then combined to derive a single overall score for each output variable:

(v) \ S_{overall}=\frac{S_{bias}+2S_{rmse}+S_{phase}+S_{iav}+S_{dist}}{1+2+1+1+1}.

Note that S_{rmse} is weighted by a factor of two, which emphasizes its importance.

Collier, Nathan, Forrest M. Hoffman, David M. Lawrence, Gretchen Keppel-Aleks, Charles D. Koven, William J. Riley, Mingquan Mu, and James T. Randerson. 2018. “The International Land Model Benchmarking (ILAMB) System: Design, Theory, and Implementation.” Journal of Advances in Modeling Earth Systems 10 (11): 2731–54.

Melton, Joe R., Vivek K. Arora, Eduard Wisernig-Cojoc, Christian Seiler, Matthew Fortier, Ed Chan, and Lina Teckentrup. 2020. “CLASSIC v1.0: The Open-Source Community Successor to the Canadian Land Surface Scheme (CLASS) and the Canadian Terrestrial Ecosystem Model (CTEM) - Part 1: Model Framework and Site-Level Performance.” https://doi.org/10.5194/gmd-13-2825-2020.

amber documentation built on Aug. 28, 2020, 5:08 p.m.