In chhetrid/rangemapR: A Modular Platform for Reproducible Modeling of Species Niches and Distributions

Component: Model

Many approaches and algorithms exist for building models of species niches/distributions, as well as for evaluating them (Guisan and Thuiller, 2005; Elith et al. 2006; Franklin 2010 chaps. 6, 7, 8). Component Model uses the output from earlier components to build models using either presence-only or presence-background data (and associated environmental information). Wallace currently allows users to build models using either: 1) the presence-only approach BIOCLIM (Module BIOCLIM); or 2) the presence-background algorithm Maxent (Module Maxent).

As noted, various evaluation statistics exist for assessing niche/distributional model performance, and many are based on the ability to predict localities that are held out of the model for evaluation (Peterson et al. 2011). These held-out localities are called "testing" localities, and those used to construct the model are called "training" localities. As part of the Component: Model, Wallace provides a table of a few commonly used evaluation metrics, namely AUC, omission rate, and AIC.

AUC stands for the Area Under the Curve (AUC) of a Receiver Operating Characteristic (ROC) plot. AUC is a non-parametric measure of the ability of a classifier (here, the model) to rank positive records higher than negative ones across the full range of suitability values of the model, and thus judges the model's discriminative ability. AUC ranges from 0 to 1, but major complications for interpreting AUC exist when true negative data (e.g., absences) do not exist (Lobo et al. 2008; Peterson et al. 2008; Peterson et al. 2011 chap. 9). As the niche/distributional models offered in Wallaceare presence-only or presence-background, AUC values should only be considered as relative indicators of performance (e.g., between different settings of the same algorithm, for the same dataset of a given species). Wallace uses ENMeval to both: 1) calculate AUC using the model made with all occurrence localities and "evaluate" it with the same records, i.e. the training localities (full AUC), 2) calculate AUC for each iteration of k-fold cross validation separately, using each model built with the training localities and evaluating based on the testing localities (test AUC), and 3) take the difference between the training and testing AUC for each iteration of k-fold cross validation (AUC diff). Test AUC constitutes the standard way that AUC usually is applied in the field, and is important to consider when performing model projections to other areas/times (Roberts et al. 2017). Higher AUC diff should indicate higher model overfitting, as overfit models should perform better on training than testing data (Warren and Seifert 2011). Five fields in the "Results" table pertain to AUC: + full.AUC: AUC calculated using all occurrence localities + avg.test.AUC and var.test.AUC: mean and variance of the k test AUCs (one for each partition) + avg.diff.AUC and var.diff.AUC: mean and variance of all differences between the k training and test AUCs

The omission rate (OR) is a method for evaluating the ability of a binary classifier (here, the model) to predict test localities, typically after applying a threshold to a continuous or ordinal model prediction (see Module Map Prediction). Application of the threshold makes the prediction binary (e.g., 0s and 1s), and the omission rate then equals the proportion of test localities that fall in grid cells with a 0 (i.e. below the threshold; Peterson et al. 2011). An OR of 0 indicates that no localities fall outside the prediction, whereas an OR of 1 indicates that all of them do. There are many possible thresholding rules, but Wallace offers two that are commonly used: minimum training presence (MTP) and 10 percentile training presence (10pct). MTP is the lowest suitability score for any occurrence localities used to train the model. 10pct is the lowest suitability score for such localities after excluding the lowest 10% of them. Thus, 10pct is more strict than MTP. As with AUC, Wallace calculates OR for each partition; it does so by applying a threshold to the continuous prediction and finding the proportion of testing localities that fall outside the resulting binary prediction. Four fields in the "Results" table pertain to omission rate: + avg.test.orMTP and var.test.orMTP: mean and variance of all test MTP omission rates + avg.test.or10pct and var.test.or10pct: mean and variance of all test 10pct omission rates

AIC, or the Akaike Information Criterion, is a model evaluation metric used often with regression-based models. Models with the lowest AIC are identified as optimal among candidate models. It is calculated in this way:

$AIC = 2k - 2ln(L)$

Here, k = number of model parameters, and ln(L) is the log likelihood of the model (note that here k has a different meaning than used elsewhere in Wallace for k-fold portioning). Models with more parameters will have a greater positive number for the first term, and those with higher likelihoods will have a greater negative number for the second term. Therefore, simpler models with fewer parameters and models with high likelihoods will both receive lower AIC scores. As an example, if two models have approximately the same likelihood, the one with fewer parameters will have a lower AIC. In this way, AIC penalizes complex models, but also rewards complex ones with high likelihoods, with the intent of finding the best balance between complexity and fit (Burnham and Anderson 2002). For Maxent, Wallace calculates AICc (corrected for finite sample sizes) using the methodology outlined in Warren and Seifert (2011); it does not calculate AIC for BIOCLIM. AIC is calculated on the full model only, and thus does not consider partitions. Four fields in the "Results" table pertain to AIC: + nparm: number of parameters in the model (for Maxent, this includes features of variables as well(e.g. four hinges of bio1 means 4 variables, a quadratic term for bio11 means 2 variables, etc.) + AICc + delta.AICc: absolute difference between the lowest AICc and each AICc (e.g. delta.AICc = 0 is the model with the lowest AICc) + w.AIC: the AIC weight, calculated as the average relative model likelihood (exp(-0.5 * delta.AICc)) across all models, and can be used in model averaging (Burnham and Anderson 2002)

There is no single "best" way to evaluate niche/distributional models (especially without absence data), so Wallace seeks to provide a number of them and let users decide which ones fit their research purposes. If run in sequence, the current model results will overwrite the previous ones. We envision that this component will grow substantially in future releases, with the addition of new modules implementing other modeling techniques.

REFERENCES

Burnham, K. P., and D. R. Anderson. (2002). Model selection and multimodel inference : a practical information-theoretic approach. Springer, New York.

Elith J., Graham C.H., Anderson R.P., Dudík M., Ferrier S., Guisan A., Hijmans R.J., Huettmann F., Leathwick J.R., Leahmann A., Li J., Lohmann L.G., Loiselle B.A., Manion G., Moritz C., Nakamura M., Nakazawa Y., Overton J.M., Peterson A.T., Phillips S.J., Richardson K.S., Scachetti-Pereira R., Schapire R.E., Soberón J., Williams S., Wisz M.S., Zimmermann N.E. 2006. Novel methods improve prediction of species' distributions from occurrence data. Ecography. 29: 129-151.

Franklin J. (2010). Mapping Species Distributions: Spatial Inference and Prediction. Statistical models - modern regression; Machine learning methods; Classification, similarity and other methods for presence-only data. In: Mapping species distributions: spatial inference and prediction. Cambridge: Cambridge University Press.

Guisan A., Thuiller W. (2005). Predicting species distribution: offering more than simple habitat models. Ecology Letters. 8: 993-1009.

Lobo, J. M., Jiménez-Valverde, A., & Real, R. (2008). AUC: a misleading measure of the performance of predictive distribution models. Global Ecology and Biogeography. 17: 145-151.

Peterson, A. T., Papeş, M., & Soberón, J. (2008). Rethinking receiver operating characteristic analysis applications in ecological niche modeling. Ecological Modelling. 213: 63-72.

Peterson A. T., Soberón J., Pearson R. G., Anderson R. P., Martinez-Meyer E., Nakamura M., Araújo M. B. (2011). Evaluating Model Performance and Significance. In: Ecological Niches and Geographic Distributions. Princeton, New Jersey: Monographs in Population Biology, 49. Princeton University Press.

Roberts, D. R., Bahn, V., Ciuti, S., Boyce, M. S., Elith, J., Guillera-Arroita, G., Hauenstein, S., Lahoz-Monfort, J. J., Schröder, B., Thuiller, W., Warton, D. I., Wintle, B. A., Hartig, F. and Dormann, C. F. (2017), Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure. Ecography. 40: 913-929.

Warren, D. L., & Seifert, S. N. (2011). Ecological niche modeling in Maxent : the importance of model complexity and the performance of model selection criteria. Ecological Applications. 21: 335-342.

chhetrid/rangemapR documentation built on May 13, 2019, 11:09 a.m.