sdm_eval: Calculate different model performance metrics

In sjevelazco/flexsdm: Tools for Data Preparation, Fitting, Prediction, Evaluation, and Post-Processing of Species Distribution Models

Calculate different model performance metrics

Description

This function calculates threshold dependent and independent model performance metrics.

Usage

sdm_eval(p, a, bg = NULL, thr = NULL)

Arguments

p	numeric. Predicted suitability for presences
a	numeric. Predicted suitability for absences
bg	numeric. Predicted suitability for background points, used for BOYCE metric. It bg is set as NULL, BOYCE metric will be calculated with presences and absences suitabilities values
thr	character. Threshold criterion used to get binary suitability values (i.e. 0,1). Used for threshold-dependent performance metrics. It is possible to use more than one threshold type. A vector must be provided for this argument. The following threshold criteria are available: lpt: The highest threshold at which there is no omission. equal_sens_spec: Threshold at which the Sensitivity and Specificity are equal. max_sens_spec: Threshold at which the sum of the Sensitivity and Specificity is the highest (aka threshold that maximizes the TSS). max_jaccard: The threshold at which the Jaccard index is the highest. max_sorensen: The threshold at which the Sorensen index is the highest. max_fpb: The threshold at which FPB (F-measure on presence-background data) is the highest. sensitivity: Threshold based on a specified Sensitivity value. Usage thr = c('sensitivity', sens='0.6') or thr = c('sensitivity'). 'sens' refers to Sensitivity value. If a sensitivity value is not specified, the default value is 0.9 If more than one threshold type is used, concatenate threshold types, e.g., thr=c('lpt', 'max_sens_spec', 'max_jaccard'), or thr=c('lpt', 'max_sens_spec', 'sensitivity', sens='0.8'), or thr=c('lpt', 'max_sens_spec', 'sensitivity'). Function will use all thresholds if no threshold type is specified

Details

This function is used for evaluating different models approaches base on the combination of presence-absences or presence-pseudo-absences and background point data and suitability predicted by any model or flexsdm modeling function families (fit_, esm_, and tune_.)

It calculates the next performance metric:

Performance metric	Threshold dependent	Values ranges
TPR (True Positive Rate, also called Sensitivity)	yes	0 - 1
TNR (True Negative Rate, also called Specificity)	yes	0 - 1
SORENSEN	yes	0 - 1
JACCARD	yes	0 - 1
FPB (F-measure on presence-background)	yes	0 - 2
OR (Omission Rate)	yes	0 - 1
TSS (True Skill Statistic)	yes	-1 - 1
KAPPA	yes	0 - 1
AUC (Area Under Curve)	no	0 - 1
BOYCE (continuous Boyce index)*	no	-1 - 1
IMAE (Inverse Mean Absolute Error)**	no	0 - 1

\* BOYCE is calculated based on presences and background points, in case that background points is not provided it is calculated using presences and absences. The codes for calculating this metric is and adaptation of enmSdm package (https://github.com/adamlilith/enmSdm)

\** IMAE is calculated as 1-(Mean Absolute Error) in order to be consistent with the other metrics where the higher the value of a given performance metric, the greater the model's accuracy

Value

a tibble with next columns

• threshold: threshold names

• thr_value: threshold values

• n_presences: number of presences

• n_absences: number of absences

• from TPR to IMAE: performance metrics

Examples

## Not run: 
require(dplyr)

set.seed(0)
p <- rnorm(50, mean = 0.7, sd = 0.3) %>% abs()
p[p > 1] <- 1
p[p < 0] <- 0

set.seed(0)
a <- rnorm(50, mean = 0.3, sd = 0.2) %>% abs()
a[a > 1] <- 1
a[a < 0] <- 0

set.seed(0)
backg <- rnorm(1000, mean = 0.4, sd = 0.4) %>% abs()
backg[backg > 1] <- 1
backg[backg < 0] <- 0

# Function use without threshold specification
e <- sdm_eval(p, a)
e

# Function use with threshold specification
sdm_eval(p, a, thr = "max_sorensen")
sdm_eval(p, a, thr = c("lpt", "max_sens_spec", "max_jaccard"))
sdm_eval(p, a, thr = c("lpt", "max_sens_spec", "sensitivity"))
sdm_eval(p, a, thr = c("lpt", "max_sens_spec", "sensitivity", sens = "0.95"))

# Use of bg argument (it will only be used for calculating BOYCE index)
sdm_eval(p, a, thr = "max_sens_spec")
sdm_eval(p, a, thr = c("max_sens_spec"), bg = backg)

# If background will be used to calculate all other metrics
# background values can be used in "a" argument
sdm_eval(p, backg, thr = "max_sens_spec")

## End(Not run)

sjevelazco/flexsdm documentation built on Feb. 28, 2025, 9:07 a.m.