4. Using blockCV with caret
In blockCV: Spatial and Environmental Blocking for Cross-Validation

Introduction

This tutorial shows how to use spatial cross-validation folds generated by blockCV in the caret modelling framework. The key step is to pass the training and testing row indices stored in folds_list to caret::trainControl() through its index and indexOut arguments.

options(scipen = 10)

Prepare the data

Load the simulated presence-absence data and environmental raster covariates included with blockCV.

library(blockCV)
library(sf)
library(terra)

# import presence-absence species data
points <- read.csv(system.file("extdata/", "species.csv", package = "blockCV"))

# make an sf object from the data.frame
pa_data <- sf::st_as_sf(points, coords = c("x", "y"), crs = 7845)

# load raster covariates
covars <- terra::rast(
  list.files(system.file("extdata/au/", package = "blockCV"), full.names = TRUE)
)

Extract the raster covariate values at the species records. This is the modelling table that will be supplied to caret.

training <- terra::extract(covars, pa_data, ID = FALSE)
training$occ <- as.factor(pa_data$occ)

head(training)

Create spatial folds

Create spatial blocks with cv_spatial(). Each item in folds_list contains two integer vectors: the first is the training row indices and the second is the held-out testing row indices.

set.seed(123)

sb1 <- cv_spatial(
  x = pa_data,
  column = "occ",
  r = covars,
  size = 450000,
  k = 5,
  selection = "random",
  iteration = 50,
  progress = FALSE,
  report = TRUE,
  plot = TRUE
)

Fit a caret model

The only conversion needed for caret is to separate the training and testing indices from folds_list. The following example uses a random forest model through caret::train(). The modelling chunk is not evaluated when the vignette is built because caret and randomForest are optional modelling packages.

library(caret)

train_index <- lapply(sb1$folds_list, function(fold) fold[[1]])
test_index <- lapply(sb1$folds_list, function(fold) fold[[2]])

names(train_index) <- paste0("Fold", seq_along(train_index))
names(test_index) <- names(train_index)

control <- trainControl(
  method = "cv",
  number = length(train_index),
  index = train_index,
  indexOut = test_index,
  search = "random"
)

set.seed(123)

rf_model <- train(
  occ ~ .,
  data = training,
  method = "rf",
  trControl = control,
  tuneLength = 4,
  ntree = 500
)

rf_model
rf_model$resample
plot(rf_model)

The resampling results are now based on the spatial folds from blockCV, rather than on random non-spatial folds generated internally by caret.

The CAST package uses the same caret interface. Its CreateSpacetimeFolds() function creates train and test lists from pre-defined spatial, temporal, or spatio-temporal groups that can be passed directly to trainControl(index = ..., indexOut = ...).

Please cite blockCV by: Valavi R, Elith J, Lahoz-Monfort JJ, Guillera-Arroita G. blockCV: An R package for generating spatially or environmentally separated folds for k-fold cross-validation of species distribution models. Methods Ecol Evol. 2019; 10:225-232. doi: 10.1111/2041-210X.13107

References

Meyer H, Reudenbach C, Hengl T, Katurji M, Nauss T. 2018. Improving performance of spatio-temporal machine learning models using forward feature selection and target-oriented validation. Environmental Modelling & Software 101: 1-9.
Kuhn M. 2008. Building predictive models in R using the caret package. Journal of Statistical Software 28: 1-26.
Valavi R, Elith J, Lahoz-Monfort JJ, Guillera-Arroita G. blockCV: An R package for generating spatially or environmentally separated folds for k-fold cross-validation of species distribution models. Methods Ecol Evol. 2019; 10:225-232. doi: 10.1111/2041-210X.13107