2. Choosing and diagnosing spatial folds

Introduction

This tutorial shows how to check whether a cross-validation design is appropriate after the folds have been created. It covers cv_summary, cv_similarity, cv_distance, cv_spatial_autocor, and cv_block_size.

The diagnostics answer different questions:

Please cite blockCV by: Valavi R, Elith J, Lahoz-Monfort JJ, Guillera-Arroita G. blockCV: An R package for generating spatially or environmentally separated folds for k-fold cross-validation of species distribution models. Methods Ecol Evol. 2019; 10:225–232. doi: 10.1111/2041-210X.13107

options(scipen = 10)

Prepare data and folds

Load the example presence-absence records and environmental covariates included with blockCV.

library(blockCV)
library(sf)
library(terra)

points <- read.csv(system.file("extdata/", "species.csv", package = "blockCV"))
pa_data <- sf::st_as_sf(points, coords = c("x", "y"), crs = 7845)

rasters <- terra::rast(
  list.files(system.file("extdata/au/", package = "blockCV"), full.names = TRUE)
)

Create several fold designs to compare. See Tutorial 1 for a fuller explanation of the fold-creation functions.

sb <- cv_spatial(x = pa_data,
                 column = "occ",
                 r = rasters,
                 k = 5,
                 size = 350000,
                 selection = "random",
                 iteration = 50,
                 progress = FALSE,
                 report = FALSE,
                 plot = FALSE)

set.seed(6)
bscv <- cv_cluster(x = pa_data,
                   column = "occ",
                   k = 5,
                   balance = TRUE,
                   k_multiplier = 3,
                   report = FALSE)

set.seed(6)
ecv <- cv_cluster(x = pa_data,
                  column = "occ",
                  r = rasters,
                  k = 5,
                  scale = TRUE)

bloo <- cv_buffer(x = pa_data,
                  column = "occ",
                  size = 350000,
                  progress = FALSE,
                  report = FALSE)

knn_blocks <- cv_knndm(x = pa_data,
                       column = "occ",
                       r = rasters,
                       k = 5,
                       clustering = "blocks",
                       keep_blocks = TRUE,
                       num_sample = 3000,
                       nk_len = 40,
                       seed = 6,
                       report = FALSE,
                       plot = FALSE)

Fold-quality summary

The cv_summary function is the quickest way to inspect a fold design. Without a prediction domain it reports the structural summary from the cv object: number of folds, leave-one-out status, train/test counts, and fold warnings.

summary_basic <- cv_summary(sb)
summary_basic

When r, pred_points, or model_domain is supplied, cv_summary also adds the distance diagnostic. When r is supplied, it also adds the environmental novelty diagnostic. These are the same data-oriented computations used by cv_distance and cv_similarity.

summary_full <- cv_summary(cv = sb,
                           x = pa_data,
                           r = rasters,
                           num_sample = 3000,
                           seed = 6,
                           progress = FALSE)

summary_full$distances
summary_full$novelty
summary_full$warnings

The warnings are returned as data, not as R warning conditions, so they can be filtered or saved in an analysis pipeline. For k-fold designs they can flag empty or tiny test folds, single-class test folds, classes missing from training, severe imbalance, and high leakage. For leave-one-out designs such as cv_buffer and cv_nndm, structural fold-size warnings are skipped because each fold holds a single test point by design.

Environmental similarity diagnostics

The cv_similarity function evaluates environmental similarity between training and testing folds. It supports method = "MESS", method = "L1", and method = "L2". Negative values indicate environmental novelty: at least one testing point is less similar to its fold's training data than the selected method's reference.

The function returns a cv_similarity object. The plot is stored in $plot, the per-fold table in $extrapolation, and the overall novelty rate in $overall. Set plot = FALSE when you want to save or customise the plot object instead of drawing it immediately.

sim <- cv_similarity(cv = ecv,
                     x = pa_data,
                     r = rasters,
                     method = "MESS",
                     plot = FALSE,
                     progress = FALSE)

sim$plot
sim$extrapolation
sim$overall

Setting type = "map" shows where extrapolation occurs in geographical space. Testing points are coloured by their similarity value (here MESS), so novel points are easy to locate.

cv_similarity(cv = ecv,
              x = pa_data,
              r = rasters,
              method = "MESS",
              type = "map",
              plot = TRUE,
              progress = FALSE)

The distance-based L1 and L2 scores use a random raster sample as their baseline, so set seed when reproducibility matters. Use num_plots to show a subset of folds.

cv_similarity(cv = ecv,
              x = pa_data,
              r = rasters,
              method = "L2",
              num_plots = 1:3,
              num_sample = 3000,
              seed = 6,
              plot = TRUE,
              progress = FALSE)

Nearest-neighbour distance diagnostics

The cv_distance function compares the nearest-neighbour distance distribution of a fold design with the distance distribution expected across the prediction domain. It returns a cv_distance object with the plot in $plot, per-fold distance summaries in $distances, and Wasserstein-1 distances in $W. Lower Wasserstein-1 values indicate a closer match to the prediction domain.

distance_theme <- ggplot2::theme(
  legend.position = "bottom",
  plot.subtitle = ggplot2::element_text(size = 7)
)

dist_sb <- cv_distance(cv = sb,
                       x = pa_data,
                       r = rasters,
                       num_sample = 3000,
                       num_random = 5,
                       seed = 6,
                       plot = FALSE)

dist_bscv <- cv_distance(cv = bscv,
                         x = pa_data,
                         r = rasters,
                         num_sample = 3000,
                         num_random = 5,
                         seed = 6,
                         plot = FALSE)

dist_bloo <- cv_distance(cv = bloo,
                         x = pa_data,
                         r = rasters,
                         add_random = FALSE,
                         num_sample = 3000,
                         seed = 6,
                         plot = FALSE)

dist_knn <- cv_distance(cv = knn_blocks,
                        x = pa_data,
                        r = rasters,
                        num_sample = 3000,
                        num_random = 5,
                        seed = 6,
                        plot = FALSE)

cowplot::plot_grid(
  dist_sb$plot +
    ggplot2::labs(title = "Spatial blocks") +
    distance_theme,
  dist_bscv$plot +
    ggplot2::labs(title = "Spatial clustering (balanced)") +
    distance_theme,
  dist_bloo$plot +
    ggplot2::labs(title = "Buffering LOO") +
    distance_theme,
  dist_knn$plot +
    ggplot2::labs(title = "kNNDM blocks") +
    distance_theme,
  ncol = 2
)

The W vector summarises the area between each curve and the prediction curve. The distances table shows the spread of test-to-nearest-train distances within each fold. The pct_below_pred column reports the percentage of test points that are closer to training data than the median prediction distance; high values indicate folds that may be easier than the prediction task.

dist_sb$W
dist_sb$distances

Distances are geographical by default. Set space = "feature" to compute distances in the covariate space of r, optionally scaled with scale = TRUE.

dist_feature <- cv_distance(cv = sb,
                            x = pa_data,
                            r = rasters,
                            space = "feature",
                            num_sample = 3000,
                            seed = 6,
                            plot = FALSE)

dist_feature$plot

Choosing a candidate block size

To support a first choice of block size before model fitting, cv_spatial_autocor estimates the effective range of spatial autocorrelation in the response or continuous predictor rasters. This tool is a guide, not an absolute rule.

Roberts et al. (2017) recommend choosing spatial blocks with reference to the autocorrelation range of model residuals. The response and predictor variograms shown here are useful exploratory guides before a model has been fitted, but they are not residual autocorrelation ranges and can under- or over-estimate the block size needed for residual-based guidance. To estimate residual autocorrelation with cv_spatial_autocor, fit the model first, add the residuals as a column in the sample points object, and pass that column name to column.

When only r is supplied, the function fits variograms to the continuous raster covariates and reports the median range.

sac_raster <- cv_spatial_autocor(r = rasters,
                                 num_sample = 3000,
                                 progress = FALSE,
                                 plot = FALSE)

plot(sac_raster)
sac_raster$range
summary(sac_raster)

Alternatively, supply the sample points and a response, count, continuous, or residual column.

sac_points <- cv_spatial_autocor(x = pa_data,
                                 column = "occ",
                                 plot = FALSE)

plot(sac_points)

cv_block_size provides an interactive Shiny app for manually exploring spatial block sizes. The app cannot be shown inside the vignette because it requires an interactive R session.

cv_block_size(x = pa_data,
              column = "occ",
              r = rasters,
              min_size = 2e5,
              max_size = 9e5)

References:



Try the blockCV package in your browser

Any scripts or data that you put into this service are public.

blockCV documentation built on Aug. 20, 2026, 5:10 p.m.