crossval: Cross-validation of palaeoecological transfer function models
In analogue: Analogue and Weighted Averaging Methods for Palaeoecology

Description Usage Arguments Value Author(s) See Also Examples

Performs leave-one-out, k-fold, n k-fold and bootstrap cross-validation of palaeoecological transfer function models.

crossval(obj, ...)

## S3 method for class 'wa'
crossval(obj, method = c("LOO","kfold","bootstrap"),
         nboot = 100, nfold = 10, folds = 5,
         verbose = getOption("verbose"), ...)

## S3 method for class 'pcr'
crossval(obj, method = c("LOO","kfold","bootstrap"),
         ncomp, nboot = 100, nfold = 10, folds = 5,
         verbose = getOption("verbose"), ...)

`obj`	A fitted transfer function model. Currently, only objects of class `wa` and `pcr` are supported.
`method`	character; type of cross-validation.
`ncomp`	numeric; number of components to fit, as in models with `1:ncomp` components.
`nboot`	numeric; number of bootstrap samples.
`nfold`	numeric; number of chunks into which the training data are split. The k in k-fold.
`folds`	numeric; the number of times k-fold CV is performed.
`verbose`	logical; should progress of the CV be displayed?
`...`	Arguments passed to other methods.

Returns an object of class "crossval", a list with the following components:

`fitted.values`	numeric vector; the cross-validated estimates of the response.
`residuals`	numeric vector; residuals computed from the cross-validated estimates of the response.
`performance`	data frame; cross-validation performance statistics for the model.
`CVparams`	list; parameters holding details of the cross-validation process.
`call`	the matched call.

Gavin L. Simpson

wa

## Load the Imbrie & Kipp data and
## summer sea-surface temperatures
data(ImbrieKipp)
data(SumSST)
     
## fit the WA model
mod <- wa(SumSST ~., data = ImbrieKipp)
mod

## Leave one out CV
cv.loo <- crossval(mod)
cv.loo

## k-fold CV (k == 10)
cv.kfold <- crossval(mod, method = "kfold", kfold = 10, folds = 1)
cv.kfold

## n k-fold CV (k == 10, n = 10)
cv.nkfold <- crossval(mod, method = "kfold", kfold = 10, folds = 10)
cv.nkfold

## bootstrap with 100 bootstrap samples
cv.boot <- crossval(mod, method = "bootstrap", nboot = 100)
cv.boot

## extract fitted values and residuals
fitted(cv.boot)
resid(cv.boot)

## Principal Components Regression
mpcr <- pcr(SumSST ~., data = ImbrieKipp, ncomp = 10)
crossval(mpcr, method = "kfold", kfold = 10, folds = 2, ncomp = 10)

crossval(mpcr, method = "bootstrap", nboot = 100, ncomp = 10)

Loading required package: vegan
Loading required package: permute
Loading required package: lattice
This is vegan 2.4-4
analogue version 0.17-0

	Weighted Averaging Transfer Function

Call:
wa(formula = SumSST ~ ., data = ImbrieKipp) 

Deshrinking  : Inverse 
Tolerance DW : No 
No. samples  : 61 
No. species  : 27 

Performance:
     RMSE  R-squared  Avg. Bias  Max. Bias  
   2.0188     0.9173     0.0000    -3.8155  

	Model Cross-validation:

crossval(obj = mod)

Method: LOO

       R2   avgBias maxBias  RMSEP RMSEP2 s1 s2
1 0.90028 -0.013652 -4.5985 2.2179     NA NA NA
	Model Cross-validation:

crossval(obj = mod, method = "kfold", folds = 1, kfold = 10)

Method: kfold
k: 10
No. of folds: 1

       R2   avgBias maxBias  RMSEP RMSEP2 s1 s2
1 0.90322 -0.018626 -4.4282 2.1847     NA NA NA
	Model Cross-validation:

crossval(obj = mod, method = "kfold", folds = 10, kfold = 10)

Method: kfold
k: 10
No. of folds: 10

       R2   avgBias maxBias  RMSEP RMSEP2     s1     s2
1 0.90092 -0.010127 -4.5215 2.2109 2.2213 0.2145 2.2109
	Model Cross-validation:

crossval(obj = mod, method = "bootstrap", nboot = 100)

Method: bootstrap
No. Bootstraps: 100

       R2   avgBias maxBias  RMSEP RMSEP2      s1     s2
1 0.90099 -0.044511 -4.5609 2.2099 2.2769 0.54829 2.2099
 [1]  4.116456  3.870780  4.018940  4.152943  8.404266  9.242525  3.094267
 [8] 14.560914 14.798103 17.027962 16.031453 19.111681 18.664651 18.800838
[15] 17.557194 20.608958 20.012712 19.838856 18.755710 22.918771 22.506189
[22] 20.805816 22.482609 22.154880 21.545310 23.347238 23.344083 22.825157
[29] 24.286114 25.689882 25.533797 23.253123 23.600082 22.923795 24.542569
[36] 25.378440 25.792991 26.388380 24.034490 25.455282 26.373776 25.838319
[43] 26.766042 26.451282 26.150157 25.651790 25.847750 26.322030 26.863535
[50] 26.731681 26.878528 25.967405 26.848698 26.720265 26.451047 26.054483
[57] 26.571672 27.198487 26.773283 26.898516 26.778190
 [1] -2.11645573  1.12921991  1.48105967  2.84705663 -1.40426573  1.25747478
 [7]  7.90573286 -4.56091372 -1.79810344 -5.02796198 -2.03145270 -4.61168084
[13] -3.66465112 -4.30083788 -1.55719362 -2.60895838 -0.01271249 -1.83885611
[19]  0.24428962 -4.41877096 -1.00618882  0.19418428 -1.48260892  1.84512028
[25]  2.45468980 -0.34723773  0.65591688  0.17484324 -1.28611351 -1.68988196
[31] -0.53379652  2.74687667  2.39991832  3.07620527  0.45743058  0.62155986
[37]  0.20700935 -1.88838030  2.96550995  0.74471790 -1.37377587  0.66168146
[43] -0.56604247 -0.45128197 -0.15015704  1.34820970  1.15225011  1.17797041
[49]  0.13646535  0.26831906  0.12147188  1.03259454  0.15130224  2.27973519
[55]  2.04895298  1.44551700  0.92832777  0.30151307  0.22671687  0.10148400
[61]  1.22181009
	Model Cross-validation:

crossval(obj = mpcr, method = "kfold", ncomp = 10, folds = 2, 
    kfold = 10)

Method: kfold
k: 10
No. of folds: 2

   comp      R2    avgBias maxBias  RMSEP RMSEP2      s1     s2
1     1 0.93106 -0.0197524 -6.6385 2.5624 2.5876 0.36036 2.5624
2     2 0.94973 -0.0324673 -5.5408 2.1979 2.2024 0.14050 2.1979
3     3 0.95298 -0.0635650 -4.5457 2.1282 2.1369 0.19284 2.1282
4     4 0.95361 -0.0605307 -4.5273 2.1143 2.1242 0.20557 2.1143
5     5 0.95547 -0.0343966 -5.1337 2.0719 2.0860 0.24199 2.0719
6     6 0.95522 -0.0164388 -4.7891 2.0786 2.0944 0.25673 2.0786
7     7 0.95561 -0.0095796 -4.6085 2.0698 2.0902 0.29124 2.0698
8     8 0.95840  0.0227161 -4.9490 2.0071 2.0202 0.22963 2.0071
9     9 0.95806 -0.0261129 -5.0331 2.0140 2.0349 0.29086 2.0140
10   10 0.95784  0.0106131 -5.1693 2.0177 2.0505 0.36569 2.0177
	Model Cross-validation:

crossval(obj = mpcr, method = "bootstrap", ncomp = 10, nboot = 100)

Method: bootstrap
No. Bootstraps: 100

   comp      R2   avgBias maxBias  RMSEP RMSEP2      s1     s2
1     1 0.94400 -0.061900 -6.4432 2.3508 2.6385 1.19829 2.3508
2     2 0.95168 -0.012417 -5.3873 2.1567 2.2068 0.46796 2.1567
3     3 0.95493 -0.020736 -4.5503 2.0840 2.1677 0.59640 2.0840
4     4 0.95521 -0.034405 -4.2820 2.0782 2.1771 0.64883 2.0782
5     5 0.95697 -0.029393 -5.0530 2.0384 2.1677 0.73758 2.0384
6     6 0.95769 -0.030147 -5.0560 2.0210 2.1605 0.76369 2.0210
7     7 0.95887 -0.016955 -4.8758 1.9932 2.1334 0.76066 1.9932
8     8 0.96021 -0.018091 -4.8335 1.9615 2.1150 0.79111 1.9615
9     9 0.96057 -0.016072 -4.8733 1.9525 2.1226 0.83280 1.9525
10   10 0.96157 -0.027477 -4.9112 1.9277 2.1109 0.86010 1.9277