spectralBacktest: spectralBacktest: Package for spectral backtesting of...
In ajmcneil/spectralBacktest: Spectral Backtests of Forecast Distributions

spectralBacktest

R Documentation

spectralBacktest: Package for spectral backtesting of forecast distributions

Description

The spectralBacktest package implements spectral backtests of Z-test form as defined in Gordy and McNeil (2018). This package accommodates both discrete and continuous kernels, and tests of both unconditional and conditional coverage.

Monokernel, bikernel and multikernel objects

A monokernel is a list with the following named elements:

name is an optional name.
type is string value 'mono'.
nu is a closure taking parameters support, param, and standardize, and returning a function mapping PIT values to transformed PIT values, i.e., W=ν(P). When standardize is TRUE, W will be standardized to mean zero and variance one under the null.
support is the support of the kernel. For continuous kernels, support is a vector (α_1,α_2). For discrete kernels, support is the vector of points receiving positive weight.
param is an optional list of parameters to the kernel, e.g., ζ for the exponential kernel. For discrete kernels, param is the vector of weights on the points in support.

Here is an example of a continuous monokernel:
     ZV <- list( name = 'Epanechnikov',
                 type = 'mono',
                 nu = nu_epanechnikov,
                 support = c(alpha1, alpha2),
                 param = NULL )
and an example of a discrete monokernel:
    ZU3 <- list( name = 'Discrete Uniform 3',
                 type = 'mono',
                 nu = nu_discrete,
                 support = c(alpha1, 0.99, alpha2),
                 param = c(1, 1, 1) )
The binomial score test is represented as:
    B99 <- list( name = 'Binomial score at 99pct',
                 type = 'mono',
                 nu = nu_discrete,
                 support = 0.99,
                 param = 1 )

When writing your own monokernels (say, 'nu_custom'), it is a good idea to define a separate function (say, 'mu_custom') to provide the first and second moments of the unstandardized kernel. This can be useful when combining kernel functions into bikernel or multikernel objects.

A bikernel is a list with the following named elements:

name is an optional name.
type is string value 'bi'.
nu is a list of closures, each as described in the monokernel case.
correlation is a function taking parameters support and param, and returning the correlation between standardized transformed PIT values W_1 and W_2.
support is the common support of the kernels, as in the monokernel case.
param is a list of optional parameter lists. Element j of list is passed to to the kernel in position j in nu. The full list is passed to the correlation function.

Here is an example of a continuous bikernel:
    ZAE <- list( name = 'Arcsin/Epanechnikov',
                 type = 'bi',
                 nu = list(nu_arcsin, nu_epanechnikov),
                 correlation = rho_arcsin_epanechnikov,
                 support = c(alpha1, alpha2),
                 param = list(NULL, NULL) )

For tractability, in the higher-order multikernel tests we assume that the component kernels are all in the same family. The simplifies the representation of a multikernel as a list with the following named elements:

name is a (string) name for the kernel.
type is string value 'multi'.
nu is a closure as in the monokernel case.
correlation is a bivariate correlation function as in the bikernel case.
support is the common support of the kernels as in the monokernel case.
param is a list of optional parameter lists. Element j of list is passed to to the kernel in position j in nu. Pairs of elements are passed to the correlation function.

The Pearson multinomial test is expressed as a multikernel. The component kernels do not share a common support, but rather each has its own "level," and therefore the levels are passed as a param list rather than in the support field:

    Pearson3 <- list( name = 'Pearson',
                      type = 'multi',
                      nu = nu_pearson,
                      correlation = rho_pearson,
                      support = NULL,
                      param = list(0.985, 0.99, 0.995) )

Tests of unconditional coverage

Calls to the unconditional tests take a generic form:

  p_value <- spectral_Ztest(kernel, PIT, twosided)

The type field in kernel redirects to specific handlers for mono/bi/multi. For the bi- and multi-spectral cases, the twosided parameter is unnecessary and would be ignored if supplied.

Martingale difference tests

The conditional tests require an additional list structure to specify the h function which is to be applied to the lagged PIT values. We refer to this structure as a CVT (conditioning variable transformation). This structure takes slightly different forms in the monospectral and bispectral cases.

A monospectral CVT is a list with the following elements:

name is a (string) name for the CVT.
type is string value 'mono'.
h_closure is a closure taking parameter param, and returning a function h mapping P values to h(P). Without loss of generality, it is required that the resulting function be standardized such that h(P) has mean zero and variance one when P is distributed Uniform.
h_param is a list of optional parameters to h_closure.
lags (integer) is the number of lagged PIT to include in h.

Here is an example based on the function h(P) = |2P-1|^θ for θ>0

    CVT <- list( name = 'Power 4',
                 type = 'mono',
                 h_closure = CVT_PtildePower,
                 h_param = 4,
                 lags = 4L )

The closure CVT_PtildePower is included in the package.

A bispectral CVT is a list with the following elements:

name is a (string) name for the CVT.
type is string value 'bi'.
h_closure is a list of two closures, each as in the monokernel case.
h_param is a list of optional parameter lists. Element j of list is passed to to h_closure[[j]].
lags (vector of integers) is the number of lagged PIT to include in each h.
correlation is a scalar function mapping h_param to the correlation of h_closure[[1]] and h_closure[[2]] under the null.

Here is an example with 4 lags and 2 lags applied to ν_1 and ν_2, respectively:

    CVT2 <- list( name = 'Power 4/Power 0.5',
                  type = 'bi',
                  h_closure = list(CVT_PtildePower, CVT_PtildePower),
                  correlation = rho_PtildePower,
                  h_param = list(4,1/2),
                  lags = c(4L,2L) )

In principle, we can extend the MD tests to the multispectral case, but leave this for a future version of the package.

The MD test is called as follows:

  p_value <- spectral_MDtest(kernel, CVT, PIT, h_list=NULL, estimateH=TRUE, contingencyH=FALSE)

where

kernel: is a mono- or bi-kernel object as used in the unconditional tests.
CVT: is a mono- or bi-spectral CVT object.
PIT: is a vector of PIT values.
h_list: (optional) is a precalculated matrix (in the monospectral case) or list of matrices (in the bispectral case).
estimateH: (boolean). If TRUE, then estimated H is preferred whenever non-singular. If FALSE, use the theoretical H, which is the identity matrix in the monospectral case.
contingencyH: (boolean). If TRUE, then use theoretical H when the preferred H is singular. If FALSE, then return NA when preferred H singular.

The parameter h_list needs elaboration. If this input is not provided by the user, the matrix (or list of matrices) h are computed simply by applying the output of h_closure to the PIT values. In the context of simulation studies, however, it is often much more efficient to precalculate the h matrix for a set of tests. Allowing for h_list as a specified input also permits flexibility in the situation in which inputed PIT values are substituted for missing PIT values in generating h.