int_conformal_cv: Prediction intervals via conformal inference CV+

In probably: Tools for Post-Processing Class Probability Estimates

Prediction intervals via conformal inference CV+

Description

Nonparametric prediction intervals can be computed for fitted regression workflow objects using the CV+ conformal inference method described by Barber at al (2018).

Usage

int_conformal_cv(object, ...)

## Default S3 method:
int_conformal_cv(object, ...)

## S3 method for class 'resample_results'
int_conformal_cv(object, ...)

## S3 method for class 'tune_results'
int_conformal_cv(object, parameters, ...)

Arguments

object	An object from a tidymodels resampling or tuning function such as tune::fit_resamples(), tune::tune_grid(), or similar. The object should have been produced in a way that the .extracts column contains the fitted workflow for each resample (see the Details below).
...	Not currently used.
parameters	An tibble of tuning parameter values that can be used to filter the predicted values before processing. This tibble should select a single set of hyper-parameter values from the tuning results. This is only required when a tuning object is passed to object.

Details

This function implements the CV+ method found in Section 3 of Barber at al (2018). It uses the resampled model fits and their associated holdout residuals to make prediction intervals for regression models.

This function prepares the objects for the computations. The predict() method computes the intervals for new data.

This method was developed for V-fold cross-validation (no repeats). Interval coverage is unknown for any other resampling methods. The function will not stop the computations for other types of resamples, but we have no way of knowing whether the results are appropriate.

Value

An object of class "int_conformal_cv" containing the information to create intervals. The predict() method is used to produce the intervals.

References

Rina Foygel Barber, Emmanuel J. Candès, Aaditya Ramdas, Ryan J. Tibshirani "Predictive inference with the jackknife+," The Annals of Statistics, 49(1), 486-507, 2021

See Also

predict.int_conformal_cv()

Examples

library(workflows)
library(dplyr)
library(parsnip)
library(rsample)
library(tune)
library(modeldata)

set.seed(2)
sim_train <- sim_regression(200)
sim_new   <- sim_regression(  5) %>% select(-outcome)

sim_rs <- vfold_cv(sim_train)

# We'll use a neural network model
mlp_spec <-
  mlp(hidden_units = 5, penalty = 0.01) %>%
  set_mode("regression")

# Use a control function that saves the predictions as well as the models.
# Consider using the butcher package in the extracts function to have smaller
# object sizes

ctrl <- control_resamples(save_pred = TRUE, extract = I)

set.seed(3)
nnet_res <-
  mlp_spec %>%
  fit_resamples(outcome ~ ., resamples = sim_rs, control = ctrl)

nnet_int_obj <- int_conformal_cv(nnet_res)
nnet_int_obj

predict(nnet_int_obj, sim_new)

probably documentation built on July 10, 2023, 2:03 a.m.