Managing custom formats
In pins: Pin, Discover and Share Resources

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  eval = rlang::is_installed("arrow")
)

The pins package provides a robust set of functions to read and write standard types of files using standard tools, e.g. CSV files using read.csv() and write.csv(). However, from time to time, you may wish read or write using other tools. You may want to read and write:

CSV files using readr or vroom
Arrow files without using compression
Parquet files
Whole directories that are archived/zipped

An escape hatch for a customized approach is provided: pin_upload() and pin_download(). The goal of this vignette is to show how you can incorporate these into your workflow.

Two points to keep in mind:

pin_upload() takes a vector of paths to local files.
pin_download() returns a vector of paths to local files.

We'll follow an example where we write and read uncompressed Arrow files, starting by creating a temporary board:

library(pins)

board <- board_temp()

Upload a one-off file

If you are writing a one-off file, you can do everything directly:

pin_name <- "mtcars-arrow"

# file name will be `mtcars-arrow.arrow`
path <- fs::path_temp(fs::path_ext_set(pin_name, "arrow"))

arrow::write_feather(mtcars, path, compression = "uncompressed")

pin_upload(board, paths = path, name = pin_name)

Reading from the downloaded pin is straightforward; pin_download() returns a local path that can be piped to arrow::read_feather():

mtcars_download <- 
  pin_download(board, pin_name) %>%
  arrow::read_feather()

head(mtcars_download)

Function to manage uploading

If you want to write more than one custom file of a certain type, or using a certain tool, you might consider writing a helper function:

pin_upload_arrow <- function(board, x, name, ...) {
  # path deleted when `pin_upload_arrow()` exits
  path <- fs::path_temp(fs::path_ext_set(name, "arrow"))
  withr::defer(fs::file_delete(path))

  # custom writer
  arrow::write_feather(x, path, compression = "uncompressed")

  pin_upload(board, paths = path, name = name, ...) 
}

This helper function is designed to work like pin_write():

pin_upload_arrow(board, x = mtcars, name = "mtcars-arrow2")

As before, you can pipe the result of pin_download() to your reader function:

pin_download(board, name = "mtcars-arrow2") %>%
  arrow::read_feather() %>%
  head()

Another example function

If you want to use this same approach to archive and pin a whole directory, you can write a helper function like:

pin_upload_archive <- function(board, dir, name, ...) {
  path <- fs::path_temp(fs::path_ext_set(name, "tar.gz"))
  withr::defer(fs::file_delete(path))
  archive::archive_write_dir(path, dir)
  pin_upload(b, paths = path, name = name, ...)
}

You can download the compressed archive via pin_download(board, name) and then pipe that path straight to archive::archive_extract() to extract your archive in a new diretory.