Managing custom formats

  collapse = TRUE,
  comment = "#>",
  eval = rlang::is_installed("arrow")

The pins package provides a robust set of functions to read and write standard types of files using standard tools, e.g. CSV files using read.csv() and write.csv(). However, from time to time, you may wish read or write using other tools. You may want to read and write:

An escape hatch for a customized approach is provided: pin_upload() and pin_download(). The goal of this vignette is to show how you can incorporate these into your workflow.

Two points to keep in mind:

We'll follow an example where we write and read uncompressed Arrow files, starting by creating a temporary board:


board <- board_temp()

Upload a one-off file

If you are writing a one-off file, you can do everything directly:

pin_name <- "mtcars-arrow"

# file name will be `mtcars-arrow.arrow`
path <- fs::path_temp(fs::path_ext_set(pin_name, "arrow"))

arrow::write_feather(mtcars, path, compression = "uncompressed")

pin_upload(board, paths = path, name = pin_name)

Reading from the downloaded pin is straightforward; pin_download() returns a local path that can be piped to arrow::read_feather():

mtcars_download <- 
  pin_download(board, pin_name) %>%


Function to manage uploading

If you want to write more than one custom file of a certain type, or using a certain tool, you might consider writing a helper function:

pin_upload_arrow <- function(board, x, name, ...) {
  # path deleted when `pin_upload_arrow()` exits
  path <- fs::path_temp(fs::path_ext_set(name, "arrow"))

  # custom writer
  arrow::write_feather(x, path, compression = "uncompressed")

  pin_upload(board, paths = path, name = name, ...) 

This helper function is designed to work like pin_write():

pin_upload_arrow(board, x = mtcars, name = "mtcars-arrow2")

As before, you can pipe the result of pin_download() to your reader function:

pin_download(board, name = "mtcars-arrow2") %>%
  arrow::read_feather() %>%

Another example function

If you want to use this same approach to archive and pin a whole directory, you can write a helper function like:

pin_upload_archive <- function(board, dir, name, ...) {
  path <- fs::path_temp(fs::path_ext_set(name, "tar.gz"))
  archive::archive_write_dir(path, dir)
  pin_upload(b, paths = path, name = name, ...)

You can download the compressed archive via pin_download(board, name) and then pipe that path straight to archive::archive_extract() to extract your archive in a new diretory.

Try the pins package in your browser

Any scripts or data that you put into this service are public.

pins documentation built on Jan. 22, 2023, 1:55 a.m.