A Basic Usage of 'LazyArray

In lazyarray: Persistent Large Data Array with Lazy-Loading on Demand

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)

library(lazyarray)

Initialize

Create a blank array and assign R object

# Sample data (~24 MB)
x <- rnorm(3e6); dim(x) <- c(10, 100, 100, 30)

# Save array to a path
path <- tempfile()
arr <- create_lazyarray(path, 'double', dim(x), multipart = TRUE)
arr[] <- x

Load existing array

# Load existing array
arr <- load_lazyarray(path)

The loaded array is read-only by default. However, they can be set writable.

# Make loaded array writable
arr$make_writable()
arr$can_write

S3 methods

1. Set dimension names

arr$make_writable()
dimnames(arr) <- list(
  A = 1:10,
  B = 1:100,
  C = 1:100,
  D = 1:30
)

1. Subset and subset assign

# Subset/read array
y1 <- arr[]              
y2 <- arr[,,,3]          

# Write to slice of data, writing to slices along the 
# last dimension is optimized
arr[,,,1] <- seq_len(1e5)

1. Subset by formula

sub <- subset(arr, A ~ A <= 2, B ~ B == 10)
dim(sub)

Remove Arrays

Data created via lazyarray does not remove automatically. You need to finalize array by yourself. This is because multiple lazy array instances might point to a same dataset. If one of the object is garbage collected, you might not want to remove the data on hard drive as this will invalidate the other instances. To manually remove data, use

arr$remove_data()

lazyarray documentation built on July 18, 2020, 9:06 a.m.