In dipterix/lazyarray: Persistent Large Data Array with Lazy-Loading on Demand

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)

library(lazyarray)

Initialize

Create a blank array and assign R object

# Sample data (~24 MB)
x <- rnorm(3e6); dim(x) <- c(10, 100, 100, 30)

# Save array to a path
path <- tempfile()
arr <- lazyarray(path, dim = dim(x), storage_format = 'double')
arr[] <- x

Load existing array

# Load existing array
arr <- lazyarray(path)

To protect array from further changes, make it read-only.

arr$make_readonly()
arr$can_write

To make a read-only array writable:

arr$make_writable()
arr$can_write

S3 methods

1. Set dimension names

arr$make_writable()
dimnames(arr) <- list(
  A = 1:10,
  B = 1:100,
  C = 1:100,
  D = 1:30
)

1. Subset and subset assign

# Subset/read array
y1 <- arr[]              
y2 <- arr[,,,3]          

# Write to slice of data, writing to slices along the 
# last dimension is optimized
arr[,,,1] <- seq_len(1e5)

1. Subset by formula

sub <- subset(arr, A ~ A <= 2, B ~ B == 10)
dim(sub)

Remove Arrays

Data created via lazyarray does not remove automatically. You need to finalize array by yourself. This is because multiple lazy array instances might point to a same dataset. If one of the object is garbage collected, you might not want to remove the data on hard drive as this will invalidate the other instances. To manually remove data, use

arr$remove_data()

dipterix/lazyarray documentation built on June 30, 2023, 6:30 a.m.