load_lazyarray: Load Lazy Array from Given Path

In lazyarray: Persistent Large Data Array with Lazy-Loading on Demand

Description

Load Lazy Array from Given Path

Usage

load_lazyarray(path, read_only = TRUE, meta_name = "lazyarray.meta")

Arguments

path	character, path of the array
read_only	whether setting data is allowed
meta_name	header file name, default is "lazyarray.meta"

Value

A ClassLazyArray instance

Author(s)

Zhengjia Wang

Examples

path <- tempfile()
create_lazyarray(path, 'double', dim = c(3,4,5), multipart = TRUE)

x <- load_lazyarray(path, read_only = FALSE)
x[2,3:4, 2:1] <- 1:4
x[ , , 2]

# Expend dimension for multiple partition data only
dim(x) <- c(3,4,6)
dimnames(x) <- list(dim1 = as.character(1:3),
                    dim2 = letters[1:4], 
                    dim3 = LETTERS[1:6])
x[ , , 'B', drop = FALSE]

# Non-standard subset methods
names(dimnames(x))
subset(x, dim1 ~ dim1 == '2', dim2 ~ dim2 %in% c('a', 'c'), drop = TRUE)

# Free up space
x$remove_data()



# This example needs at least 4 GB hard disk space and it takes
# time to run for performance profile

# Speed test
path <- tempfile()
x <- create_lazyarray(path, 'complex', dim = c(100,200,300,20), 
                      multipart = TRUE, multipart_mode = 1)

# automatically call x$remove_data() upon garbage collection
x$flag_auto_clean(TRUE)


# set data (4 GB data) using 4 cores, compression level 50
# data creation ~10 s, disk IO ~15-20 seconds, ~250MB/s

system.time({
  lapply(1:20, function(ii){
    # Generating partition data (~10 sec)
    tmp <- rnorm(100*200*300) * (1+2i)
    
    # Write to disk (~16 sec)
    x[,,,ii] <- tmp
    NULL
  })
})

# Reading 64 MB data using 4 cores
# ~0.25 seconds

system.time({
  x[1:100, sample(200, 200), 100:1, 2:4]
})

# This call requires 4GB of RAM
# Reading all 4GB data using 4 cores
# ~4 seconds (1 GB/s)

system.time({
  x[]
})

lazyarray documentation built on July 18, 2020, 9:06 a.m.