compress_incremental: Combine Results Saved by 'save_objects' with...
In indexr: A Thoughtful Saver of Results

compress_incremental

R Documentation

Combine Results Saved by `save_objects` with `incremental=TRUE`

Description

This function is only intended to be used after save_objects with incremental=TRUE. In this case, save_objects with save results under temporary hashes in a folder with the hash corresponding the the parameters. compress_incremental then combines the results and saves them under the corresponding hash and deletes the old directory with the temporary results.

Usage

compress_incremental(
  folder,
  parameters_list,
  hash_includes_timestamp = FALSE,
  ignore_na = TRUE,
  alphabetical_order = TRUE,
  algo = "xxhash64",
  ignore_script_name = FALSE,
  remove_folder = TRUE
)

Arguments

`folder`	Character string specifying the path to the directory where the temporary folder was saved (should be the same as supplied to `save_objects`.
`parameters_list`	The named list of arguments used with `save_objects`.
`hash_includes_timestamp`	Logical. If `TRUE`, the timestamp is included in the hash generation.
`ignore_na`	Logical. If `TRUE`, `NA` values in `parameters_list` are ignored during hash generation.
`alphabetical_order`	Logical. If `TRUE`, the names in `parameters_list` are sorted alphabetically before hash generation.
`algo`	Character string specifying the hashing algorithm to use. Default is `"xxhash64"`. See `?digest`
`ignore_script_name`	Logical. If `TRUE`, the script name is ignored during hash generation.
`remove_folder`	Logical. If `TRUE`, the folder and the temporary results files will be discarded after the combined results are saved.

Details

If the individual results can be put into a data.frame they will be, otherwise they will be stored as a list.

Value

No return value. This function is called for its side effects.

Examples

## Save results incrementally
params <- list(a = "1", b = "2")

tmp_dir <- file.path(tempdir(), "example")
dir.create(tmp_dir)
for (i in 1:10) {
  save_objects(tmp_dir, data.frame(idx = i, val = rnorm(1)), params, incremental = TRUE)
}

## See contents of tmp directory for incremental file
list.files(file.path(tmp_dir, generate_hash(params)))

## Compress results into a single file
compress_incremental(tmp_dir, params)
list.files(tmp_dir)

## Read in compressed file and view results
read_objects(tmp_dir, params)

## Cleanup
unlink(tmp_dir, recursive = TRUE)

indexr documentation built on April 12, 2025, 1:32 a.m.