s3_load_folder: Load and combine S3 folder of tabular files

In fcocquemas/hfty.helpers: Helper functions (paths, input/output, cloud...)

Load and combine S3 folder of tabular files

Description

Load and combine S3 folder of tabular files

Usage

s3_load_folder(
  prefix,
  conf = list(),
  FUN = function(x, files) {     x },
  pattern = "",
  max = Inf,
  quiet = FALSE,
  cl = 1,
  relaunch_times = 1,
  relaunch_cl = 1,
  relaunch_check_fun = is.data.frame,
  ...
)

Arguments

prefix	string prefix of S3 files to select. Needs to include the URL scheme and bucket name (s3://bucket_name/)
conf	list with 'key', 'secret', 'region', and 'base_url' parameters. If missing, will look for AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_DEFAULT_REGION in environment variables.
FUN	function to apply to every file being loaded
pattern	string regexp to filter results
max	integer max number of records to return, pre-filtering. Default to Inf
quiet	logical turn off output messages, default to FALSE
cl	integer number of cores to use

Examples

## Not run: 
s3_load_folder("s3://hfty-test-bucket/", pattern = "\\.rds$")

## End(Not run)

fcocquemas/hfty.helpers documentation built on Feb. 13, 2023, 7:25 a.m.