s3GetData: Loads data from several files in S3 based on the path prefix

In madedotcom/retl: Support for common ETL operations and connections

Loads data from several files in S3 based on the path prefix

Description

Loads data from several files in S3 based on the path prefix

Usage

s3GetData(
  path,
  header = TRUE,
  bucket = s3DefaultBucket(),
  root = s3DefaultRoot(),
  s3.get.fun = s3GetFile,
  fill = TRUE
)

Arguments

path	is the path to the S3 object
header	flag defines whether file has header
bucket	name of the S3 bucket. Defaults to value in AWS_S3_BUCKET environment variable.
root	project root path that is appended before the path in the argument. Defaults to value in AWS_S3_ROOT environment variable.
s3.get.fun	Function that will be used to read data from the individual files.
fill	see data.table::rbindList

Value

data.table with data combined from files. It will be Null data.table if path did not match any of the files in S3 bucket.

Note

You need make sure that all targeted files have the same header signature: order of the fields should not matter.

madedotcom/retl documentation built on Sept. 19, 2022, 3:28 a.m.