s3GetFile: Loads data from AWS S3 into data.table object

In madedotcom/retl: Support for common ETL operations and connections

Loads data from AWS S3 into data.table object

Description

Convention based wrapper functions that allow to load data files in S3 into data.table

Usage

s3GetFile(path, bucket = s3DefaultBucket(), root = s3DefaultRoot(), ...)

s3GetFile.csv(
  path,
  bucket = s3DefaultBucket(),
  root = s3DefaultRoot(),
  header = TRUE
)

s3GetFile.gz(path, bucket = s3DefaultBucket(), root = s3DefaultRoot())

s3GetFile.rds(path, bucket = s3DefaultBucket(), root = s3DefaultRoot())

s3GetFile.zip(
  path,
  bucket = s3DefaultBucket(),
  root = s3DefaultRoot(),
  fread.fill = FALSE
)

s3GetFile.json.gz(path, bucket = s3DefaultBucket(), root = s3DefaultRoot())

Arguments

path	is the path to the S3 object
bucket	name of the S3 bucket. Defaults to value in AWS_S3_BUCKET environment variable.
root	project root path that is appended before the path in the argument. Defaults to value in AWS_S3_ROOT environment variable.
...	additional arguments that will be passed to extension specific calls.
header	flag defines whether file has header
fread.fill	boolean value to indicate if fill param should be used. Defaults to FALSE.

Details

Environment variables required for aws.s3 access:

• AWS_ACCESS_KEY_ID = "mykey"

• AWS_SECRET_ACCESS_KEY = "mysecretkey"

• AWS_DEFAULT_REGION = "us-east-1"

• AWS_SESSION_TOKEN = "mytoken"

s3GetFile calls extension specific files based on the path of the file.

Value

s3GetFile gets data from source .csv file

's3GetFile.csv' loads data from '.csv' files

's3GetFile.gz' loads data from '.gz' files

's3GetFile.rds' loads data from '.rds' files

's3GetFile.zip' loads data from '.zip' files

's3GetFile.json.gz' loads data from '.json.gz' files

See Also

aws.s3 package documentation for access details: https://github.com/cloudyr/aws.s3

madedotcom/retl documentation built on Sept. 19, 2022, 3:28 a.m.