process_chunks: process a table in chunks

Description Usage Arguments Examples

View source: R/process_chunks.R

Description

process a table in chunks

Usage

1
2
3
4
5
6
7
8
process_chunks(
  file,
  process_fn,
  streamable_table = NULL,
  lines = 50000L,
  encoding = Sys.getenv("encoding", "UTF-8"),
  ...
)

Arguments

file

path to a file

process_fn

a function of a chunk

streamable_table

interface for serializing/deserializing in chunks

lines

number of lines to read in a chunk.

encoding

encoding to be assumed for input files.

...

additional arguments to streamable_table$read method.

Examples

1
2
3
con <- system.file("extdata/mtcars.tsv.gz", package="arkdb")
dummy <- function(x) message(paste(dim(x), collapse = " x "))
process_chunks(con, dummy, lines = 8)

arkdb documentation built on April 5, 2021, 5:07 p.m.