In talegari/fileplyr: Chunk Processing or Split-Apply-Combine on Delimited Files and Distributed Dataframes

fileplyr

plyr for CSV files

Perform chunkwise processing or split-apply-combine on data in a delimited file(example: CSV) across multiple cores of a single machine with low memory footprint. These functions are a convenient wrapper over the versatile package 'datadr'.

Examples

Load fileplyr package

library("fileplyr")

• split-apply-combine example

write.table(mtcars, "mtcars.csv", row.names = FALSE, sep = ",")
temp <- fileply(file     = "mtcars.csv"
             , groupby = c("carb", "gear")
             , fun     = identity
             , collect = "list"
             , sep     =  ","
             , header  = TRUE
             )
temp
to_tibble(temp)
unlink("mtcars.csv")

• chunkwise processing example

write.table(mtcars, "mtcars.csv", row.names = FALSE, sep = ",")
temp <- fileply(file     = "mtcars.csv"
             , chunk   = 10
             , fun     = function(x){list(nrow(x))}
             , collect = "dataframe"
             , sep     =  ","
             , header  = TRUE
             )
temp
unlink("mtcars.csv")

talegari/fileplyr documentation built on May 31, 2019, 2:51 a.m.