big.read.table: Read in chunks from a large file with row/column filtering to...

In jayemerson/big.data.frame: Extend big.char and bigmemory's big.matrix to support larger-than-RAM data.frame-like objects in R.

Description

Read in chunks from a large file with row/column filtering to obtain a reasonable-sized data.frame.

Usage

big.read.table(file, nrows = 1e+05, sep = ",", header = TRUE,
  row.names = NULL, cols = NULL, rowfilter = NULL, as.is = TRUE,
  estimate = FALSE, location = NULL)

Arguments

file	the name of the file, obviously
nrows	the chunk size; consider reducing this if there are lots of columns
sep	by default we expect a CSV file
header	is TRUE by default
row.names	I really dislike row names
cols	for filtering column by name or number (supporting negative indexing)
rowfilter	a function that is assumed to take a chunk as a data frame and return a smaller data frame (with fewer rows), separately from the column filtering.
as.is	TRUE by default
estimate	do a preliminary estimation of the work to be done, and then have a chance to bail out if it looks like a bad idea
location	where do you want it?

Examples

data(CO2)
write.csv(CO2, "CO2.csv", row.names=FALSE)
x <- big.read.table("CO2.csv", nrows=10)
dim(x)
y <- big.read.table("CO2.csv", nrow=10,
                    rowfilter=function(a) a[a$conc!=1000,])
dim(y)
head(x)

jayemerson/big.data.frame documentation built on May 18, 2019, 5:57 p.m.