read.regress.data.ff: Read in Tabulated Data and Compute Summary Statistics

In BayesSummaryStatLM: MCMC Sampling of Bayesian Linear Models via Summary Statistics

Description

This function reads in tabulated data sets and produces summary statistics needed for Bayesian linear regression models for use in the function bayes.regress() (in this package). Big data sets that are too large to fit into R memory are handled using functions from package ff. The function takes as input data files with the predictor variables X and response values Y, and returns the summary statistics X'X, X'Y and Y'Y that are used as an input to the function bayes.regress() (in this package) for Bayesian linear regression models. The function supports reading data sets that are split across multiple files.

Usage

read.regress.data.ff(filename=NULL,predictor.cols=NA,response.col=NA,update.summaries=NULL
                   , fileEncoding = "", nrows = -1, first.rows = 1e5, next.rows = 1e5
                   , levels = NULL, appendLevels = TRUE,FUN = "read.table",transFUN = NULL
                   , asffdf_args = list(), BATCHBYTES = getOption("ffbatchbytes")
                   , VERBOSE = FALSE, header = FALSE, sep = ",", quote = "\"'", dec = "."
                   , numerals = c("allow.loss", "warn.loss", "no.loss")
                   , na.strings = "NA", colClasses = "numeric", skip = 0
                   , check.names = TRUE, fill = TRUE, strip.white = FALSE
                   , blank.lines.skip = TRUE, comment.char = "#", allowEscapes = FALSE
                   , flush = FALSE, skipNul = FALSE)

Arguments

filename	the name of a file or a list of file names, from which the data will be read. By default it is assumed that the file(s) contain data in comma separated format; this can be changed using the sep argument. File names must be in the format acceptable by standard functions such as read.table. Default = NULL.
predictor.cols	a vector of integers that specifies the columns to treat as predictor variables, to create the design matrix X. By default, all columns after the first are treated as columns with predictor variables. Default = NA.
response.col	an integer that specifies the column that contains the response variable values, to create the response vector Y. By default, the first column that is not used as a predictor variable column is selected as the response values column. Default = NA.
update.summaries	The name of the R object containing previously-calculated summary statistics (if applicable), to be updated with new data. This must be a list similar in structure to the returned value, containing entries xtx, xty, yty and numsamp.data of appropriate dimensions and class that match the data contained in filename. Default = NULL.

The remaining arguments are passed directly to the function read.table.ffdf(). Below is a short description of the arguments we recommend to set manually in accordance with memory limitations and data structure:

first.rows	the number of rows to read in the first chunk of data. Default = 100,000.
next.rows	the number of rows to read in the remaining chunks of data. Default = 100,000.
sep	the character that separates the columns of data. Default = ",".

For the arguments below the default settings should perform well. However, in some situations adjusting these arguments may improve memory use and running time.

fileEncoding	a string that describes the file's character encoding
nrows	an integer specifying how many rows should be read from the file
levels	an optional list of items with col.names or factor columns. See read.table.ffdf().
appendLevels	a logical vector of permission to expand levels
FUN	specifies which standard R function is used to read the data.
transFUN	an optional filtering function to be applied to each chunk of data. See read.table.ffdf().
asffdf_args	an optional list of parameters to be passed to as.ffdf() function.
BATCHBYTES	an integer limiting the size of the data.frame used to store each chunk of the data
VERBOSE	See read.table.ffdf().
header	a logical value indicating if the first row is the header row
quote	a character string specifying which character will be treated as quoting characters
dec	a character used for decimal dot
numerals	see read.table().
na.strings	strings treated as NA values
colClasses	a vector that describes the data types in each column. Numeric by default.
skip	how many first lines in the file should be skipped
check.names	see read.table().
fill	logical value that turns on automatic padding of the rows in case they have different lengths
strip.white	affects the processing of the columns with declared character type. See scan().
blank.lines.skip	logical value, indicating whether empty lines should be ignored
comment.char	character specifying the comment marker
allowEscapes	logical. If TRUE, the escaped strings will be parsed. See scan().
flush	see read.table().
skipNul	logical: should nuls be skipped?

Details

The function reads in data and computes summary statistics to be used in Bayesian linear regression by the function bayes.regress() (in this package). The function assumes the linear regression model will have a non-zero y-intercept; this option can be changed in the bayes.regress() function (see bayes.regress() help for details).

Value

The returned value for the read.regress.data.ff() function is a list containing the summary statistics named xtx (for X'X), xty (for X'Y), yty (for Y'Y) and the total number of data values numsamp.data. The summary statistic xtx contains a square matrix obtained by computing a dot product of the predictor variables data X with itself; a leading column of 1's is added to X for the y-intercept term. xty contains the vector obtained by computing the dot product of the transposed predictor variables data X with response variable data Y; a leading column of 1's is added to X for the y-intercept term. yty contains the dot product of the response variable data Y with itself. numsamp.data is the number of data values read from the data file(s); this number may be smaller than the number of rows in the data file, since some of the rows with missing data may be skipped according to specified function arguments. The summary statistics X'X, X'Y and Y'Y are summed over data chunks by the following, for m = 1,...,M chunks:

X'X = sum_{m=1}^{M} (X'_{m})( X_{m})

X'Y = sum_{m=1}^{M} (X'_{m})( Y_{m})

Y'Y = sum_{m=1}^{M} (Y'_{m})( Y_{m})

The returned values are used as input to the function bayes.regress() (in this package). Note that the matrix X is given a leading column of 1's by default, for the y-intercept term of the Bayesian linear regression model. This can be removed by specifying a model with zero intercept in the function bayes.regress() (see bayes.regress() help for details).

References

Carlin, B.P. and Louis, T.A. (2009) Bayesian Methods for Data Analysis, 3rd ed., Boca Raton, FL: Chapman and Hall/CRC Press.

Gelman, A., Carlin, J.B., Stern, H.S., Dunson, D.B., Vehtari, A. and Rubin, D.B. (2013) Bayesian Data Analysis, 3rd ed., Boca Raton, FL: Chapman and Hall/CRC Press.

Adler, D., Glaser, C., Nenadic, O., Oehlschlagel, J. and Zucchini, W. (2013) ff: memory-efficient storage of large data on disk and fast access functions. R package: https://CRAN.R-project.org/package=ff.

Examples

# The package includes several example data files, illustrated here.

#########
# Example
#########
# The following command finds the location of the data file
# that includes 4 predictor variables and 20,000 simulated data values.

filename <- system.file('data/regressiondata.nz.all.csv.gz', package='BayesSummaryStatLM')

# The file is formatted so that the simulated response variable is in the
# first column, and columns 2 to 5 contain simulated predictor variables.
# The simulated coefficients are: beta <- c(0.76, -0.92, 0.64, 0.57, -1.65), 
# where the first value is the y-intercept term in the Bayesian linear 
# regression model.  The sigma-squared term, i.e. the variance of the normally 
# distributed error terms, is simulated as: sigmasq <- 0.25

## Next, read the data and compute the summary statistics using the
# "read.regress.data.ff()" function.  By default, the first column is assumed 
# to be the response variable, and the remaining columns are assumed to contain
# predictor variable values.  The function will check if the file exists and 
# can be read.

data.values <- read.regress.data.ff(filename)
data.values

BayesSummaryStatLM documentation built on July 1, 2021, 5:06 p.m.