SchemaOnRead Vignettes
In SchemaOnRead: Automated Schema on Read

Introduction

SchemaOnRead provides tools for implementing the schema-on-read technique for R, including a single function call (e.g., schemaOnRead("filename")) that reads text (TXT), comma separated value (CSV), raster image (BMP, PNG, GIF, TIFF, and JPG), R data (RDS), HDF5 (H5), NetCDF (CS), spreadsheet (XLS, XLSX, ODS, and DIF), Weka Attribute-Relation File Format (ARFF), Epi Info (REC), Pajek network (PAJ), R network (NET), Hypertext Markup Language (HTML), SPSS (SAV), Systat (SYS), and Stata (DTA) files. It also recursively reads folders (e.g., schemaOnRead("folder")), returning a nested list of the contained elements.

Example Uses

One way to use SchemaOnRead is to recursively load a folder. The result is a named list of elements for each entry in the folder's tree. The element names are converted to valid R variable names corresponding to the file or folder names. Sub-elements (e.g., files or subfolders) of a folder can be accessed using the R named list ("\$") operator followed by the sub-element name. Files or folders with names that do not conform to standard R variable naming requirements can be accessed using single quote notation (e.g., "results$'Nonconforming Name!'").

Another way to use SchemaOnRead is to conveniently load a file without needing to handle the specifics of the file format. In this case the result is a variable containing the file contents.

An example showing how to read a folder tree starting in "../inst/extdata" is shown below. In this case, the contents of the "dir1/Data.csv" file within "../inst/extdata" is shown by accessing "results\$dir1\$Data.csv" as needed. The text file is as follows:

  Name   Size Weight
1    A  Small      1
2    B Medium      2
3    C  Large      3

The results in R are as follows:

library(SchemaOnRead)
results <- schemaOnRead("../inst/extdata")
print(results$dir1$Data.csv)

Individual files can also be easily accessed. An example XML source file in the "../inst/extdata/data.xml" folder is as follows:

{xml_document}
<example>
[1] <to>A</to>
[2] <from>B</from>
[3] <title>Important<subtitle>File</subtitle></title>
[4] <text>Read me.</text>

The results in R are as follows:

library(SchemaOnRead)
xmlFile <- schemaOnRead("../inst/extdata/data.xml")
print(xmlFile)

The 'verbose' flag can be used to trace a call's progress or diagnose issues.

library(SchemaOnRead)
folder <- schemaOnRead("../inst/extdata", verbose = TRUE)

Implementation

The SchemaOnRead package uses a recursive implementation. The initial user function call iterates over the given list of processors, invoking each in turn until one returns a non-null value. Processors are sequentially invoked in the order given by the input list, scanning from index number one upwards. Processing continues as long as each processor returns null. The results from the first processor to return a non-null value is stored as the content for the entry and processing of that entry stops. All of the results are stored in a named list. The order of the resulting list is given by the processing order. The list names are taken from the entry names (e.g., file or folder names). Files and and directories with names that do not conform to R variable naming requirements can be accessed using single quote notation (e.g., "results$'Nonconforming Name!'").

Several special processors are defined. These include processors for nonexistent entries, directories, and entries of unknown types.

The "schemaOnReadProcessDirectory" processor handles directories recursively as previously discussed. It is intended to be the second processor to run in normal lists.

The "processDefaultFile" processor accepts all entries that exist. It returns the value "File Type Unknown" as a string when it succeeds. It is meant to be last processor to run to provide a default value for file types that are not otherwise recognized.

SchemaOnRead itself includes two processing lists. The default list from "defaultProcessors()" is used for standard SchemaOnRead entry processing. The simple processing list from "simpleProcessors()" provides an easy to use starting point for fully customized user processor lists.

User-Defined Processors

New processors can be defined to support user-specified work. New processors are normally prepended to the front of the default list to let them to take precedence while still allowing the standard processors to work, if needed. Alternatively, a list of processors that just recursively scans folders can be found by calling the schemaOnReadSimpleProcessors function. User-specified processors can be added to this list to create a fully customized tool chain. An example showing how to define a new processor follows.

## Load the needed library.
library(SchemaOnRead)

## Define a new processor.
newProcessor <- function(path, ...) {

  # Check the file existance and extensions.
  if (!SchemaOnRead::checkExtensions(path, c("xyz"))) return(NULL)

  ## As an example, attempt to read an XYZ file as a CSV file.
  read.csv(path, header = FALSE)

}

## Define a new processors list.
newProcessors <- c(newProcessor, SchemaOnRead::defaultProcessors())

# Use the new processors list.
schemaOnRead(path = "../inst/extdata", processors = newProcessors)

Summary

SchemaOnRead provides tools for implementing the schema-on-read technique for R, including a single function call that reads a wide range of file types. It also recursively reads folders (e.g., schemaOnRead("folder")), returning a nested list of the contained elements.

Acknowledgements

The submitted manuscript has been created by UChicago Argonne, LLC, Operator of Argonne National Laboratory "Argonne". Argonne, a U.S. Department of Energy Office of Science laboratory, is operated under Contract No. DE-AC02-06CH11357. The U.S. Government retains for itself, and others acting on its behalf, a paid-up nonexclusive, irrevocable worldwide license in said article to reproduce, prepare derivative works, distribute copies to the public, and perform publicly and display publicly, by or on behalf of the Government. Argonne National Laboratory's work was supported under U.S. Department of Energy contract DE-AC02-06CH11357. Argonne National Laboratory???s work was supported under U.S. Department of Energy contract DE-AC02-06CH11357.