read_dat: Read compressed files in R

In bcgov/dipr: Provide functions to efficiently import SRE data

Read compressed files in R

Description

Unzip and read into memory. The read_dat_dt function will return a data.table object, replacing the previous as.data.table argument which is now deprecated. You can use the same arguments as the read_dat function. If cache = TRUE, the first time you call this function, the data is cached in a folder called .dipr in a format specified by the cache_type argument. This function expects a data dictionary with the following columns:

• start

• stop

• name

Usage

read_dat(
  data_path,
  data_dict,
  as.data.table = FALSE,
  use_cache = FALSE,
  col_select = NULL,
  col_types = NULL,
  data_format = c("fwf", "csv", "tsv", "csv2"),
  tz = "UTC",
  date_format = "%AD",
  time_format = "%AT",
  ...
)

read_dat_dt(...)

Arguments

data_path	A path or a vector of paths to a .dat.gz file. If supplying a vector of paths, they must share a common data dictionary.
data_dict	A data.frame with start, stop and name columns
as.data.table	Deprecated. See read_dat_dt
use_cache	deprecated,
col_select	A vector of column names
col_types	One of NULL, a cols() specification, or a string. See vignette("readr") for more details. If NULL, all column types will be imputed from guess_max rows on the input interspersed throughout the file. This is convenient (and fast), but not robust. If the imputation fails, you'll need to increase the guess_max or supply the correct types yourself. Column specifications created by list() or cols() must contain one column specification for each column. If you only want to read a subset of the columns, use cols_only(). Alternatively, you can use a compact string representation where each character represents one column: c = character i = integer n = number d = double l = logical f = factor D = date T = date time t = time ? = guess _ or - = skip By default, reading a file without a column specification will print a message showing what readr guessed they were. To remove this message, set show_col_types = FALSE or set 'options(readr.show_col_types = FALSE).
data_format	the format of the input data. Default is "fwf", other choices are "csv", "csv2", "tsv"
tz	what timezone should datetime fields use? Default UTC. This is recommended to avoid timezone pain, but remember that the data is in UTC when doing analysis. See OlsonNames() for list of available timezones.
date_format	date format for columns where date format is not specified in col_types
time_format	time format for columns where time format is not specified in col_types
...	Arguments passed on to readr::read_fwf file Either a path to a file, a connection, or literal data (either a single string or a raw vector). Files ending in .gz, .bz2, .xz, or .zip will be automatically uncompressed. Files starting with http://, https://, ftp://, or ftps:// will be automatically downloaded. Remote gz files can also be automatically downloaded and decompressed. Literal data is most useful for examples and tests. To be recognised as literal data, the input must be either wrapped with I(), be a string containing at least one new line, or be a vector containing at least one string with a new line. Using a value of clipboard() will read from the system clipboard. col_positions Column positions, as created by fwf_empty(), fwf_widths() or fwf_positions(). To read in only selected fields, use fwf_positions(). If the width of the last column is variable (a ragged fwf file), supply the last end position as NA. id The name of a column in which to store the file path. This is useful when reading multiple input files and there is data in the file paths, such as the data collection date. If NULL (the default) no extra column is created. locale The locale controls defaults that vary from place to place. The default locale is US-centric (like R), but you can use locale() to create your own locale that controls things like the default time zone, encoding, decimal mark, big mark, and day/month names. na Character vector of strings to interpret as missing values. Set this option to character() to indicate no missing values. comment A string used to identify comments. Any text after the comment characters will be silently ignored. trim_ws Should leading and trailing whitespace (ASCII spaces and tabs) be trimmed from each field before parsing it? skip Number of lines to skip before reading data. n_max Maximum number of lines to read. guess_max Maximum number of lines to use for guessing column types. See vignette("column-types", package = "readr") for more details. progress Display a progress bar? By default it will only display in an interactive session and not while knitting a document. The automatic progress bar can be disabled by setting option readr.show_progress to FALSE. name_repair Handling of column names. The default behaviour is to ensure column names are "unique". Various repair strategies are supported: "minimal": No name repair or checks, beyond basic existence of names. "unique" (default value): Make sure names are unique and not empty. "check_unique": no name repair, but check they are unique. "universal": Make the names unique and syntactic. A function: apply custom name repair (e.g., name_repair = make.names for names in the style of base R). A purrr-style anonymous function, see rlang::as_function(). This argument is passed on as repair to vctrs::vec_as_names(). See there for more details on these terms and the strategies used to enforce them. num_threads The number of processing threads to use for initial parsing and lazy reading of data. If your data contains newlines within fields the parser should automatically detect this and fall back to using one thread only. However if you know your file has newlines within quoted fields it is safest to set num_threads = 1 explicitly. show_col_types If FALSE, do not show the guessed column types. If TRUE always show the column types, even if they are supplied. If NULL (the default) only show the column types if they are not explicitly supplied by the col_types argument. lazy Read values lazily? By default the file is initially only indexed and the values are read lazily when accessed. Lazy reading is useful interactively, particularly if you are only interested in a subset of the full dataset. Note, if you later write to the same file you read from you need to set lazy = FALSE. On Windows the file will be locked and on other systems the memory map will become invalid. skip_empty_rows Should blank rows be ignored altogether? i.e. If this option is TRUE then blank rows will not be represented at all. If it is FALSE then they will be represented by NA values in all the columns.

Functions

• read_dat_dt:

Examples

data_dict_path <- dipr_example("starwars-dict.txt")
dict <- read.table(data_dict_path)
dat_path <- dipr_example("starwars-fwf.dat.gz")
read_dat(data_path = dat_path,
         data_dict = dict,
         col_types = "cddlcD",
         date_format = "%Y%m%d")

bcgov/dipr documentation built on June 1, 2022, 12:32 p.m.