ingest: Read a data file and normalize it to Parquet

In rtemis: Schema-Driven Machine Learning and Visualization

Read a data file and normalize it to Parquet

Description

The first step of any pipeline. A delimited file or a spreadsheet carries no usable type information; Parquet does. Ingesting once means nothing downstream infers a type a second time, and the decisions that were made are in the config.

The config's format is checked against the file, and a mismatch is an error: a config written for a delimited file sets options a Parquet reader would ignore.

Returns a manifest describing what happened: the file in, the file out, a DataFingerprint of the data, the config, and which engine ran it. The engine is named from the start because it is unrecoverable later – once a second implementation exists, nothing else could say which produced a given Parquet.

Usage

ingest(path, outfile, config = NULL, overwrite = FALSE, verbosity = 1L)

Arguments

path	Character: Path to the input file.
outfile	Character: Path to write the Parquet file to.
config	Optional IngestConfig: How to read it. If NULL, the defaults for the file's own format.
overwrite	Logical: If TRUE, overwrite outfile if it exists.
verbosity	Integer: Verbosity level.

Value

Named list: the ingest manifest.

Author(s)

EDG

Examples

## Not run: 
ingest("data.csv", "data.parquet")
ingest("data.csv", "data.parquet", setup_DelimitedIngest(sep = ";"))

## End(Not run)

rtemis documentation built on Oct. 20, 2026, 1:06 a.m.