Reading and writing data: CSV, Parquet, HDF5 and more

In SynthesizerPlus: Synthetic Data Generation with Copulas, Evaluation and Multi-Format I/O

knitr::opts_chunk$set(collapse = TRUE, comment = "#>")
has <- function(p) requireNamespace(p, quietly = TRUE)

Real data come in many formats, and synthetic data often have to be delivered in the format of the original. SynthesizerPlus offers one pair of functions, read_data() and write_data(), that dispatch on the file extension.

library(SynthesizerPlus)

Supported formats

supported_formats()

Only CSV, TSV, RDS and RData need nothing beyond base R. The other backends are optional: install the package listed in the read/write columns for the formats you need (for example install.packages("arrow") for Parquet and Feather). The available column shows what is installed on this machine. Text formats can be compressed by appending .gz, .bz2 or .xz.

Writing and reading

dir <- tempfile("sp-formats-")
dir.create(dir)
syn <- synthesize(iris, n = 300, seed = 1)

write_data(syn, file.path(dir, "iris.csv"))
write_data(syn, file.path(dir, "iris.tsv.gz"))
write_data(syn, file.path(dir, "iris.rds"))
list.files(dir)

Text formats do not store R classes. Pass a template to restore them — factor levels, Date, POSIXct (with time zone), integer and logical columns:

x <- read_data(file.path(dir, "iris.csv"))
class(x$Species)
x <- read_data(file.path(dir, "iris.csv"), template = iris)
levels(x$Species)

match_types() does the same for any data frame you read by other means.

Parquet and Feather (Apache Arrow)

Columnar formats are the usual choice for large data in data science pipelines (Spark, Python/pandas, DuckDB, Polars). Parquet is read and written with arrow or, if that is not installed, the lightweight nanoparquet; Feather / Arrow IPC needs arrow.

f <- file.path(dir, "iris.parquet")
write_data(syn, f)
str(read_data(f))

HDF5

HDF5 is common in scientific computing and is readable from Python (h5py, pandas), MATLAB and Julia. write_data() stores one dataset per column in a group (dataset = "data" by default), with attributes that record column names, factor levels, dates and time zones; read_data() restores the data frame exactly. A plain 1- or 2-dimensional dataset written by other software can be read by giving its name in dataset.

f <- file.path(dir, "iris.h5")
write_data(syn, f)
identical(read_data(f), syn)

JSON, NDJSON, Excel, SPSS, Stata, SAS, fst

write_data(head(syn, 2), file.path(dir, "iris.json"))
cat(readLines(file.path(dir, "iris.json")), sep = "\n")

write_data(syn, file.path(dir, "iris.sav"))   # SPSS
str(read_data(file.path(dir, "iris.sav")))
write_data(syn, file.path(dir, "iris.dta"))   # Stata: names are made valid
names(read_data(file.path(dir, "iris.dta")))

Value labels from SPSS/Stata files are converted to factors when reading. Stata only allows letters, digits and underscores in variable names, so write_data() renames columns such as Sepal.Length (with a warning).

Synthesising straight from a file to a file

real <- read_data(file.path(dir, "iris.rds"))
synthesize(real, n = 1000, seed = 1, file = file.path(dir, "synthetic.csv.gz"))
nrow(read_data(file.path(dir, "synthetic.csv.gz")))

Very large synthetic data sets

generate_to_file() generates data in chunks, so the full data set never has to fit in memory. Appendable formats (CSV, TSV, NDJSON) are written to a single file; for other formats a directory of part files is created, which read_data() — or arrow::open_dataset(), Spark, DuckDB — reads as one table.

fit <- fit_synthesizer(iris)
res <- generate_to_file(fit, n = 50000, path = file.path(dir, "big.csv"),
                        chunk_size = 10000, seed = 1)
nrow(read_data(file.path(dir, "big.csv")))

res <- generate_to_file(fit, n = 50000, path = file.path(dir, "big_rds"),
                        format = "rds", chunk_size = 20000, seed = 1)
basename(res$files)
nrow(read_data(file.path(dir, "big_rds")))

# e.g. 100 million rows as a partitioned Parquet data set
generate_to_file(fit, n = 1e8, path = "synthetic_parquet", format = "parquet",
                 chunk_size = 1e6, seed = 1)
ds <- arrow::open_dataset("synthetic_parquet")

Output is reproducible for a given seed and chunk_size.

unlink(dir, recursive = TRUE)

SynthesizerPlus documentation built on Oct. 16, 2026, 5:10 p.m.