generate_to_file: Generate a large synthetic data set directly to disk

In SynthesizerPlus: Synthetic Data Generation with Copulas, Evaluation and Multi-Format I/O

Generate a large synthetic data set directly to disk

Description

Generates n synthetic rows in chunks of chunk_size, so that data sets much larger than memory can be produced. For appendable formats (csv, tsv, ndjson, optionally compressed) all chunks go to one file. For other formats (Parquet, Feather, HDF5, fst, RDS, ...) a directory path is created holding one part file per chunk (⁠part-00001.parquet⁠, ...), which read_data() (or e.g. arrow::open_dataset()) reads as a whole.

Usage

generate_to_file(
  object,
  n,
  path,
  format = NULL,
  chunk_size = 1e+05,
  seed = NULL,
  progress = interactive(),
  ...
)

Arguments

object	A fitted sp_synthesizer.
n	Total number of rows.
path	Output file (appendable formats) or directory (other formats).
format	Format name; inferred from path for appendable formats. Required when path is a directory name without extension.
chunk_size	Rows per chunk.
seed	Optional random seed (the whole output is reproducible).
progress	Print progress messages?
...	Passed to generate() (e.g. dependence).

Value

Invisibly, a list with the path, format, number of rows and the files written.

Examples

fit <- fit_synthesizer(iris)
out <- tempfile(fileext = ".csv")
generate_to_file(fit, n = 1000, path = out, chunk_size = 300, seed = 1)
nrow(read_data(out))

dir <- tempfile()
generate_to_file(fit, n = 1000, path = dir, format = "rds", chunk_size = 400)
list.files(dir)

SynthesizerPlus documentation built on Oct. 16, 2026, 5:10 p.m.