fastglm_source: Read a large dataset in repeatable blocks of rows

View source: R/source.R

fastglm_sourceR Documentation

Read a large dataset in repeatable blocks of rows

Description

A source describes how to obtain a block of observations when a fitting function needs it. This permits repeated model-fitting passes without loading the complete dataset into memory. The supplied function, called a callback, receives the block number and returns its observations.

Usage

fastglm_source(
  callback,
  n_chunks,
  reset = NULL,
  formula = NULL,
  prototype = NULL,
  weights = NULL,
  offset = NULL
)

Arguments

callback

function of chunk index returning a data frame, or a list with 'X', 'y', and optional 'weights' and 'offset'. Two-component models may also supply 'Z' for the binary design and a list of 'count' and 'zero' offsets; otherwise 'Z' defaults to 'X' and a vector offset applies to the count or positive-outcome component.

n_chunks

positive number of chunks.

reset

optional function called before chunk one on every pass.

formula

optional model formula for data-frame blocks. A formula such as 'y ~ x | z' specifies separate count/positive and binary designs.

prototype

example data frame defining all intended factor levels and transformations; it need not contain all observations.

weights, offset

optional column names in data-frame blocks. For a two-component formula, two offset column names supply count/positive and binary offsets, respectively.

Details

When a formula is supplied, 'prototype' is an example data frame used to establish the design columns, factor levels, and transformations. This recorded specification, called the schema, is reused for every block so that a column has the same meaning throughout the analysis. For splines and other data-dependent transformations, the prototype must establish the intended basis, including knots and boundary knots.

Value

A 'fastglm_source' object for [fastglm_streaming()], [fastglm_ordinal_streaming()], or the count and two-part fitting functions.


fastglm documentation built on Nov. 11, 2026, 5:07 p.m.