to_arrow_stream: Stream a dbplyr table on DuckDB into Arrow

In duckdb: DBI Package for the DuckDB Database Management System

Stream a dbplyr table on DuckDB into Arrow

Description

to_arrow_stream() is a streaming counterpart of arrow::to_arrow() for dbplyr tables on a DuckDB connection. It sends the query with DBI::dbGetQueryArrow() and hands the stream to arrow::as_record_batch_reader(), so the rows arrive batch by batch and the result is not held twice. arrow::to_arrow() materializes the whole result first, through dbSendQuery(arrow = TRUE).

The streaming comes with hard limits, listed in the handbook, in ⁠usage/integrations/⁠. Use to_arrow_stream() where a large result goes straight into Arrow and nothing else runs on its connection until the reader has been read to the end. Where that cannot be arranged, use arrow::to_arrow(), or run everything else on a second connection.

Usage

to_arrow_stream(.data)

Arguments

.data

A dbplyr table on a DuckDB connection, or an Arrow object, which is returned unchanged.

Value

An Arrow RecordBatchReader, or an arrow_dplyr_query over it if .data is grouped.

Examples

con <- dbConnect(duckdb())
dbWriteTable(con, "mtcars", mtcars)

# Read the reader to the end before anything else runs on `con`.
reader <- to_arrow_stream(dplyr::filter(dplyr::tbl(con, "mtcars"), cyl == 4))
as.data.frame(reader$read_table())

# Another statement on `con` breaks a reader that has not been read yet.
reader <- to_arrow_stream(dplyr::tbl(con, "mtcars"))
dbGetQuery(con, "SELECT 1")
try(reader$read_table())

# A second connection to the same database leaves the reader alone.
other <- dbConnect(con@driver)
reader <- to_arrow_stream(dplyr::tbl(con, "mtcars"))
dbGetQuery(other, "SELECT count(*) FROM mtcars")
reader$read_table()$num_rows

dbDisconnect(other)
dbDisconnect(con)

duckdb documentation built on Oct. 23, 2026, 5:06 p.m.