chunk_loader: Function to load big data to SQL (chunk.loader())

In PHSKC-APDE/Housing: Organizes public housing authority data into a longitudinal record

Function to load big data to SQL (chunk.loader())

Description

chunk_loader divides a data.frame/ data.table into smaller tables so it can be easily loaded to SQL. Experience has shows that loading large tables in 'chunks' is less likely to cause errors.

Usage

chunk_loader(
  DTx,
  connx,
  chunk.size = 1000,
  schemax = NULL,
  tablex = NULL,
  overwritex = F,
  appendx = T,
  field.typesx = NULL
)

Arguments

DTx	A data.table/data.frame
connx	The name of the relevant database connection that you have open
chunk.size	The number of rows that you desire to have per chunk
schemax	The name of the schema where you want to write the data
tablex	The name of the table where you want to write the data
overwritex	Do you want to overwrite the existing tables? Logical (T|F).
appendx	Do you want to append to an existing table? Logical (T|F). intentionally redundant with overwritex to ensure that tables are not accidentally overwritten.
field.typesx	Optional ability to specify the fieldtype, e.g., INT, VARCHAR(32), etc.

PHSKC-APDE/Housing documentation built on Feb. 6, 2024, 3:27 a.m.