csv_to_sqlite: Fork of inborutils::csv_to_sqlite to save a delimited text...

In apoorvalal/LalRUtils: Lal's utility scripts for personal use

Fork of inborutils::csv_to_sqlite to save a delimited text table into a single table sqlite database that can then be munged w dplyr/ dbplyr / sqldf

Description

The table can be a comma separated (csv) or a tab separated (tsv) or any other delimited text file. The file is read in chunks. Each chunk is copied in the same sqlite table database before the next chunk is loaded into memory. See the INBO tutorial Handling large files in R to learn more.

Usage

csv_to_sqlite(
  csv_file,
  sqlite_file,
  table_name,
  delimiter = ",",
  pre_process_size = 1000,
  chunk_size = 50000,
  show_progress_bar = TRUE,
  ...
)

Arguments

csv_file	Name of the text file to convert.
sqlite_file	Name of the newly created sqlite file.
table_name	Name of the table to store the data table in the sqlite database.
pre_process_size	Number of lines to check the data types of the individual columns (default 1000).
chunk_size	Number of lines to read for each chunk (default 50000).
show_progress_bar	Show progress bar (default TRUE).
...	Further arguments to be passed to read_delim.
delim	Text file delimiter (default ",").

Value

a SQLite database

Remark

The callback argument in the read_delim_chunked function call refers to the custom written callback function 'append_to_sqlite' applied to each chunk.

apoorvalal/LalRUtils documentation built on Sept. 22, 2023, 5:53 p.m.