read_text_batches: Load data from text files in batches

In TheOpenAIR: Integrate 'OpenAI' Large Language Models into Your 'R' Workflows

Load data from text files in batches

Description

This function reads in data from text files in batches using the read_lines_chunked function from the readr package.

Usage

read_text_batches(text, batch_size = 3500)

Arguments

text	character string, either containing a path to a text file to read in or containing the text.
batch_size	integer indicating the number of lines to read in per batch (default is 3500)

Value

a numeric value indicating the total number of tokens in the text file

Author(s)

Ulrich Matter [e-mail]

Examples

## Not run: 
data_path <- system.file("text", "lorem.txt", package = "OpenAIR")
text_data <- read_text_batches(data_path)

## End(Not run)

TheOpenAIR documentation built on April 27, 2023, 5:10 p.m.