split_text: Split Text into Chunks

In TheOpenAIR: Integrate 'OpenAI' Large Language Models into Your 'R' Workflows

Split Text into Chunks

Description

This function splits a text string into a vector of strings with a specified number of tokens each.

Usage

split_text(text, N)

Arguments

text	A character vector containing the text to be split.
N	An integer specifying the number of tokens per chunk.

Value

A character vector containing the chunks of text with N tokens each.

Examples

large_text <- "This is an example of a large text string 
that will be split into chunks of N tokens each by our custom R function."
num_tokens_per_chunk <- 5
split_text(large_text, num_tokens_per_chunk)

TheOpenAIR documentation built on April 27, 2023, 5:10 p.m.