pre_process: Preprocess a Vector of Text Documents (Chunked, With...
In quickSentiment: A Fast and Flexible Pipeline for Text Classification

pre_process

R Documentation

Preprocess a Vector of Text Documents (Chunked, With Row-Level Fault Isolation)

Description

This function cleans raw text data using the same fast, vectorized string and tokenization operations as before, but runs them in chunks. If a chunk processes cleanly (the common case), it stays fully vectorized and fast. If a chunk throws an error (e.g. from a malformed row), only THAT chunk falls back to row-by-row processing so the exact offending row(s) can be isolated, quarantined, and set to NA – without slowing down or risking the rest of the dataset.

Usage

pre_process(
  doc_vector,
  remove_brackets = TRUE,
  remove_urls = TRUE,
  remove_html = TRUE,
  remove_nums = FALSE,
  remove_emojis_flag = TRUE,
  to_lowercase = TRUE,
  remove_punct = TRUE,
  remove_stop_words = TRUE,
  custom_stop_words = NULL,
  keep_words = NULL,
  lemmatize = TRUE,
  retain_negations = TRUE,
  chunk_size = 5000
)

Arguments

`doc_vector`	A character vector where each element is a document.
`remove_brackets`	A logical value indicating whether to remove text in square brackets.
`remove_urls`	A logical value indicating whether to remove URLs and email addresses.
`remove_html`	A logical value indicating whether to remove HTML tags.
`remove_nums`	A logical value indicating whether to remove numbers.
`remove_emojis_flag`	A logical value indicating whether to remove common emojis.
`to_lowercase`	A logical value indicating whether to convert text to lowercase.
`remove_punct`	A logical value indicating whether to remove punctuation.
`remove_stop_words`	A logical value indicating whether to remove English stopwords.
`custom_stop_words`	A character vector of additional custom words to remove (e.g., c("rt", "via")). Default is NULL.
`keep_words`	A character vector of words to protect from deletion (e.g., c("no", "not", "nor")). Default is NULL.
`lemmatize`	A logical value indicating whether to lemmatize words to their dictionary form.
`retain_negations`	Logical. If `TRUE` (the default), automatically protects common negation words.
`chunk_size`	Integer. Number of documents processed per vectorized batch. Defaults to 5000. Larger chunks are faster on clean data; smaller chunks isolate failures faster when a bad chunk needs to fall back to row-by-row.

Value

A character vector of cleaned text, same length as doc_vector. Rows that could not be processed are set to NA_character_. Details on any failures are attached as attr(result, "quarantine"), a data.frame with row_index, original_text (truncated to 200 chars), and error_message.

Examples

raw_text <- c(
  "This is a <b>test</b>! Visit https://example.com",
  "Email me at test.user@example.org [important]"
)
clean_text <- pre_process(raw_text)
print(clean_text)

# Check for any quarantined rows after a run
attr(clean_text, "quarantine")

quickSentiment documentation built on Aug. 29, 2026, 1:07 a.m.