clean_text: Perform invalid character removal and whitespace cleanup on...

View source: R/tokenization.R

clean_textR Documentation

Perform invalid character removal and whitespace cleanup on text.

Description

(R implementation of BasicTokenizer._clean_text from BERT: tokenization.py.)

Usage

clean_text(text)

Arguments

text

A character scalar.

Value

Cleaned up text.


jonathanbratt/RBERT documentation built on Jan. 26, 2023, 4:15 p.m.