textTokenizeAndCount: Tokenize and count

In OscarKjell/text: Analyses of Text using Transformers Models from HuggingFace, Natural Language Processing and Machine Learning

Tokenize and count

Description

Tokenize and count

Usage

textTokenizeAndCount(data, n_remove_threshold = 3)

Arguments

data	(string) Language to tokenise and count.
n_remove_threshold	(numeric) Threshold deciding which words to remove

Value

A word-frequency data frame (can be saved to a model object or compared in textDomainCompare).

See Also

see textDomainCompare

Examples

## Not run: 
textTokenizeAndCount(Language_based_assessment_data_8["harmonytexts"])

## End(Not run)

OscarKjell/text documentation built on April 3, 2025, 3:07 p.m.