SentencePieceTokenizer: SentencePieceTokenizer
In fastai: Interface to 'fastai'

View source: R/text_core.R

SentencePieceTokenizer

R Documentation

SentencePieceTokenizer

Description

SentencePiece tokenizer for 'lang'

Usage

SentencePieceTokenizer(
  lang = "en",
  special_toks = NULL,
  sp_model = NULL,
  vocab_sz = NULL,
  max_vocab_sz = 30000,
  model_type = "unigram",
  char_coverage = NULL,
  cache_dir = "tmp"
)

Arguments

`lang`	lang
`special_toks`	special_toks
`sp_model`	sp_model
`vocab_sz`	vocab_sz
`max_vocab_sz`	max_vocab_sz
`model_type`	model_type
`char_coverage`	char_coverage
`cache_dir`	cache_dir

Value

None

fastai documentation built on March 31, 2023, 11:41 p.m.

fastai index

README.md Audio Classification Basic Image Classification Basic Tabular Bayesian Optimisation Callbacks Custom Image Classification Data augmentation GPT2 Head pose Low-level ops Medical image Migrating from Catalyst Migrating from Ignite Migrating from Lightning Migrating from Pytorch Multilabel classification Object detection Optimizer Question-Answering RoBERTa Speech Recognition Super-Resolution GAN Text-summarization Time-Series

rdrr.io home R language documentation Run R code online

CRAN packages Bioconductor packages R-Forge packages GitHub packages

Note that we can't provide technical support on individual packages. You should contact the package authors for that.

Tweet to @rdrrHQ

GitHub issue tracker

ian@mutexlabs.com