subword_text_encoder: Subword Text Encoder

In rstudio/tfds: Interface to 'TensorFlow' Collection of Datasets

Description

Invertible TextEncoder using word pieces with a byte-level fallback.

Usage

subword_text_encoder(vocab_list)

Arguments

vocab_list

list of subwords for the vocabulary. Note that an underscore at the end of a subword indicates the end of the word (i.e. a space will be inserted afterwards when decoding). Underscores in the interior of subwords are disallowed and should use the underscore escape sequence.

See Also

save_subword_text_encoder(), load_subword_text_encoder(), encode() and decode()

rstudio/tfds documentation built on Nov. 25, 2021, 6:20 p.m.