API for tok
Fast Text Tokenization
| Global functions | |
|---|---|
| decode | Source code |
| decode_batch | Source code |
| decoder_byte_level | Man page |
| enable_padding | Source code |
| enable_truncation | Source code |
| encode | Source code |
| encode_batch | Source code |
| encoding | Man page |
| from_file | Source code Source code |
| from_model | Source code |
| from_pretrained | Source code |
| get_attention_mask | Source code |
| get_decoder | Source code |
| get_ids | Source code |
| get_normalizer | Source code |
| get_padding | Source code |
| get_post_processor | Source code |
| get_pre_tokenizer | Source code |
| get_special_tokens_mask | Source code |
| get_tokens | Source code |
| get_truncation | Source code |
| get_type_ids | Source code |
| get_vocab_size | Source code |
| get_word_ids | Source code |
| len | Source code |
| model_bpe | Man page |
| model_unigram | Man page |
| model_wordpiece | Man page |
| new | Source code Source code Source code Source code Source code Source code Source code Source code Source code Source code Source code Source code Source code Source code Source code Source code Source code Source code Source code |
| no_padding | Source code |
| no_truncation | Source code |
| normalizer_nfc | Man page |
| normalizer_nfkc | Man page |
| pre_tokenizer | Man page |
| pre_tokenizer_byte_level | Man page |
| pre_tokenizer_whitespace | Man page |
| processor_byte_level | Man page |
| save | Source code |
| set_decoder | Source code |
| set_normalizer | Source code |
| set_post_processor | Source code |
| set_pre_tokenizer | Source code |
| tok | Man page |
| tok-package | Man page |
| tok_decoder | Man page |
| tok_model | Man page |
| tok_normalizer | Man page |
| tok_processor | Man page |
| tok_trainer | Man page |
| tokenizer | Man page |
| train_from_files | Source code |
| train_from_sequences | Source code |
| trainer_bpe | Man page |
| trainer_unigram | Man page |
| trainer_wordpiece | Man page |
