hf_load_tokenizer: Load an AutoTokenizer from a pre-tained model

In farach/huggingfaceR: Hugging Face State-of-the-Art Models in R

Load an AutoTokenizer from a pre-tained model

Description

Load Tokenizer for Hugging Face Model

Usage

hf_load_tokenizer(model_id, ...)

Arguments

model_id	The id of the model given in the url by https://huggingface.co/model_name.
...	sent to the 'AutoTokenizer.from_pretained()', accepts named arguments e.g. use_fast https://huggingface.co/docs/transformers/main_classes/tokenizer

Value

A Hugging Face model's tokenizer.

See Also

https://huggingface.co/docs/transformers/main/en/pipeline_tutorial

farach/huggingfaceR documentation built on Feb. 4, 2023, 10:31 p.m.