hf_load_tokenizer: Load an AutoTokenizer from a pre-tained model

In huggingfaceR: Access 'Hugging Face' Models and Datasets

Load an AutoTokenizer from a pre-tained model

Description

Load Tokenizer for Hugging Face Model

Usage

hf_load_tokenizer(model_id, ...)

Arguments

model_id	The id of the model given in the url by https://huggingface.co/model_name.
...	sent to the 'AutoTokenizer.from_pretained()', accepts named arguments e.g. use_fast https://huggingface.co/docs/transformers/main_classes/tokenizer

Value

A Hugging Face model's tokenizer.

See Also

https://huggingface.co/docs/transformers/main/en/pipeline_tutorial

Examples

## Not run: 
tokenizer <- hf_load_tokenizer("distilbert-base-uncased")
tokenizer$tokenize("Hello from R")

## End(Not run)

huggingfaceR documentation built on Aug. 30, 2026, 1:06 a.m.