sentencepiece_load_model: Load a Sentencepiece model

In sentencepiece: Text Tokenization using Byte Pair Encoding and Unigram Modelling

Load a Sentencepiece model

Description

Load a Sentencepiece model which either was trained with sentencepiece or which you have found in the wild.

Usage

sentencepiece_load_model(file = "sentencepiece.model")

Arguments

file	path to the file containing the Sentencepiece model

Value

an object of class sentencepiece which is a list with elements

• model: an Rcpp pointer to the model

• model_path: the path to the model

• vocab_size: the size of the Sentencepiece vocabulary

• vocabulary: the Sentencepiece vocabulary which is a data.frame with columns id and subword

Examples

model <- system.file(package = "sentencepiece", "models", "nl-fr-dekamer.model")
model <- sentencepiece_load_model(file = model)

txt <- c("De eigendomsoverdracht aan de deelstaten is ingewikkeld.",
         "On est d'accord sur le prix de la biere?")
sentencepiece_encode(model, x = txt, type = "subwords")
sentencepiece_encode(model, x = txt, type = "ids")

sentencepiece documentation built on Nov. 13, 2022, 5:05 p.m.