In paithiov909/kagomer: An R Wrapper for 'vibrato'

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%"
)
pkgload::load_all(export_all = FALSE)

vibrrt

An R wrapper for vibrato: Viterbi-based accelerated tokenizer.

Installation

To install from source package, the Rust toolchain is required.

install.packages("vibrrt", repos = c("https://paithiov909.r-universe.dev", "https://cloud.r-project.org"))

Usage

You can download the model files from ryan-minato/vibrato-models using hfhub package.

sample_text <- jsonlite::read_json(
  "https://paithiov909.r-universe.dev/gibasa/data/ginga/json",
  simplifyVector = TRUE
)

# withr::with_envvar(c(HUGGINGFACE_HUB_CACHE = tempdir()), {
ipadic <- hfhub::hub_download("ryan-minato/vibrato-models", "ipadic-mecab-2_7_0/system.dic")
# })

vibrrt::tokenize(
  sample_text[5:8],
  tagger = vibrrt::create_tagger(ipadic)
)

Versioning

This package is versioned by copying the version number of vibrato, where the first three digits represent that version number and the fourth digit (if any) represents the patch release for this package.

paithiov909/kagomer documentation built on June 12, 2025, 7:44 a.m.