macmillancontentscience/morphemepiece: Morpheme Tokenization
Tokenize text into morphemes. The morphemepiece algorithm uses a lookup table to determine the morpheme breakdown of words, and falls back on a modified wordpiece tokenization algorithm for words not found in the lookup table.
Getting started
Browse package contents
Package details |
|
|---|---|
| Maintainer | |
| License | Apache License (>= 2) |
| Version | 1.2.3 |
| URL | https://github.com/macmillancontentscience/morphemepiece |
| Package repository | View on GitHub |
| Installation |
Install the latest version of this package by entering the following in R:
|
