piecemaker-package: piecemaker: Tools for Preparing Text for Tokenizers

piecemaker-packageR Documentation

piecemaker: Tools for Preparing Text for Tokenizers

Description

Tokenizers break text into pieces that are more usable by machine learning models. Many tokenizers share some preparation steps. This package provides those shared steps, along with a simple tokenizer.

Author(s)

Maintainer: Jon Harmon jonthegeek@gmail.com (ORCID)

Authors:

Other contributors:

  • Bedford Freeman & Worth Pub Grp LLC DBA Macmillan Learning [copyright holder]

See Also

Useful links:


piecemaker documentation built on June 7, 2023, 5:55 p.m.