dot-validate_tokenizer_metadata: Choose Tokenizer Metadata

In macmillancontentscience/torchtransformers: Transformer Models in Torch

Choose Tokenizer Metadata

Description

Choose Tokenizer Metadata

Usage

.validate_tokenizer_metadata(
  bert_type = NULL,
  tokenizer_scheme = NULL,
  n_tokens = NULL,
  allow_null = TRUE
)

Arguments

bert_type	A bert_type from available_berts() to use to choose the other properties. If bert_type and n_tokens are set, they overrule this setting.
tokenizer_scheme	A character scalar that indicates vocabulary + tokenizer.
n_tokens	An integer scalar indicating the number of tokens in the output.
allow_null	A logical scalar indicating whether the scheme can be NULL.

Value

A list with elements tokenizer_scheme and n_tokens.

macmillancontentscience/torchtransformers documentation built on Aug. 6, 2023, 5:35 a.m.