dot-finalize_bert_tokens: Clean and Return BERT Tokens

In macmillancontentscience/torchtransformers: Transformer Models in Torch

Clean and Return BERT Tokens

Description

Clean and Return BERT Tokens

Usage

.finalize_bert_tokens(tokenized_text, token_types, increment_index)

Arguments

tokenized_text	A list of integer vectors of token ids.
token_types	A list of integer vectors indicating which segment tokens belong to.
increment_index	Logical; if TRUE, add 1L to all token ids to convert from the Python-inspired 0-indexed standard to the torch 1-indexed standard.

Value

An object of class "bert_tokens", which is a list containing a matrix of token ids, a matrix of token type ids, and a matrix of token names.

macmillancontentscience/torchtransformers documentation built on Aug. 6, 2023, 5:35 a.m.