dot-combine_segments: Combine a pair of segments

In macmillancontentscience/torchtransformers: Transformer Models in Torch

Combine a pair of segments

Description

Combine a pair of segments

Usage

.combine_segments(..., cls_index, sep_index, pad_index, n_tokens)

Arguments

...	Two lists of tokenized segments. In the future we will support any number of lists.
cls_index, sep_index, pad_index	Named integer vectors for the special characters.
n_tokens	Integer scalar; the number of tokens expected for each example.

Value

A list with two components: token_ids and token_type_ids.

macmillancontentscience/torchtransformers documentation built on Aug. 6, 2023, 5:35 a.m.