simplify_bert_token_list: Simplify Token List to Matrix

View source: R/tokenize.R

simplify_bert_token_listR Documentation

Simplify Token List to Matrix

Description

BERT-like models expect a matrix of tokens for each example. This function converts a list of equal-length vectors (such as a padded list of tokens) into such a matrix.

Usage

simplify_bert_token_list(token_list)

Arguments

token_list

A list of vectors. Each vector should have the same length.

Value

A matrix of tokens. Rows are text sequences, and columns are tokens.

Examples

simplify_bert_token_list(
  list(
    1:5,
    2:6,
    3:7
  )
)

macmillancontentscience/torchtransformers documentation built on Aug. 6, 2023, 5:35 a.m.