create_model: Create a classification model

In jonathanbratt/RBERT: R Implementation of BERT

Create a classification model

Description

Takes the output layer from a BERT "spine" and appends a classifier layer to it. The output taken from BERT is the pooled first token layers (may want to modify the code to use token-level outputs). The classifier is essentially a single dense layer with softmax.

Usage

create_model(
  bert_config,
  is_training,
  input_ids,
  input_mask,
  segment_ids,
  labels,
  num_labels
)

Arguments

bert_config	BertConfig instance.
is_training	Logical; TRUE for training model, FALSE for eval model. Controls whether dropout will be applied.
input_ids	Integer Tensor of shape [batch_size, seq_length].
input_mask	Integer Tensor of shape [batch_size, seq_length].
segment_ids	Integer Tensor of shape [batch_size, seq_length].
labels	Integer Tensor; represents training example classification labels. Length = batch size.
num_labels	Integer; number of classification labels.

Value

A list including the loss (for training) and the model output (softmax probabilities, log probs).

Examples

## Not run: 
with(tensorflow::tf$variable_scope("examples",
  reuse = tensorflow::tf$AUTO_REUSE
), {
  input_ids <- tensorflow::tf$constant(list(
    list(31L, 51L, 99L),
    list(15L, 5L, 0L)
  ))

  input_mask <- tensorflow::tf$constant(list(
    list(1L, 1L, 1L),
    list(1L, 1L, 0L)
  ))
  token_type_ids <- tensorflow::tf$constant(list(
    list(0L, 0L, 1L),
    list(0L, 2L, 0L)
  ))
  config <- BertConfig(
    vocab_size = 32000L,
    hidden_size = 768L,
    num_hidden_layers = 8L,
    num_attention_heads = 12L,
    intermediate_size = 1024L
  )
  class_model <- create_model(
    bert_config = config,
    is_training = TRUE,
    input_ids = input_ids,
    input_mask = input_mask,
    segment_ids = token_type_ids,
    labels = c(1L, 2L),
    num_labels = 2L,
  )
})

## End(Not run)

jonathanbratt/RBERT documentation built on Jan. 26, 2023, 4:15 p.m.