embedding_postprocessor | R Documentation |
This function (optionally) adds to the word embeddings additional embeddings for token type and position.
embedding_postprocessor( input_tensor, use_token_type = FALSE, token_type_ids = NULL, token_type_vocab_size = 16L, token_type_embedding_name = "token_type_embeddings", use_position_embeddings = TRUE, position_embedding_name = "position_embeddings", initializer_range = 0.02, max_position_embeddings = 512L, dropout_prob = 0.1 )
input_tensor |
Float Tensor of shape |
use_token_type |
Logical; whether to add embeddings for
|
token_type_ids |
(optional) Integer Tensor of shape |
token_type_vocab_size |
Integer; the vocabulary size of
|
token_type_embedding_name |
Character; the name of the embedding table variable for token type ids. |
use_position_embeddings |
Logical; whether to add position embeddings for the position of each token in the sequence. |
position_embedding_name |
Character; the name of the embedding table variable for positional embeddings. |
initializer_range |
Numeric; range of the weight initialization. |
max_position_embeddings |
Integer; maximum sequence length that might ever be used with this model. This can be longer than the sequence length of input_tensor, but cannot be shorter. |
dropout_prob |
Numeric; dropout probability applied to the final output tensor. |
See figure 2 in the BERT paper:
https://arxiv.org/pdf/1810.04805.pdf
Both type and position embeddings are learned model variables. Note that token "type" is essentially a sentence identifier, indicating which sentence (or, more generally, piece of text) the token belongs to.
Float Tensor with same shape as input_tensor
.
## Not run: batch_size <- 10 seq_length <- 512 embedding_size <- 200 with(tensorflow::tf$variable_scope("examples", reuse = tensorflow::tf$AUTO_REUSE ), { input_tensor <- tensorflow::tf$get_variable( "input", dtype = "float", shape = tensorflow::shape(batch_size, seq_length, embedding_size) ) token_type_ids <- tensorflow::tf$get_variable( "ids", dtype = "int32", shape = tensorflow::shape(batch_size, seq_length) ) }) embedding_postprocessor(input_tensor, use_token_type = TRUE, token_type_ids = token_type_ids ) ## End(Not run)
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.