starspace: Interface to Starspace for training a Starspace model

In ruimtehol: Learn Text 'Embeddings' with 'Starspace'

Interface to Starspace for training a Starspace model

Description

Interface to Starspace for training a Starspace model, providing raw access to the C++ functionality.

Usage

starspace(
  model = "textspace.bin",
  file,
  trainMode = 0,
  fileFormat = c("fastText", "labelDoc"),
  label = "__label__",
  dim = 100,
  epoch = 5,
  lr = 0.01,
  loss = c("hinge", "softmax"),
  margin = 0.05,
  similarity = c("cosine", "dot"),
  negSearchLimit = 50,
  adagrad = TRUE,
  ws = 5,
  minCount = 1,
  minCountLabel = 1,
  ngrams = 1,
  thread = 1,
  ...
)

Arguments

model	the full path to where the model file will be saved. Defaults to 'textspace.bin'.
file	the full path to the file on disk which will be used for training.
trainMode	integer with the training mode. Possible values are 0, 1, 2, 3, 4 or 5. Defaults to 0. The use cases are 0: tagspace (classification tasks) and search tasks 1: pagespace & docspace (interest-based or content-based recommendation) 2: articlespace (sentences within document) 3: sentence embeddings and entity similarity 4: multi-relational graphs 5: word embeddings
fileFormat	either one of 'fastText' or 'labelDoc'. See the documentation of StarSpace
label	labels prefix (character string identifying how a label is prefixed, defaults to '__label__')
dim	the size of the embedding vectors (integer, defaults to 100)
epoch	number of epochs (integer, defaults to 5)
lr	learning rate (numeric, defaults to 0.01)
loss	loss function (either 'hinge' or 'softmax')
margin	margin parameter in case of hinge loss (numeric, defaults to 0.05)
similarity	cosine or dot product similarity in cas of hinge loss (character, defaults to 'cosine')
negSearchLimit	number of negatives sampled (integer, defaults to 50)
adagrad	whether to use adagrad in training (logical)
ws	the size of the context window for word level training - only used in trainMode 5 (integer, defaults to 5)
minCount	minimal number of word occurences for being part of the dictionary (integer, defaults to 1 keeping all words)
minCountLabel	minimal number of label occurences for being part of the dictionary (integer, defaults to 1 keeping all labels)
ngrams	max length of word ngram (integer, defaults to 1, using only unigrams)
thread	integer with the number of threads to use. Defaults to 1.
...	arguments passed on to ruimtehol:::textspace. See the details below.

Value

an object of class textspace which is a list with elements

• model: a Rcpp pointer to the model

• args: a list with elements

  1. file: the binary file of the model saved on disk

  2. dim: the dimension of the embedding

  3. data: data-specific Starspace training parameters

  4. param: algorithm-specific Starspace training parameters

  5. dictionary: parameters which define ths dictionary of words and labels in Starspace

  6. options: parameters specific to duration of training, the text preparation and the training batch size

  7. test: parameters specific to model testing

• iter: a list with element epoch, lr, error and error_validation showing the error after each epoch

Note

The function starspace is a tiny wrapper over the internal function ruimtehol:::textspace which allows direct access to the C++ code in order to run Starspace.
The following arguments are available in that functionality when you do the training. Default settings are shown next to the definition. Some of these arguments are directly set in the starspace function, others can be passed on with ... .

Arguments which define how the training is done:

• dim: size of embedding vectors [100]

• epoch: number of epochs [5]

• lr: learning rate [0.01]

• loss: loss function: hinge, softmax [hinge]

• margin: margin parameter in hinge loss. It's only effective if hinge loss is used. [0.05]

• similarity: takes value in [cosine, dot]. Whether to use cosine or dot product as similarity function in hinge loss. It's only effective if hinge loss is used. [cosine]

• negSearchLimit: number of negatives sampled [50]

• maxNegSamples: max number of negatives in a batch update [10]

• p: normalization parameter: normalize sum of embeddings by dividing Size^p [0.5]

• adagrad: whether to use adagrad in training [1]

• ws: only used in trainMode 5, the size of the context window for word level training. [5]

• dropoutLHS: dropout probability for LHS features. [0]

• dropoutRHS: dropout probability for RHS features. [0]

• shareEmb: whether to use the same embedding matrix for LHS and RHS. [1]

• initRandSd: initial values of embeddings are randomly generated from normal distribution with mean=0, standard deviation=initRandSd. [0.001]

Arguments specific to the dictionary of words and labels:

• minCount: minimal number of word occurences [1]

• minCountLabel: minimal number of label occurences [1]

• ngrams: max length of word ngram [1]

• bucket: number of buckets [100000]

• label: labels prefix [__label__]

Arguments which define early stopping or proceeding of model building:

• initModel: if not empty, it loads a previously trained model in -initModel and carry on training.

• validationFile: validation file path

• validationPatience: number of iterations of validation where does not improve before we stop training [10]

• saveEveryEpoch: save intermediate models after each epoch [0]

• saveTempModel: save intermediate models after each epoch with an unique name including epoch number [0]

• maxTrainTime: max train time (secs) [8640000]

Other:

• trainWord: whether to train word level together with other tasks (for multi-tasking). [0]

• wordWeight: if trainWord is true, wordWeight specifies example weight for word level training examples. [0.5]

• useWeight whether input file contains weights [0]

References

https://github.com/facebookresearch

Examples

## Not run: 
data(dekamer, package = "ruimtehol")
x <- strsplit(dekamer$question, "\\W")
x <- lapply(x, FUN = function(x) x[x != ""])
x <- sapply(x, FUN = function(x) paste(x, collapse = " "))

idx <- sample.int(n = nrow(dekamer), size = round(nrow(dekamer) * 0.7))
writeLines(x[idx], con = "traindata.txt")
writeLines(x[-idx], con = "validationdata.txt")

set.seed(123456789)
m <- starspace(file = "traindata.txt", validationFile = "validationdata.txt", 
               trainMode = 5, dim = 10, 
               loss = "softmax", lr = 0.01, ngrams = 2, minCount = 5,
               similarity = "cosine", adagrad = TRUE, ws = 7, epoch = 3,
               maxTrainTime = 10)
str(starspace_dictionary(m))              
wordvectors <- as.matrix(m)
wv <- starspace_embedding(m, 
                          x = c("Nationale Loterij", "migranten", "pensioen"),
                          type = "ngram")
wv
mostsimilar <- embedding_similarity(wordvectors, wv["pensioen", ])
head(sort(mostsimilar[, 1], decreasing = TRUE), 10)
starspace_knn(m, "koning")

## clean up for cran
file.remove(c("traindata.txt", "validationdata.txt"))

## End(Not run)

ruimtehol documentation built on May 29, 2024, 5:26 a.m.