textmodel_evaluate: Model evaluation function

In quanteda/quanteda.classifiers: Models for supervised text classification

Model evaluation function

Description

Designed to streamline the parameter tuning and evaluation process. Users chose a function to evaluate and include parameter values as a list. If multiple parameter values are provided, the function will perform a grid search by estimating a separate model for every combination of parameters.

Usage

textmodel_evaluate(
  x,
  y,
  model,
  fun = "f1_score",
  k = 5,
  parameters = list(),
  seed = as.numeric(Sys.time()),
  time = TRUE,
  by_class = FALSE
)

Arguments

x	the dfm or tokens object on which the model will be fit. Does not need to contain only the training documents.
y	vector of training labels associated with each document identified in train. (These will be converted to factors if not already factors.)
model	the name of the machine learning function that will be evaluated
fun	the name of the function that will be used to evaluate the machine learning model. Can take the values "accuracy", "precision", "recall", or "f1_score"
k	number of folds
parameters	model hyperparameters
seed	a seed that can allow for replication of k training data splits. If seed is not provided a seed is chosen based on the current time.
time	a logical parameter that determines whether output will include training time (in seconds) of model
by_class	estimates a separate value of provided evaluation function for every class of the true vector

Examples

# evaluate immigration classification performance
## Not run: 
dfmat <- dfm(data_corpus_manifestosentsUK)
codes <- docvars(data_corpus_manifestosentsUK, "crowd_immigration_label")
evaluation <- textmodel_evaluate(dfmat, codes, k = 3,
                                 model = "textmodel_mlp", fun = "f1_score",
                                 parameters = list(epochs = c(3, 4)))
head(evaluation)
aggregate(evaluation, by = list(evaluation$cost), FUN = "mean")

## End(Not run)

quanteda/quanteda.classifiers documentation built on Oct. 20, 2023, 6:53 a.m.