knitr::opts_chunk$set( collapse = TRUE, comment = "#>" )
In this tutorial, we'll look at how to create tfidf feature matrix in R in two simple steps with superml. Superml borrows speed gains using parallel computation and optimised functions from data.table R package. Tfidf matrix can be used to as features for a machine learning model. Also, we can use tdidf features as an embedding to represent the given texts.
You can install latest cran version using (recommended):
You can install the developmemt version directly from github using:
For machine learning, superml is based on the existing R packages. Hence, while installing the package, we don't install all the dependencies. However, while training any model, superml will automatically install the package if its not found. Still, if you want to install all dependencies at once, you can simply do:
First, we'll create a sample data. Feel free to run it alongside in your laptop and check the results.
library(superml) # should be a vector of texts sents <- c('i am going home and home', 'where are you going.? //// ', 'how does it work', 'transform your work and go work again', 'home is where you go from to work') # generate more sentences n <- 10 sents <- rep(sents, n) length(sents)
For sample, we've generated 50 documents. Let's create the features now. For ease, superml uses the similar API layout as python scikit-learn.
# initialise the class tfv <- TfIdfVectorizer$new(max_features = 10, remove_stopwords = FALSE) # generate the matrix tf_mat <- tfv$fit_transform(sents) head(tf_mat, 3)
remove_stopwords = FALSEdefaults to
TRUE. We set it to
FALSEsince most of the words in our dummy
max_features = 10select the top 10 features (tokens) based on frequency.
norm = TRUEis set by default.
Now, let's generate the matrix using its
# initialise the class tfv <- TfIdfVectorizer$new(min_df = 0.4, remove_stopwords = FALSE, ngram_range = c(1, 3)) # generate the matrix tf_mat <- tfv$fit_transform(sents) head(tf_mat, 3)
ngram_range = c(1,3)set the lower and higher range respectively of the resulting ngram tokens.
min_df = 0.4says to keep the tokens which occurs in atleast 40% & above of the documents.
In order to use Tfidf Vectorizer for a machine learning model, sometimes it gets confusing as to which method
transform should be used to generate tfidf features for the given data. Here's a way to do:
library(data.table) library(superml) # use sents from above sents <- c('i am going home and home', 'where are you going.? //// ', 'how does it work', 'transform your work and go work again', 'home is where you go from to work', 'how does it work') # create dummy data train <- data.table(text = sents, target = rep(c(0,1), 3)) test <- data.table(text = sample(sents), target = rep(c(0,1), 3))
Let's see how the data looks like:
Now, we generate features for train-test data:
# initialise the class tfv <- TfIdfVectorizer$new(min_df = 0.3, remove_stopwords = FALSE, ngram_range = c(1,3)) # we fit on train data tfv$fit(train$text) train_tf_features <- tfv$transform(train$text) test_tf_features <- tfv$transform(test$text) dim(train_tf_features) dim(test_tf_features)
We generate 15 features for each of the given data. Let's see how they look:
Finally, to train a machine learning model on this, you can simply do:
# ensure the input to classifier is a data.table or data.frame object x_train <- data.table(cbind(train_tf_features, target = train$target)) x_test <- data.table(test_tf_features) xgb <- XGBTrainer$new(n_estimators = 10, objective = "binary:logistic") xgb$fit(x_train, "target") predictions <- xgb$predict(x_test) predictions
In this tutorial, we discussed how to use superml's tfidfvectorizer to create tfidf matrix and train a machine learning model on it.
Any scripts or data that you put into this service are public.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.