vectorize.docs: Document vectorization

In fdm2id: Data Mining and R Programming for Beginners

Document vectorization

Description

Vectorize a corpus of documents.

Usage

vectorize.docs(
  vectorizer = NULL,
  corpus = NULL,
  lang = "en",
  stopwords = lang,
  excludewords = NULL,
  ngram = 1,
  mincount = 10,
  minphrasecount = NULL,
  transform = c("tfidf", "lsa", "l1", "none"),
  latentdim = 50,
  returndata = TRUE,
  removesinglechars = TRUE,
  sparse = FALSE,
  ...
)

Arguments

vectorizer	The document vectorizer.
corpus	The corpus of documents (a vector of characters).
lang	The language of the documents (NULL if no stemming).
stopwords	The language whose stop words are removed ("en", ...), or NULL to keep them. A list of words of your own goes to excludewords.
excludewords	An optional custom vector of additional words to exclude from the vocabulary (e.g. corpus-specific stop words), on top of (or instead of) the language stopwords given through stopwords.
ngram	maximum size of n-grams.
mincount	Minimum word count to be considered as frequent.
minphrasecount	Minimum collocation of words count to be considered as frequent.
transform	Transformation (TF-IDF, LSA, L1 normanization, or nothing).
latentdim	Number of latent dimensions if LSA transformation is performed.
returndata	If true, the vectorized documents are returned. If false, a "vectorizer" is returned.
removesinglechars	Whether single-character tokens are removed during cleanup.
sparse	Whether the document-term matrix is returned as a sparse matrix (dgCMatrix) rather than as an ordinary data.frame. A document-term matrix is mostly zeros, and storing them all takes about 4.5 GB for 20000 documents and 30000 terms, so anything but a small corpus needs TRUE. Every method of the package accepts either.
...	Other parameters.

Value

The vectorized documents, as a data.frame or, if sparse is TRUE, as a sparse matrix.

See Also

query.docs, stopwords, vectorizers

Examples

require (text2vec)
# A small subset of movie_review is used here so this example runs quickly.
data ("movie_review")
reviews = movie_review [1:300, ]
# Clustering
docs = vectorize.docs (corpus = reviews$review, transform = "tfidf")
km = KMEANS (docs [sample (nrow (docs), 50), ], k = 10)
# Classification
d = reviews [, 2:3]
d [, 1] = factor (d [, 1])
d = splitdata (d, 1)
vectorizer = vectorize.docs (corpus = d$train.x,
                             returndata = FALSE, mincount = 10)
train = vectorize.docs (corpus = d$train.x, vectorizer = vectorizer)
test = vectorize.docs (corpus = d$test.x, vectorizer = vectorizer)
model = NB (as.matrix (train), d$train.y)
pred = predict (model, as.matrix (test))
evaluation (pred, d$test.y)

fdm2id documentation built on Aug. 28, 2026, 9:07 a.m.