In dfalbel/ptstem: Stemming Algorithms for the Portuguese Language

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "README-"
)

ptstem

Stemming Algorithms for the Portuguese Language

This packages wraps 3 stemming algorithms for the portuguese language available in R. It unifies the API for the stemmers and provides easy stemming completion.

Installing

You can install directly from Github using:

devtools::install_github("dfalbel/ptstem")

or from CRAN using:

install.packages("ptstem")

Using

Consider the following text, extracted from Stemming in Wikipedia

text <- "Em morfologia linguística e recuperação de informação a stemização (do inglês, stemming) é
o processo de reduzir palavras flexionadas (ou às vezes derivadas) ao seu tronco (stem), base ou
raiz, geralmente uma forma da palavra escrita. O tronco não precisa ser idêntico à raiz morfológica
da palavra; ele geralmente é suficiente que palavras relacionadas sejam mapeadas para o mesmo
tronco, mesmo se este tronco não for ele próprio uma raiz válida. O estudo de algoritmos para
stemização tem sido realizado em ciência da computação desde a década de 60. Vários motores de
buscas tratam palavras com o mesmo tronco como sinônimos como um tipo de expansão de consulta, em
um processo de combinação."

This will use the rslp algorithm to stem the text.

library(ptstem)
ptstem(text, algorithm = "rslp", complete = FALSE)

You can complete stemmed words using the argument complete = T.

ptstem(text, algorithm = "rslp", complete = TRUE)

The other implemented algorithms are:

• hunspell: the same algorithm used in OpenOffice corrector. (available via hunspell package)
• porter: available via SnowballC package.

You can stem using those algorithms by changing the algorithm argument in ptstem function.

library(ptstem)
ptstem(text, algorithm = "hunspell")
ptstem(text, algorithm = "porter")

Performance

The goal of stemming algorithms is to group related words and to separate unrelated words. With this in mind, you can talk about two kinds of possible errors when stemming:

• Understemming: Related words were not grouped because you didn't stem enought.
• Overstemming: Unrelated words were grouped because you removed a large part of the word when stemming.

To measure these errors the function performance was implemented. It returns a data.frame with 3 columns. The name of the stemmer and 2 metrics:

• UI: the undersampling index. It's the proportion of related words that were not grouped.
• OI: the overstemming index. It's the proportion of unrelated words that were grouped.

Remember that OI is 0 if you don't stem. So I think the true objective of a stemming algorithm is to reduce UI without augmenting OI too much.

ptstem package provides a dataset of grouped words for the portuguese language (found in this link). It's in this dataset that performance function calculates the metrics described above.

See results:

performance()

This is not the only approach for measuring performance of the those algorithms. The article Assessing the impact of Stemming Accuracy on Information Retrieval – A multilingual perspective describes various ways to analyse stemming performance.

dfalbel/ptstem documentation built on May 16, 2020, 11:45 a.m.