Description Usage Arguments Value Warning Author(s) References See Also Examples
View source: R/generate_stoplist.R
Generate a vector of stop words in one or several languages.
generate_stoplist(language = NULL, output_form = 1)
single string or a character vector.
The function comes with three output options.
Option '1' outputs a character vector of unique word forms.
Option '2' outputs a named character vector of word forms. The names denote 'stop classes' roughly corresponding to parts of speech. Note that, in this output, the word forms are not unique. For instance, in English stopwords, *that* would occur as a subordinating conjunction as well as as a pronoun.
Option '3' (the default) outputs a data frame, where each row represents a combination of language (columns 'lang_name' and 'lang_id'), word form and word lemma (columns 'form' and 'lemma'), and several other columns explained below.
All outputs are encoded in UTF-8.
The function stops when no language is selected.
The stop classes (pre-defined linguistic filters) are not mutually exclusive. Their overlap varies among languages.
The stoplists are fully data-driven. We have set a threshold of 3 occurrences of a combination of language, form, lemma, and upos to remove obvious noise, but some noise is bound to have come through anyway. It is mainly foreign words that were given a regular upos tag (e.g. the English "and" has sneaked in among the German coordinating conjunctions). Another known case is the
contraction stop class in English, which, among well-suited instances such as *ain't* includes uses of the so-called Saxonic genitive (e.g. *world's*).
Many languages are represented by balanced and large corpora of standard written texts, but some are not; e.g. based mainly on a Bible translation or Wikipedia. Hence also their stopwords can be biased.
Silvie Cinková, Maciej Eder
The underlying data frame 'multilingual_stoplist' is based on the official release of Version 2.8 of Universal Dependencies.
Zeman, Daniel; et al., 2021, Universal Dependencies 2.8.1, LINDAT/CLARIAH-CZ digital library at the Institute of Formal and Applied Linguistics (ÚFAL), Faculty of Mathematics and Physics, Charles University, http://hdl.handle.net/11234/1-3687.
1 2 3 4 5
generate_stoplist(language = "English", output_form = 1) generate_stoplist(language = "English", output_form = 2) generate_stoplist(language = "English", output_form = 3)
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.