ntoken: count the number of tokens or types

Description Usage Arguments Details Value Note Examples

Description

Get the count of tokens (total features) or types (unique tokens) in a text, corpus, or dfm.

Usage

1
2
3
ntoken(x, ...)

ntype(x, ...)

Arguments

x

a quanteda object: a character, corpus, or tokens object

...

additional arguments passed to tokens

Details

The precise definition of "tokens" for objects not yet tokenized (e.g. character or corpus objects can be controlled through optional arguments passed to tokens through ....

For dfm objects, ntype will only return the count of features that occur more than zero times in the dfm.

Value

count of the total tokens or types

Note

Due to differences between raw text tokens and features that have been defined for a dfm, the counts be different for dfm objects and the texts from which the dfm was generated. Because the method tokenizes the text in order to count the tokens, your results will depend on the options passed through to tokenize

Examples

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
# simple example
txt <- c(text1 = "This is a sentence, this.", text2 = "A word. Repeated repeated.")
ntoken(txt)
ntype(txt)
ntoken(char_tolower(txt))  # same
ntype(char_tolower(txt))   # fewer types
ntoken(char_tolower(txt), removePunct = TRUE)
ntype(char_tolower(txt), removePunct = TRUE)

# with some real texts
ntoken(corpus_subset(data_corpus_inaugural, Year<1806), removePunct = TRUE)
ntype(corpus_subset(data_corpus_inaugural, Year<1806), removePunct = TRUE)
ntoken(dfm(corpus_subset(data_corpus_inaugural, Year<1800)))
ntype(dfm(corpus_subset(data_corpus_inaugural, Year<1800)))


Search within the quanteda package
Search all R packages, documentation and source code

Questions? Problems? Suggestions? or email at ian@mutexlabs.com.

Please suggest features or report bugs with the GitHub issue tracker.

All documentation is copyright its authors; we didn't write any of that.