lma_dtm: Document-Term Matrix Creation

In miserman/lingmatch: Linguistic Matching and Accommodation

Document-Term Matrix Creation

Description

Creates a document-term matrix (dtm) from a set of texts.

Usage

lma_dtm(text, exclude = NULL, context = NULL, replace.special = FALSE,
  numbers = FALSE, punct = FALSE, urls = TRUE, emojis = FALSE,
  to.lower = TRUE, word.break = " +", dc.min = 0, dc.max = Inf,
  sparse = TRUE, tokens.only = FALSE)

Arguments

text	Texts to be processed. This can be a vector (such as a column in a data frame) or list. When a list, these can be in the form returned with tokens.only = TRUE, or a list with named vectors, where names are tokens and values are frequencies or the like.
exclude	A character vector of words to be excluded. If exclude is a single string matching 'function', lma_dict(1:9) will be used.
context	A character vector used to reformat text based on look- ahead/behind. For example, you might attempt to disambiguate like by reformatting certain likes (e.g., context = c('(i) like*', '(you) like*', '(do) like'), where words in parentheses are the context for the target word, and asterisks denote partial matching). This would be converted to regular expression (i.e., '(? <= i) like\\b') which, if matched, would be replaced with a coded version of the word (e.g., "Hey, i like that!" would become "Hey, i i-like that!"). This would probably only be useful for categorization, where a dictionary would only include one or another version of a word (e.g., the LIWC 2015 dictionary does something like this with like, and LIWC 2007 did something like this with kind (of), both to try and clean up the posemo category).
replace.special	Logical: if TRUE, special characters are replaced with regular equivalents using the lma_dict special function.
numbers	Logical: if TRUE, numbers are preserved.
punct	Logical: if TRUE, punctuation is preserved.
urls	Logical: if FALSE, attempts to replace all urls with "repurl".
emojis	Logical: if TRUE, attempts to replace emojis (e.g., ":(" would be replaced with "repfrown").
to.lower	Logical: if FALSE, words with different capitalization are treated as different terms.
word.break	A regular expression string determining the way words are split. Default is ' +' which breaks words at one or more blank spaces. You may also like to break by dashes or slashes ('[ /-]+'), depending on the text.
dc.min	Numeric: excludes terms appearing in the set number or fewer documents. Default is 0 (no limit).
dc.max	Numeric: excludes terms appearing in the set number or more. Default is Inf (no limit).
sparse	Logical: if FALSE, a regular dense matrix is returned.
tokens.only	Logical: if TRUE, returns a list rather than a matrix, with these entries: tokens A vector of indices with terms as names. frequencies A vector of counts with terms as names. WC A vector of term counts for each document. indices A list with a vector of token indices for each document.

Value

A sparse matrix (or regular matrix if sparse = FALSE), with a row per text, and column per term, or a list if tokens.only = TRUE. Includes an attribute with options (opts), and attributes with word count (WC) and column sums (colsums) if tokens.only = FALSE.

Note

This is a relatively simple way to make a dtm. To calculate the (more or less) standard forms of LSM and LSS, a somewhat raw dtm should be fine, because both processes essentially use dictionaries (obviating stemming) and weighting or categorization (largely obviating 'stop word' removal). The exact effect of additional processing will depend on the dictionary/semantic space and weighting scheme used (particularly for LSA). This function also does some processing which may matter if you plan on categorizing with categories that have terms with look- ahead/behind assertions (like LIWC dictionaries). Otherwise, other methods may be faster, more memory efficient, and/or more featureful.

Examples

text <- c(
  "Why, hello there! How are you this evening?",
  "I am well, thank you for your inquiry!",
  "You are a most good at social interactions person!",
  "Why, thank you! You're not all bad yourself!"
)

lma_dtm(text)

# return tokens only
(tokens <- lma_dtm(text, tokens.only = TRUE))

## convert those to a regular DTM
lma_dtm(tokens)

# convert a list-representation to a sparse matrix
lma_dtm(list(
  doc1 = c(why = 1, hello = 1, there = 1),
  doc2 = c(i = 1, am = 1, well = 1)
))

miserman/lingmatch documentation built on Jan. 19, 2024, 4:44 p.m.