dict_add: Add the dictionary matches to a data.frame

In kasperwelbers/boolydict: Dictionary search with advanced Boolean operators

Add the dictionary matches to a data.frame

Description

Adding dictionary matches to a data.frame is not trivial because one row in the data.frame can have multiple matches. This function deals with this issue by summarizing the matches. You can select which of the columns in the dictionary (dict) to add by providing a name-value pair (...) , where the name is the new column name and the value is any expression using the columns in dict that returns a scalar (i.e. single value).

For example, if dict has a 'sentiment' column, you can say: avg_sentiment = mean(sentiment). Another common use case would be selecting a single value from a categorical variable (e.g., label). You could for instance use label = label[1] to pick the first match. Expressions can use any column in dict, so if the dict has a label and weight column, you could also use: label = label[which.max(weight)]

Usage

dict_add(
  df,
  dict,
  ...,
  by_label = NULL,
  fill = NULL,
  text_col = "text",
  context_col = NULL,
  index_col = NULL,
  mode = c("hits", "terms"),
  keep_longest = TRUE,
  as_ascii = FALSE,
  use_wildcards = TRUE,
  cache = NULL
)

Arguments

df	A data.frame (or tibble, data.table, whatever). The column name specified in the text_col argument (default "text") will be matched to the dictionary
dict	A dictionary data.frame or a character vector. If data.frame, needs to have a column called 'string'. When importing a dictionary (e.g., from quanteda.dictionaries or textdata), please check out import_dictionary.
...	name-value pairs where the name becomes a new column, and the value is an expression for summarizing a column in dict.
by_label	The name of a column in dict with labels. If given, the summary will be calculated per label, with each summary getting it's own column. The name of the column will be [summary_name].[label]
fill	How to fill NA values.
text_col	The column in df with the text to query. Defaults to 'text'.
context_col	Optionally, a column in df with context ids. If used, texts across rows are grouped together so that you can perform Boolean queries across rows. The primary use case is if texts are tokens/words, such as produced by tidytext, udpipe or spacyr.
index_col	Optionally, a column in df with indices for texts within a context. In particular, if texts are tokens, these are the token positions. This is only relevant if not all tokens are used, and we therefore don't know these positions. The indices then need to be provided to correctly match multitoken strings and proximity queries.
mode	There are three modes: "hits" and "terms" and "unique". The "hits" mode prioritizes finding full and unique matches. For example, if we query <climate chang*>~10, then in the text "climate change is changing the world" we'll only find one unique hit for "climate change". Alternatively, in "terms" mode we would match "climate", "change" and "changing". "hits" mode is often what you want for counting occurrences. "terms" mode is especially useful if you are matching a dictionary to tokens, and want to match every token that satisfies the query.
keep_longest	If TRUE, then overlapping in case of overlapping queries strings in unique_hits mode, the query with the most separate terms is kept. For example, in the text "mr. Bob Smith", the query [smith OR "bob smith"] would match "Bob" and "Smith". If keep_longest is FALSE, the match that is used is determined by the order in the query itself. The same query would then match only "Smith".
as_ascii	if TRUE, perform search in ascii. Can be useful if you know text contains things like accents, and these are either used inconsistently or you simply can't be bothered to type them in your queries.
use_wildcards	Set to FALSE if you want to disable wildcards. For instance useful if you have a huge dictionary without wildcards that might have ? or * in emoticons and stuff. Note that you can also always escape wildcards with a double backslash (\? or \*)
cache	Cache the search index to speed up subsequent searches. if cache is a filename or path, the cache will be stored on disk. If cache is a number, the cache will be stored in memory, and the number will indicate the maximum nr of Mb to keep in cache. If NULL, no cache will be kept.

Value

The input df with columns added for the aggregated dictionary matches

Examples

full_text = data.frame(text = c('This is just a simple example', 'Simple is good'))
tokens = data.frame(
   text = c('This','is','just','a','simple','example', 'Simple', 'is','good'),
   doc_id = c(1,1,1,1,1,1,2,2,2))

dict = data.frame(string = c('<this is just>', '<a example>~3'),
                  label = c('FIRST QUERY','SECOND QUERY'),
                  value =  c(1, 100))

## by default just counts
dict_add(full_text, dict, fill=0)

## can count 'by_label', for any categorical column in dict
dict_add(full_text, dict, by_label='label', fill=0)

## can use custom aggregation function
dict_add(full_text, dict, value = sum(value), by_label='label', fill=0)

## works the same for tokens. But if the goal is to annotate tokens, you
## want to set mode to 'terms' so that every match is used
dict_add(tokens, dict, context_col='doc_id', mode='terms')

## you could use by_label to label tokens, but if you just want a unique label
## you could use the custom summarize function to select a label. We'll use a
## different dict to show this.

dict = data.frame(string = c('simple OR good', 'just OR simple'),
                  label = c('FIRST QUERY', 'SECOND QUERY'))

##One option is to just pick the first. Matches are sorted by dict_index, so this
## way queries higher in the dict df get priority.
dict_add(tokens, dict, context_col='doc_id', mode='terms',
               label = label[1])

kasperwelbers/boolydict documentation built on Dec. 25, 2024, 8:56 a.m.