In gederajeg/collogetr: Collocates Retriever and Collocation Association Measure for the Indonesian Leipzig Corpora

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "README-"
)
scaps <- function(concept) {
  paste("<span style='font-variant:small-caps;'>", concept, "</span>", sep="")
}

collogetr

Overview

collogetr performs (i) collocates retrieval (currently from the sentence-based (Indonesian) Leipzig Corpora) and (ii) computation of collocation association-measures. The function colloc_leipzig() is used to retrieve window-span collocates for a set of word forms (viz. the nodeword(s) or keyword(s)).

Two other functions (namely, assoc_prepare() and assoc_prepare_dca()) serve to process the output of colloc_leipzig() into tabular/data frame formats, which then become the input data for computing the association measure between the collocates and the node (as in Stefanowitsch and Gries' [-@stefanowitsch_collostructions_2003] collostructional/collocation analysis) [@hoffmann_collostructional_2013; @stefanowitsch_corpora_2009; see also, @gries_more_2015]. The function assoc_prepare() generates input data for computing Simple Collexeme/Collocational Analysis (SCA), meanwhile assoc_prepare_dca() uses the output of assoc_prepare() to generate input data for computing Distinctive Collexeme/Collocates Analysis (DCA) [@hilpert_distinctive_2006; @gries_extending_2004]. Based on the output of assoc_prepare(), SCA can then be computed using the default collex_fye(), which is based on Fisher-Yates Exact test (other association measures available for computing SCA include (i) collex_llr() for Log-Likelihood Ratio, (ii) collex_MI() for Mutual Information score, (iii) collex_TScore() for T-Score, (iv) collex_chisq() for Chi-Square-based score, and (v) collex_logOR() for Log~10~ Odds Ratio). DCA is computed using collex_fye_dca().

collogetr is built on top of the core packages in the tidyverse.

Installation

Install collogetr from GitHub with devtools:

library(devtools)
install_github("gederajeg/collogetr")

Usages

Load collogetr

library(collogetr)

Citation for collogetr

To cite collogetr in publication, type as follows:

citation("collogetr")

Package data

The package has three data sets for demonstration. The important one is the demo_corpus_leipzig whose documentation can be accessed via ?demo_corpus_leipzig. Another data is a list of Indonesian stopwords (i.e. stopwords) that can be filtered out when performing collocational measure. The last one is leipzig_corpus_path containing character vector of full path to my Leipzig Corpus files in my computer.

Accepted inputs

colloc_leipzig() accepts two types of corpus-input data:

1. A named-list object with character-vector elements of each Leipzig Corpus Files, represented by demo_corpus_leipzig and the format of which is shown below:

lapply(demo_corpus_leipzig[1:2], sample, 2)

1. Full-paths to the Leipzig Corpus plain texts, as in the leipzig_corpus_path.

leipzig_corpus_path[1:2]

In terms of the input strings for the pattern argument, colloc_leipzig() accepts three scenarios:

1. Plain string representing a whole word form, such as "memberikan" 'to give'

2. Regex of a whole word, such as "^memberikan$" 'to give'

3. Regex of a whole word with word boundary character (\\b), such as "\\bmemberikan\\b".

All of these three forms will be used to match the exact word form of the search pattern after the corpus file is tokenised into individual words. That is, input patterns following scenario 1 or 3 will be turned into their exact search pattern represented in scenario 2 (i.e., with the beginning- and end-of-line anchors, hence "^...$"). So user can directly use the input pattern in scenario 2 for the pattern argument. If there are more than one word to be searched, put them into a character vector (e.g., c("^memberi$", "^membawa$")).

Demo

Retrieving the collocates

The codes below show how one may retrieve the collocates for the Indonesian verb mengatakan 'to say sth.'. The function colloc_leipzig() will print out progress messages of the stages onto the console. It generates warning(s) when a search pattern or node word is not found in a corpus file or in all loaded corpus files.

out <- colloc_leipzig(leipzig_corpus_list = demo_corpus_leipzig,
                       pattern = "mengatakan",
                       window = "r",
                       span = 1L,
                       save_interim = FALSE)

In the example above, the collocates are restricted to those occurring one word (i.e. span = 1L) to the right (window = "r") of mengatakan 'to say'. The "r" character in window stands for right-side collocates ("l" for left-side collocates and "b" for both right- and left-side collocates). The span argument requires integer (i.e., a whole number) to indicate the range of words covered in the specified window. The pattern argument requires one or more exact word forms; if more than one, put into a character vector (e.g., c("mengatakan", "menjanjikan")).

The save_interim is FALSE means that no output is saved into the computer, but in the console (i.e., in the out object). If save_interim = TRUE, the function will save the outputs into the files in the computer. colloc_leipzig() has specified the default file names for the outputs via these arguments: (i) freqlist_output_file, (ii) colloc_output_file, (iii) corpussize_output_file, and (iv) search_pattern_output_file. It is recommended that the output filenames are stored as a character vector. See Examples "(2)" in the documentation of colloc_leipzig() for a call when save_interim = TRUE.

Exploring the output of colloc_leipzig().

The output of colloc_leipzig() is a list of r length(out) elements:

1. colloc_df; a table/tibble of raw collocates data with columns for:
   1. corpus names
   2. sentence id in which the collocates and the node word(s) are found
   3. the collocates (column w)
   4. the span information (e.g., "r1" for one-word, right-side collocates)
   5. the node word
   6. the text/sentence match in which the collocates and the node are found
2. freqlist_df; a table/tibble of word-frequency list in the loaded corpus
3. corpussize_df; a table/tibble of total word-tokens in the loaded corpus
4. pattern; a character vector of the search pattern/node

str(out)

The freqlist_df and corpussize_df are important for performing the collocational strength measure for the search pattern with the collocates.

Preparing input data for Simple Collexeme/Collocational Analysis (SCA).

First we need to call assoc_prepare() for generating the data SCA. The demo illustrates it with in-console output of colloc_leipzig(). See the Examples "2.2" in the documentation for assoc_prepare() for handling saved outputs (?assoc_prepare()).

assoc_tb <- assoc_prepare(colloc_out = out, 
                          window_span = "r1",
                          per_corpus = FALSE, # combine all data across corpus
                          stopword_list = collogetr::stopwords,
                          float_digits = 3L)

Inspect the output of assoc_prepare():

head(assoc_tb)

The assoc_prepare() and collex_fye() functions are designed following the tidy principle so that the association/collocation measure is performed in a row-wise fashion, benefiting from the combination of nested column [cf., @wickham_r_2017, p. 409] for the input-data (using tidyr::nest()) and purrr's map_* function. assoc_prepare() includes calculating the expected co-occurrence frequencies between the collocates/collexemes and the node word/construction.

The column data in assoc_tb above consists of nested tibble/table as a list. Each contains required data for performing association measure for each of the collocates in column w [@gries_more_2015; @gries_50-something_2013; @stefanowitsch_corpora_2009; @stefanowitsch_collostructions_2003]. This nested column can be inspected as follows (for the first row, namely for the word pihaknya 'the party').

# get the tibble in the `data` column for the first row
assoc_tb$data[[1]]

Column a indicates the co-occurrence frequency between the node word and the collocates column w, meanwhile a_exp indicates the expected co-occurrence frequency between them. The n_w_in_corp represents the total token/occurrence frequency of a given collocate. The n_pattern stores the total token/occurrence frequency of the node word in the corpus. Column b, c, and d are required for the association measure that is essentially based on 2-by-2 crosstabulation table. The assoc column indicates whether the value in a is higher than that in a_exp, thus indicating attraction or positive association between the node word and the collocate. The reverse is repulsion or negative association when the value in a is less/lower than that in a_exp.

Simple Collexeme/Collocates Analysis (SCA)

As in the Collostructional Analysis [@stefanowitsch_collostructions_2003], collex_fye() uses one-tailed Fisher-Yates Exact test whose p-~FisherExact~value is log-transformed to the base of 10 to indicate the collostruction strength between the collocates and the node word [@gries_converging_2005]. collex_fye() simultaneously performs two uni-directional measures of Delta P [@gries_50-something_2013; @gries_more_2015, p. 524]. One of these shows the extent to which the presence of the node-word cues the presence of the collocates/collexemes; the other one determines the extent to which the collocates/collexemes cues the presence of the node-word.

Here is the codes to perform the SCA using collex_fye():

# perform FYE test for Collexeme Analysis
am_fye <- collex_fye(df = assoc_tb, collstr_digit = 3)

Now we can retrieve the top-10 most strongly attracted collocates to mengatakan 'to say sth.'. The association strength is shown in the collstr column, which stands for collostruction strength. The higher, the stronger the association.

# get the top-10 most strongly attracted collocates
dplyr::top_n(am_fye, 10, collstr)

Column a contains the co-occurrence frequency of the collocates (w) with the node as its R1 collocates in the demo corpus. p_fye shows the one-tailed p~FisherExact~-value.

Distinctive Collexeme/Collocate Analysis (DCA)

The idea of distinctive collexemes/collocates is to contrast two functionally/semantically similar constructions or words in terms of the collocates that are (significantly) more frequent for one of the two contrasted constructions/words [@hilpert_distinctive_2006; see @gries_extending_2004]. colloc_leipzig() can be used to retrieve collocates of two functionally/semantically similar words by specifying the pattern argument with two character vectors of words.

The following example use one of the Leipzig corpus files (not included in the package but can be downloaded from the Leipzig Corpora webpage for free), namely the "ind_mixed_2012_1M-sentences". The aim is to contrast collocational preferences of two deadjectival transitive verbs based on the root kuat 'strong' framed within two causative morphological schemas: one with per-+ADJ and the other with ADJ+-kan. Theoretically, the per- schema indicates that the direct object of the verb is caused to have more of the characteristic indicated by the adjectival root, meanwhile the -kan schema indicates that the direct object is caused to have the characteristic indicated by the root (that is not previously had). The focus here is on the R1 collocates of the verbs (i.e. one word immediately to the right of the verbs in the sentences).

my_leipzig_path <- collogetr::leipzig_corpus_path[1]
dca_coll <- collogetr::colloc_leipzig(leipzig_path = my_leipzig_path, 
                                     pattern = c("memperkuat", "menguatkan"),
                                     window = "r",
                                     span = 1,
                                     save_interim = FALSE)

Then, we prepare the output into the format required for performing DCA with collex_fye_dca().

dca_coll <- collogetr::dca_coll

assoc_tb <- assoc_prepare(colloc_out = dca_coll,
                          window_span = "r1",
                          per_corpus = FALSE,
                          stopword_list = collogetr::stopwords,
                          float_digits = 3L)

# prepare the dca input table
dca_tb <- assoc_prepare_dca(assoc_tb)

Compute DCA for the two verbs and view the results snippet.

dca_res <- collex_fye_dca(dca_tb)
head(dca_res, 10)

The package also includes a function called dca_top_collex() to retrieve the top-n distinctive collocates for one of the two contrasted words. The dist_for argument can be specified by either the character vector of the name of the contrasted words, or the character IDs of the constructions/words (e.g., ..., dist_for = "a", ... or ..., dist_for = "A", ... for construction/word appearing in the second column from the output of collex_fye_dca(); ..., dist_for = "b", ... or ..., dist_for = "B", ... for construction/word appearing in the third column).

# retrieve distinctive collocates for Construction A (i.e., memperkuat)
dist_for_a <- dca_top_collex(dca_res, dist_for = "memperkuat", top_n = 10)
head(dist_for_a)

The codes below retrieve the distinctive collocates for menguatkan 'to strengthen' or Construction B.

# retrieve distinctive collocates for Construction B (i.e., menguatkan)
dist_for_b <- dca_top_collex(dca_res, dist_for = "menguatkan", top_n = 10)
head(dist_for_b)

Session info

devtools::session_info()

References

gederajeg/collogetr documentation built on April 16, 2020, 11:58 a.m.