create_oolong: Generate an oolong test
In oolong: Create Validation Tests for Automated Content Analysis
create_oolong R Documentation
Generate an oolong test
Description
create_oolong generates an oolong test object that can either be used for validating a topic model or for creating ground truth (gold standard) of a text corpus. wi (word intrusion test), ti (topic intrusion test), witi (word and topic intrusion tests), wsi (word set intrusion test) and gs are handy wrappers to create_oolong. It is recommended to use these wrappers instead of create_oolong.
Usage
create_oolong(
input_model = NULL,
input_corpus = NULL,
n_top_terms = 5,
bottom_terms_percentile = 0.6,
exact_n = NULL,
frac = 0.01,
n_top_topics = 3,
n_topiclabel_words = 8,
use_frex_words = FALSE,
frexweight = 0.5,
input_dfm = NULL,
construct = "positive",
btm_dataframe = NULL,
n_correct_ws = 3,
wsi_n_top_terms = 20,
userid = NA,
type = "witi",
lambda = 1,
difficulty = NULL
)
wi(
input_model = NULL,
userid = NA,
n_top_terms = 5,
bottom_terms_percentile = 0.6,
frexweight = 0.5,
use_frex_words = FALSE,
lambda = 1,
difficulty = NULL
)
witi(
input_model = NULL,
input_corpus = NULL,
userid = NA,
n_top_terms = 5,
bottom_terms_percentile = 0.6,
exact_n = NULL,
frac = 0.01,
n_top_topics = 3,
n_topiclabel_words = 8,
frexweight = 0.5,
use_frex_words = FALSE,
input_dfm = NULL,
btm_dataframe = NULL,
lambda = 1,
difficulty = NULL
)
ti(
input_model = NULL,
input_corpus = NULL,
userid = NA,
exact_n = NULL,
frac = 0.01,
n_top_topics = 3,
n_topiclabel_words = 8,
frexweight = 0.5,
use_frex_words = FALSE,
input_dfm = NULL,
btm_dataframe = NULL,
lambda = 1,
difficulty = NULL
)
wsi(
input_model = NULL,
userid = NA,
n_topiclabel_words = 4,
n_correct_ws = 3,
wsi_n_top_terms = 20,
frexweight = 0.5,
use_frex_words = FALSE,
lambda = 1,
difficulty = NULL
)
gs(
input_corpus = NULL,
userid = NA,
construct = "positive",
exact_n = NULL,
frac = 0.01
)
Arguments
input_model
(wi, ti, witi, wsi) a STM, WarpLDA, topicmodels, KeyATM, seededlda, textmodel_nb, or BTM object; if it is NULL, create_oolong assumes that you want to create gold standard.
input_corpus
(wi, ti, witi, wsi, gs) if input_model is not null, it should be the corpus (character vector or quanteda::corpus object) to generate the model object. If input_model and input_corpus are not NULL, topic intrusion test cases are generated. If input_model is a BTM object, this argument is ignored. If input_model is null, it generates gold standard test cases.
n_top_terms
(wi, witi) integer, number of top topic words to be included in the candidates of word intrusion test.
bottom_terms_percentile
(wi, witi) double, a term is considered to be an word intruder when its theta less than the percentile of this theta, must be within the range of 0 to 1
exact_n
(ti, witi, gs) integer, number of topic intrusion test cases to generate, ignore if frac is not NULL
frac
(ti, witi, gs) double, fraction of test cases to be generated from the corpus
n_top_topics
(wi, witi) integer, number of most relevant topics to be shown alongside the intruder topic
n_topiclabel_words
(witi, ti, wsi) integer, number of topic words to be shown as the topic ("ti" and "witi") / word set ("wsi") label
use_frex_words
(wi, witi, ti, wsi) logical, for a STM object, use FREX words if TRUE, use PROB words if FALSE
frexweight
(wi, witi, ti, wsi) double, adjust the 'frexweight' for STM (see [stm::labelTopics()]), no effect for STM if use_frex_words is FALSE
input_dfm
(wi, witi, ti, wsi) a dfm object used for training the input_model, if input_model is a WarpLDA object
construct
(gs) string, an adjective to describe the construct you want your coders to code the the gold standard test cases
btm_dataframe
(witi, ti) dataframe used for training the input_model, if input_model is a BTM object
n_correct_ws
(wsi) number of word sets to be shown alongside the intruder word set
wsi_n_top_terms
(wsi) number of top topic words from each topic to be randomized selected as the word set label
userid
a character string to denote the name of the coder. Default to NA (no userid); not recommended
type
(create_oolong) a character string to denote what you want to create. "wi": word intrusion test; "ti": topic intrusion test; "witi": both word intrusion test and topic intrusion test; "gs": gold standard generation
lambda
(wi, witi, ti, wsi) double, adjust the 'lambda' for WarpLDA (see [text2vec::LatentDirichletAllocation()])
difficulty
(wi, witi, ti, wsi) double, deprecated, for backward compatibility
Value
an oolong test object.
Usage
Use wi, ti, witi, wsi or gs to generate an oolong test of your choice. It is recommended to supply also userid (current coder).
The names of the tests (word intrusion test and topic intrusion test) follow Chang et al (2009). In Ying et al. (2021), topic intrusion test is named "T8WSI" (Top 8 Word Set Intrusion). Word set intrusion test in this package is actually the "R4WSI" (Random 4 Word Set Intrusion) in Ying et al. The default settings of wi, witi, and ti follow Chang et al (2009), e.g. n_top_terms = 5; instead of n_top_terms = 4 as in Ying et al. The default setting of wsi follows Ying et al., e.g. n_topiclabel_words = 4.
As suggested by Song et al. (2020), 1
About create_oolong
Because create_oolong is not intuitive to use, it is no longer recommended to use create_oolong to generate oolong test. create_oolong is retained only for backward compatibility purposes. This function generates an oolong test object based on input_model and input_corpus. If input_model is not NULL, it generates oolong test for a topic model (tm). If input_model is NULL but input_corpus is not NULL, it generates oolong test for generating gold standard (gs).
Methods
An oolong object, depends on its purpose, has the following methods:
$do_word_intrusion_test()(tm) launch the shiny-based word intrusion test. The coder should find out the intruder word that is not related to other words.
$do_topic_intrusion_test()(tm) launch the shiny-based topic intrusion test. The coder should find out the intruder topic that is least likely to be the topic of the document.
$do_word_set_intrusion_test()(tm) launch the shiny-based word set intrusion test. The coder should find out the intruder word set that is not related to other word sets.
$do_gold_standard_test()(gs) launch the shiny-based test for generating gold standard. The coder should determine the level of the predetermined constructs with a 5-point Likert scale.
$lock(force = FALSE)(gs/tm) lock the object so that it cannot be changed anymore. It enables summarize_oolong and the following method.
$turn_gold()(gs) convert the oolong object into a quanteda compatible corpus.
For more details, please see the overview vignette: vignette("overview", package = "oolong")
Author(s)
Chung-hong Chan, Marius Sältzer
References
Chang, J., Gerrish, S., Wang, C., Boyd-Graber, J. L., & Blei, D. M. (2009). Reading tea leaves: How humans interpret topic models. In Advances in neural information processing systems (pp. 288-296).
Song et al. (2020) In validations we trust? The impact of imperfect human annotations as a gold standard on the quality of validation of automated content analysis. Political Communication.
Ying, L., Montgomery, J. M., & Stewart, B. M. (2021). Topics, Concepts, and Measurement: A Crowdsourced Procedure for Validating Topics as Measures. Political Analysis
Examples
## Creation of oolong test with only word intrusion test
data(abstracts_keyatm)
data(abstracts)
oolong_test <- wi(input_model = abstracts_keyatm, userid = "Hadley")
## Creation of oolong test with both word intrusion test and topic intrusion test
oolong_test <- witi(input_model = abstracts_keyatm, input_corpus = abstracts$text, userid = "Julia")
## Creation of oolong test with topic intrusion test
oolong_test <- ti(input_model = abstracts_keyatm, input_corpus = abstracts$text, userid = "Jenny")
## Creation of oolong test with word set intrusion test
oolong_test <- wsi(input_model = abstracts_keyatm, userid = "Garrett")
## Creation of gold standard
oolong_test <- gs(input_corpus = trump2k, userid = "Yihui")
## Using create_oolong(); not recommended
oolong_test <- create_oolong(input_model = abstracts_keyatm,
input_corpus = abstracts$text, userid = "JJ")
oolong_test <- create_oolong(input_model = abstracts_keyatm,
input_corpus = abstracts$text, userid = "Mara", type = "ti")
oolong_test <- create_oolong(input_corpus = abstracts$text, userid = "Winston", type = "gs")
oolong documentation built on Aug. 25, 2023, 5:16 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
| create_oolong | R Documentation |
Generate an oolong test
Description
create_oolong generates an oolong test object that can either be used for validating a topic model or for creating ground truth (gold standard) of a text corpus. wi (word intrusion test), ti (topic intrusion test), witi (word and topic intrusion tests), wsi (word set intrusion test) and gs are handy wrappers to create_oolong. It is recommended to use these wrappers instead of create_oolong.
Usage
create_oolong(
input_model = NULL,
input_corpus = NULL,
n_top_terms = 5,
bottom_terms_percentile = 0.6,
exact_n = NULL,
frac = 0.01,
n_top_topics = 3,
n_topiclabel_words = 8,
use_frex_words = FALSE,
frexweight = 0.5,
input_dfm = NULL,
construct = "positive",
btm_dataframe = NULL,
n_correct_ws = 3,
wsi_n_top_terms = 20,
userid = NA,
type = "witi",
lambda = 1,
difficulty = NULL
)
wi(
input_model = NULL,
userid = NA,
n_top_terms = 5,
bottom_terms_percentile = 0.6,
frexweight = 0.5,
use_frex_words = FALSE,
lambda = 1,
difficulty = NULL
)
witi(
input_model = NULL,
input_corpus = NULL,
userid = NA,
n_top_terms = 5,
bottom_terms_percentile = 0.6,
exact_n = NULL,
frac = 0.01,
n_top_topics = 3,
n_topiclabel_words = 8,
frexweight = 0.5,
use_frex_words = FALSE,
input_dfm = NULL,
btm_dataframe = NULL,
lambda = 1,
difficulty = NULL
)
ti(
input_model = NULL,
input_corpus = NULL,
userid = NA,
exact_n = NULL,
frac = 0.01,
n_top_topics = 3,
n_topiclabel_words = 8,
frexweight = 0.5,
use_frex_words = FALSE,
input_dfm = NULL,
btm_dataframe = NULL,
lambda = 1,
difficulty = NULL
)
wsi(
input_model = NULL,
userid = NA,
n_topiclabel_words = 4,
n_correct_ws = 3,
wsi_n_top_terms = 20,
frexweight = 0.5,
use_frex_words = FALSE,
lambda = 1,
difficulty = NULL
)
gs(
input_corpus = NULL,
userid = NA,
construct = "positive",
exact_n = NULL,
frac = 0.01
)
Arguments
input_model |
(wi, ti, witi, wsi) a STM, WarpLDA, topicmodels, KeyATM, seededlda, textmodel_nb, or BTM object; if it is NULL, create_oolong assumes that you want to create gold standard. |
input_corpus |
(wi, ti, witi, wsi, gs) if input_model is not null, it should be the corpus (character vector or quanteda::corpus object) to generate the model object. If input_model and input_corpus are not NULL, topic intrusion test cases are generated. If input_model is a BTM object, this argument is ignored. If input_model is null, it generates gold standard test cases. |
n_top_terms |
(wi, witi) integer, number of top topic words to be included in the candidates of word intrusion test. |
bottom_terms_percentile |
(wi, witi) double, a term is considered to be an word intruder when its theta less than the percentile of this theta, must be within the range of 0 to 1 |
exact_n |
(ti, witi, gs) integer, number of topic intrusion test cases to generate, ignore if frac is not NULL |
frac |
(ti, witi, gs) double, fraction of test cases to be generated from the corpus |
n_top_topics |
(wi, witi) integer, number of most relevant topics to be shown alongside the intruder topic |
n_topiclabel_words |
(witi, ti, wsi) integer, number of topic words to be shown as the topic ("ti" and "witi") / word set ("wsi") label |
use_frex_words |
(wi, witi, ti, wsi) logical, for a STM object, use FREX words if TRUE, use PROB words if FALSE |
frexweight |
(wi, witi, ti, wsi) double, adjust the 'frexweight' for STM (see [stm::labelTopics()]), no effect for STM if use_frex_words is FALSE |
input_dfm |
(wi, witi, ti, wsi) a dfm object used for training the input_model, if input_model is a WarpLDA object |
construct |
(gs) string, an adjective to describe the construct you want your coders to code the the gold standard test cases |
btm_dataframe |
(witi, ti) dataframe used for training the input_model, if input_model is a BTM object |
n_correct_ws |
(wsi) number of word sets to be shown alongside the intruder word set |
wsi_n_top_terms |
(wsi) number of top topic words from each topic to be randomized selected as the word set label |
userid |
a character string to denote the name of the coder. Default to NA (no userid); not recommended |
type |
(create_oolong) a character string to denote what you want to create. "wi": word intrusion test; "ti": topic intrusion test; "witi": both word intrusion test and topic intrusion test; "gs": gold standard generation |
lambda |
(wi, witi, ti, wsi) double, adjust the 'lambda' for WarpLDA (see [text2vec::LatentDirichletAllocation()]) |
difficulty |
(wi, witi, ti, wsi) double, deprecated, for backward compatibility |
Value
an oolong test object.
Usage
Use wi, ti, witi, wsi or gs to generate an oolong test of your choice. It is recommended to supply also userid (current coder).
The names of the tests (word intrusion test and topic intrusion test) follow Chang et al (2009). In Ying et al. (2021), topic intrusion test is named "T8WSI" (Top 8 Word Set Intrusion). Word set intrusion test in this package is actually the "R4WSI" (Random 4 Word Set Intrusion) in Ying et al. The default settings of wi, witi, and ti follow Chang et al (2009), e.g. n_top_terms = 5; instead of n_top_terms = 4 as in Ying et al. The default setting of wsi follows Ying et al., e.g. n_topiclabel_words = 4.
As suggested by Song et al. (2020), 1
About create_oolong
Because create_oolong is not intuitive to use, it is no longer recommended to use create_oolong to generate oolong test. create_oolong is retained only for backward compatibility purposes. This function generates an oolong test object based on input_model and input_corpus. If input_model is not NULL, it generates oolong test for a topic model (tm). If input_model is NULL but input_corpus is not NULL, it generates oolong test for generating gold standard (gs).
Methods
An oolong object, depends on its purpose, has the following methods:
$do_word_intrusion_test()(tm) launch the shiny-based word intrusion test. The coder should find out the intruder word that is not related to other words.
$do_topic_intrusion_test()(tm) launch the shiny-based topic intrusion test. The coder should find out the intruder topic that is least likely to be the topic of the document.
$do_word_set_intrusion_test()(tm) launch the shiny-based word set intrusion test. The coder should find out the intruder word set that is not related to other word sets.
$do_gold_standard_test()(gs) launch the shiny-based test for generating gold standard. The coder should determine the level of the predetermined constructs with a 5-point Likert scale.
$lock(force = FALSE)(gs/tm) lock the object so that it cannot be changed anymore. It enables
summarize_oolongand the following method.$turn_gold()(gs) convert the oolong object into a quanteda compatible corpus.
For more details, please see the overview vignette: vignette("overview", package = "oolong")
Author(s)
Chung-hong Chan, Marius Sältzer
References
Chang, J., Gerrish, S., Wang, C., Boyd-Graber, J. L., & Blei, D. M. (2009). Reading tea leaves: How humans interpret topic models. In Advances in neural information processing systems (pp. 288-296).
Song et al. (2020) In validations we trust? The impact of imperfect human annotations as a gold standard on the quality of validation of automated content analysis. Political Communication.
Ying, L., Montgomery, J. M., & Stewart, B. M. (2021). Topics, Concepts, and Measurement: A Crowdsourced Procedure for Validating Topics as Measures. Political Analysis
Examples
## Creation of oolong test with only word intrusion test
data(abstracts_keyatm)
data(abstracts)
oolong_test <- wi(input_model = abstracts_keyatm, userid = "Hadley")
## Creation of oolong test with both word intrusion test and topic intrusion test
oolong_test <- witi(input_model = abstracts_keyatm, input_corpus = abstracts$text, userid = "Julia")
## Creation of oolong test with topic intrusion test
oolong_test <- ti(input_model = abstracts_keyatm, input_corpus = abstracts$text, userid = "Jenny")
## Creation of oolong test with word set intrusion test
oolong_test <- wsi(input_model = abstracts_keyatm, userid = "Garrett")
## Creation of gold standard
oolong_test <- gs(input_corpus = trump2k, userid = "Yihui")
## Using create_oolong(); not recommended
oolong_test <- create_oolong(input_model = abstracts_keyatm,
input_corpus = abstracts$text, userid = "JJ")
oolong_test <- create_oolong(input_model = abstracts_keyatm,
input_corpus = abstracts$text, userid = "Mara", type = "ti")
oolong_test <- create_oolong(input_corpus = abstracts$text, userid = "Winston", type = "gs")
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.