checkAgree: Check Agreement Rate between Identical Trails

In Luwei-Ying/validateIt: Validating Topic Coherence and Topic Labels

Description

Check Agreement Rate between Identical Trails

Usage

checkAgree(results1, results2, key, type = NULL)

Arguments

results1	first batch of results; outputs from getResults()
results2	first batch of results; outputs from getResults()
key	the local task record; outputs from recordTasks()
type	Task structures to be specified. Must be one of "WI" (word intrusion), "T8WSI" (top 8 word set intrusion), "R4WSI" (random 4 word set intrusion), "LI" (Label Intrusion), and "OL" (Optimal Label)

Details

Evaluate workers' performance by agreement rate between identical trails (Notice that this means the two input, results1 and results2, must be identical.); Return 1) the exact agreement rate when both workers agree on the exact same choice, and 2) the binary agreement rate when both workers get the task either right or wrong simultaneously

Luwei-Ying/validateIt documentation built on May 12, 2021, 9:43 p.m.