auditWorkers: Audit Mechanical Turk workers

In RydenButler/labelR: Pairwise Data Labeling with Amazon's Mechanical Turk

Audit Mechanical Turk workers

Description

auditWorkers estimates worker quality scores from collected MTurk data.

Usage

auditWorkers(
  current_experiment_results,
  reference_results = NULL,
  exclude_workers = NULL,
  plot_audit = FALSE
)

Arguments

current_experiment_results	An object returned from getResults containing HIT results from Mechanical Turk.
reference_results	An optional object of HIT results from previous experiments. When available, this improves the accuracy of worker quality scores.
exclude_workers	An optional object of worker ids to be excluded from the printed output. Typically, a vector of ids for banned worker is passed here to prevent them from continually appearing in audits.
plot_audit	When TRUE, the function will produce a plot of worker ability estimates and their corresponding 95 percent credible intervals.

Details

The function audits worker quality using a Bayesian hierarchical model. Worker means less than one are generally considered poor quality, though means can be unreliable for workers with a low number of total HITs.

Value

A dataframe containing workers' posterior means and HIT frequencies.

Author(s)

Ryden Butler

RydenButler/labelR documentation built on May 9, 2023, 5:52 a.m.