checkWorkers: Detect outlying workers

In RydenButler/labelR: Pairwise Data Labeling with Amazon's Mechanical Turk

Detect outlying workers

Description

Detect outlying workers

Usage

checkWorkers(
  stan_fit,
  data,
  cut_point = 1,
  cut_proportion = 0.9,
  n.questions = 50,
  plot_hist = FALSE,
  hist_path = NULL
)

Arguments

stan_fit	A stan fit.
data	The data used to fit stan.
cut_point	A cutoff point to classify posterior coefficients. The proportion of posterior coefficients below cut_point is used to determine outliers. (Default is 1)
cut_proportion	A cutoff proportion of posterior coefficients below cut_point. If the proportion of posterior coefficients below cut_points is higher than cut_proportion, a worker will be considered as an outlier provided that she answers more than the number of questions in n.questions. (Default is 0.9)
n.questions	The number of questions to consider in order to determine suggested banned workers. (Default is 50)
plot_hist	If TRUE, plot the histogram of workers with a rug plot. Default is FALSE
hist_path	Save the histogram to path and file name specified. Default is NULL and no plot is saved.

Details

Once you have estimated a Stan model, you can use this to identify workers who are unreliable. You can use the output from the Stan model, as well as the output from the completed comparisons, to identify unreliable workers. Use the checkWorkers() function to detect these outlying workers. This function returns a vector of IDs identifying workers who are unreliable.

Reference Paper: Carlson, David and Jacob M. Montgomery. Forthcoming. “A Pairwise Comparison Framework for Fast, Flexible, and Reliable Human Coding of Political Texts.” American Political Science Review.

Value

ban_workers A vector of outlying workers' IDs whose proportion of posterior coefficients below cut_point is greater than cut_proportion and who answered more than the number of questions in n.questions

Author(s)

David Carlson

RydenButler/labelR documentation built on May 9, 2023, 5:52 a.m.