View source: R/GenerateReviewPolicy.R
GenerateHITReviewPolicy | R Documentation |
Generate a HIT ReviewPolicy and/or Assignment ReviewPolicy data structure
for use in CreateHIT
.
GenerateHITReviewPolicy(...)
... |
ReviewPolicy parameters passed as named arguments. |
Converts a list of ReviewPolicy parameters into a ReviewPolicy data structure.
A ReviewPolicy works by testing whether an assignment or a set of assignments satisfies a particular condition. If that condition is satisfied, then specified actions are taken. ReviewPolicies come in two “flavors”: Assignment-level ReviewPolicies take actions based on “known” answers to questions in the HIT and HIT-level ReviewPolicies take actions based on agreement among multiple assignments. It is possible to specify both Assignment-level and HIT-level ReviewPolicies for the same HIT.
Assignment-level ReviewPolicies involve checking whether that assignment
includes particular (“correct”) answers. For example, an assignment
might be tested to see whether a correct answer is given to one question by
each worker as a quality control measure. The ReviewPolicy works by checking
whether a specified percentage of known answers are correct. So, if a
ReviewPolicy specifies two known answers for a HIT and the worker gets one
of those known answers correct, the ReviewPolicy scores the assignment at 50
(i.e., 50 percent). The ReviewPolicy can then be customized to take three
kinds of actions depending on that score:
ApproveIfKnownAnswerScoreIsAtLeast
(approve the assignment
automatically), RejectIfKnownAnswerScoreIsLessThan
(reject the
assignment automatically), and ExtendIfKnownAnswerScoreIsLessThan
(add additional assignments and/or time to the HIT automatically). The
various actions can be combined to, e.g., both reject an assignment and add
further assignments if a score is below the threshold, or reject below a
threshold and approve above, etc.
HIT-level ReviewPolicies involve checking whether multiple assignments
submitted for the same HIT “agree” with one another. Agreement here
is very strict: answers must be exactly the same across assignments for them
to be a matched. As such, it is probably only appropriate to use
closed-ended (e.g., multiple choice) questions for HIT-level ReviewPolicies
otherwise ReviewPolicy actions might be taken on irrelevant differences
(e.g., word capitalization, spacing, etc.). The ReviewPolicy works by
checking whether answers to multiple assignments are the same (or at least
whether a specified percentage of answers to a given question are the same).
For example, if the goal is to categorize an image into one of three
categories, the ReviewPolicy will check whether two of three workers agree
on the categorization (known as the “HIT Agreement Score”, which is a
percentage of all workers who agree). Depending on the value of the HIT
Agreement Score, actions can be taken. As of October 2014, only one action
can be taken: ExtendIfHITAgreementScoreIsLessThan
(extending the HIT
in assignments by the number of assignments specified in
ExtendMaximumAssignments
or time as specified in
ExtendMinimumTimeInSeconds
).
Another agreement score (the “Worker Agreement Score”), measured the
percentage of a worker's responses that agree with other workers' answers.
Depending on the Worker Agreement Score, two actions can be taken:
ApproveIfWorkerAgreementScoreIsAtLeast
(to approve the assignment
automatically) or RejectIfWorkerAgreementScoreIsLessThan
(to reject
the assignment automatically, with an optional reject reason supplied with
RejectReason
). A logical value (DisregardAssignmentIfRejected
)
specifies whether to exclude rejected assignments from the calculation of
the HIT Agreement Score.
Note: An optional DisregardAssignmentIfKnownAnswerScoreIsLessThan
excludes assignments if those assignments score below a specified
“known” answers threshold as determined by a separate
Assignment-level ReviewPolicy.
A dictionary object
HITReviewPolicy
or AssignmentReviewPolicy
.
Tyler Burleigh, Thomas J. Leeper
API Reference (ReviewPolicies)
## Not run:
# Generate a HIT Review Policy with GenerateHITReviewPolicy
lista <- list(QuestionIds = c("Question1", "Question2"),
QuestionAgreementThreshold = 75,
ApproveIfWorkerAgreementScoreIsAtLeast = 75,
RejectIfWorkerAgreementScoreIsLessThan = 25)
policya <- do.call(GenerateHITReviewPolicy, lista)
# Manually define a HIT Review Policy
policya <- dict(
list(
'PolicyName' = 'SimplePlurality/2011-09-01',
'Parameters' = list(
dict(
'Key' = 'QuestionIds',
'Values' = list(
'Question1',
'Question2'
)
),
dict(
'Key' = 'QuestionAgreementThreshold',
'Values' = list(
'75'
)
),
dict(
'Key' = 'ApproveIfWorkerAgreementScoreIsAtLeast',
'Values' = list(
'75'
)
),
dict(
'Key' = 'RejectIfWorkerAgreementScoreIsLessThan',
'Values' = list(
'25'
)
)
)
))
# Generate an Assignment Review Policy with GenerateAssignmentReviewPolicy
listb <- list(AnswerKey = list("QuestionId1" = "B", "QuestionId2" = "A"),
ApproveIfKnownAnswerScoreIsAtLeast = 99)
policyb <- do.call(GenerateAssignmentReviewPolicy, listb)
# Manually define an Assignment Review Policy
policyb <- dict(
list(
'PolicyName' = 'ScoreMyKnownAnswers/2011-09-01',
'Parameters' = list(
dict(
'Key' = 'AnswerKey',
'MapEntries' = list(
dict(
'Key' = 'QuestionId1',
'Values' = list('B')
),
dict(
'Key' = 'QuestionId2',
'Values' = list('A')
)
)
),
dict(
'Key' = 'ApproveIfKnownAnswerScoreIsAtLeast',
'Values' = list(
'99'
)
)
)
))
## End(Not run)
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.