Description Usage Arguments Details Value Author(s) References See Also Examples
Generate a HIT ReviewPolicy and/or Assignment ReviewPolicy data structure for use in CreateHIT
.
1 2 3 |
... |
ReviewPolicy parameters passed as named arguments. See details and examples. |
Converts a list of ReviewPolicy parameters into a ReviewPolicy data structure.
A ReviewPolicy works by testing whether an assignment or a set of assignments satisfies a particular condition. If that condition is satisfied, then specified actions are taken. ReviewPolicies come in two “flavors”: Assignment-level ReviewPolicies take actions based on “known” answers to questions in the HIT and HIT-level ReviewPolicies take actions based on agreement among multiple assignments. It is possible to specify both Assignment-level and HIT-level ReviewPolicies for the same HIT.
Assignment-level ReviewPolicies involve checking whether that assignment includes particular (“correct”) answers. For example, an assignment might be tested to see whether a correct answer is given to one question by each worker as a quality control measure. The ReviewPolicy works by checking whether a specified percentage of known answers are correct. So, if a ReviewPolicy specifies two known answers for a HIT and the worker gets one of those known answers correct, the ReviewPolicy scores the assignment at 50 (i.e., 50 percent). The ReviewPolicy can then be customized to take three kinds of actions depending on that score: ApproveIfKnownAnswerScoreIsAtLeast
(approve the assignment automatically), RejectIfKnownAnswerScoreIsLessThan
(reject the assignment automatically), and ExtendIfKnownAnswerScoreIsLessThan
(add additional assignments and/or time to the HIT automatically). The various actions can be combined to, e.g., both reject an assignment and add further assignments if a score is below the threshold, or reject below a threshold and approve above, etc.
HIT-level ReviewPolicies involve checking whether multiple assignments submitted for the same HIT “agree” with one another. Agreement here is very strict: answers must be exactly the same across assignments for them to be a matched. As such, it is probably only appropriate to use closed-ended (e.g., multiple choice) questions for HIT-level ReviewPolicies otherwise ReviewPolicy actions might be taken on irrelevant differences (e.g., word capitalization, spacing, etc.). The ReviewPolicy works by checking whether answers to multiple assignments are the same (or at least whether a specified percentage of answers to a given question are the same). For example, if the goal is to categorize an image into one of three categories, the ReviewPolicy will check whether two of three workers agree on the categorization (known as the “HIT Agreement Score”, which is a percentage of all workers who agree). Depending on the value of the HIT Agreement Score, actions can be taken. As of October 2014, only one action can be taken: ExtendIfHITAgreementScoreIsLessThan
(extending the HIT in assignments by the number of assignments specified in ExtendMaximumAssignments
or time as specified in ExtendMinimumTimeInSeconds
).
Another agreement score (the “Worker Agreement Score”), measured the percentage of a worker's responses that agree with other workers' answers. Depending on the Worker Agreement Score, two actions can be taken: ApproveIfWorkerAgreementScoreIsAtLeast
(to approve the assignment automatically) or RejectIfWorkerAgreementScoreIsLessThan
(to reject the assignment automatically, with an optional reject reason supplied with RejectReason
). A logical value (DisregardAssignmentIfRejected
) specifies whether to exclude rejected assignments from the calculation of the HIT Agreement Score.
Note: An optional DisregardAssignmentIfKnownAnswerScoreIsLessThan
excludes assignments if those assignments score below a specified “known” answers threshold as determined by a separate Assignment-level ReviewPolicy.
A character string that reflects the URL-encoded HITReviewPolicy
or AssignmentReviewPolicy
.
Thomas J. Leeper
API Reference (ReviewPolicies)
API Reference (Data Structure)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 | ## Not run:
# HIT-level ReviewPolicies #
## Conditionally extend HIT based on HIT Agreement Score
lista <- list(QuestionIds = c("Question1","Question2","Question5"),
QuestionAgreementThreshold = 49, # at least 50 percent agreement
ExtendIfHITAgreementScoreIsLessThan = 50,
ExtendMinimumTimeInSeconds = 3600,
ExtendMaximumAssignments = 2,
DisregardAssignmentIfRejected = TRUE)
policya <- do.call(GenerateHITReviewPolicy, lista)
## Conditionally approve and reject based on Worker Agreement Score
listb <- list(QuestionIds = c("Question1","Question2","Question5"),
QuestionAgreementThreshold = 65, # at least two of three 'correct' answers
ApproveIfWorkerAgreementScoreIsAtLeast = 65,
RejectIfWorkerAgreementScoreIsLessThan = 34,
DisregardAssignmentIfRejected = TRUE)
policyb <- do.call(GenerateHITReviewPolicy, listb)
# Attach an assignment review policy to a HIT
hit1 <-
CreateHIT(title = "Survey",
description = "5 question survey",
reward = ".10",
expiration = seconds(days = 4),
duration = seconds(hours = 1),
keywords = "survey, questionnaire",
hit.review.policy = policyb,
question = GenerateExternalQuestion("http://www.example.com/","400"))
# GetReviewResults
GetReviewResultsForHIT(hit1$HITId)
# cleanup
ExpireHIT(hit1$HITId)
DisposeHIT(hit1$HITId)
## End(Not run)
## Not run:
# Assignment-level ReviewPolicies #
# Example Policy A
## Conditional approval of assignments based on percent of correct answers
lista <- list(AnswerKey = list("QuestionId1" = "B",
"QuestionId2" = "A"),
ApproveIfKnownAnswerScoreIsAtLeast = 99)
policya <- do.call(GenerateAssignmentReviewPolicy, lista)
# Example Policy B
## Conditional approval of assignments based on percent of correct answers
## Conditional rejection of assignments based on percent of correct answers
listb <- list(AnswerKey = list("QuestionId1" = "B",
"QuestionId2" = "A"),
ApproveIfKnownAnswerScoreIsAtLeast = 99,
RejectIfKnownAnswerScoreIsLessThan = 1)
policyb <- do.call(GenerateAssignmentReviewPolicy, listb)
# Example Policy C
## Conditionally extend HIT with more time and assignments based on score
listc <- list(AnswerKey = list("QuestionId1" = "B"),
ExtendIfKnownAnswerScoreIsLessThan = 100,
ExtendMaximumAssignments = 2, # maximum value is 25
ExtendMinimumTimeInSeconds = seconds(hours = 1))
policyc <- do.call(GenerateAssignmentReviewPolicy, listc)
# Attach an assignment review policy to a HIT
## load template file
## applying Policy B will approve if form answers are "Something" and "Yes"
## and reject if both are incorrect
f <- system.file("templates/htmlquestion3.xml", package = "MTurkR")
hit1 <-
CreateHIT(title = "Survey",
description = "5 question survey",
reward = ".10",
expiration = seconds(days = 4),
duration = seconds(hours = 1),
keywords = "survey, questionnaire",
assignment.review.policy = policyb,
question = GenerateHTMLQuestion(file = f))
# GetReviewResults
GetReviewResultsForHIT(hit1$HITId)
# cleanup
ExpireHIT(hit1$HITId)
DisposeHIT(hit1$HITId)
## End(Not run)
## Not run:
# HIT- and Assignment-level ReviewPolicies
# Example Policy D
## Conditional approval of assignments based on percent of correct answers
listd <- list(AnswerKey = list("QuestionId1" = "B",
"QuestionId2" = "A"),
ApproveIfKnownAnswerScoreIsAtLeast = 99)
policyd <- do.call(GenerateAssignmentReviewPolicy, listd)
# Example Policy E
## Conditionally extend HIT based on HIT Agreement Score
liste <- list(QuestionIds = c("QuestionId3","QuestionId4","QuestionId5"),
QuestionAgreementThreshold = 65, # at least 2/3rd agreement
ExtendIfHITAgreementScoreIsLessThan = 66,
ExtendMinimumTimeInSeconds = 3600,
ExtendMaximumAssignments = 2,
DisregardAssignmentIfRejected = TRUE)
policye <- do.call(GenerateHITReviewPolicy, liste)
## load template file
## applying 'policya' will approve if form answers are "Something" and "Yes"
f <- system.file("templates/htmlquestion3.xml", package = "MTurkR")
# Create HIT
hit2 <-
CreateHIT(title = "Survey",
description = "5 question survey",
reward = ".10",
expiration = seconds(days = 4),
duration = seconds(hours = 1),
keywords = "survey, questionnaire",
assignment.review.policy = policyd,
hit.review.policy = policye,
question = GenerateHTMLQuestion(file = f))
# GetReviewResults
GetReviewResultsForHIT(hit2$HITId)
# cleanup
ExpireHIT(hit2$HITId)
DisposeHIT(hit2$HITId)
## End(Not run)
|
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.