rates: rates
In Hunt-Laboratory/huntr: Hunt Lab Experiment Data

Description Format

One row per external rating, with related attributes. Contains raw quality of reasoning scores and other ratings such as on the redaction test for tightness.

problem: Problem title.
team: Team name.
rater: Rater’s name.
c1: Score (Poor, Fair, Good, Excellent) assigned by rater on the IC Rating Scale Criterion 1.
c1comment: Rater’s comment associated with the score on criterion 1.
c2: Score (Poor, Fair, Good, Excellent) assigned by rater on the IC Rating Scale Criterion 2.
c2comment: Rater’s comment associated with the score on criterion 2.
c3: Score (Poor, Fair, Good, Excellent) assigned by rater on the IC Rating Scale Criterion 3.
c3comment: Rater’s comment associated with the score on criterion 3.
c4: Score (Poor, Fair, Good, Excellent) assigned by rater on the IC Rating Scale Criterion 4.
c4comment: Rater’s comment associated with the score on criterion 4.
c5: Score (Poor, Fair, Good, Excellent) assigned by rater on the IC Rating Scale Criterion 5.
c5comment: Rater’s comment associated with the score on criterion 5.
c6: Score (Poor, Fair, Good, Excellent) assigned by rater on the IC Rating Scale Criterion 6.
c6comment: Rater’s comment associated with the score on criterion 6.
c7: Score (Poor, Fair, Good, Excellent) assigned by rater on the IC Rating Scale Criterion 7.
c7comment: Rater’s comment associated with the score on criterion 7.
c8: Score (Poor, Fair, Good, Excellent) assigned by rater on the IC Rating Scale Criterion 8.
c8comment: Rater’s comment associated with the score on criterion 8.
geo1: Rater’s determination of whether the team successfully identified the geolocation problem 1 (Yes, Partial Credit, No).
geo2: Rater’s determination of whether the team successfully identified the geolocation problem 2 (Yes, Partial Credit, No).
geo3: Rater’s determination of whether the team successfully identified the geolocation problem 3 (Yes, Partial Credit, No).
geo4: Rater’s determination of whether the team successfully identified the geolocation problem 4 (Yes, Partial Credit, No).
IC: Total score on the IC Rating Scale (c1 + c2 + c3 + c4 + c5 + c6 + c7 + c8).
geo1score: Numerical score for geolocation problem 1 (Yes, Partial Credit = 1, No = 0).
geo2score: Numerical score for geolocation problem 2 (Yes, Partial Credit = 1, No = 0).
geo3score: Numerical score for geolocation problem 3 (Yes, Partial Credit = 1, No = 0).
geo4score: Numerical score for geolocation problem 4 (Yes, Partial Credit = 1, No = 0).
nGeoCorrect: geo1score + geo2score + geo3score + geo4score
isRedactionTestRating: Yes/No indicating whether the rating is a redaction test rating for tightness (rather than, say, a quality of reasoning rating using the IC Rating Scale).
raterProbabilityEstimate: The rater’s probability estimate in a redaction test to measure tightness.
estTimeTaken: Rater’s estimate of the time it took them to estimate the probability in a redaction test for tightness.
estJustification: Rater’s justification for their probability estimate in a redaction test for tightness.
estComments: Rater’s comments about the process of estimating the probability in a redaction test for tightness.
bayes1: Yes/No indicating whether the team successfully solved Bayesian probability problem 1.
bayes2: Yes/No indicating whether the team successfully solved Bayesian probability problem 2.
bayes3: Yes/No indicating whether the team successfully solved Bayesian probability problem 3.
bayes1score: Numerical score for Bayesian problem 1 (Yes = 1, No = 0).
bayes2score: Numerical score for Bayesian problem 2 (Yes = 1, No = 0).
bayes3score: Numerical score for Bayesian problem 3 (Yes = 1, No = 0).
nBayesCorrect: bayes1score + bayes2score + bayes3score
flaw1: Rater’s determination of whether the team successfully identified reasoning flaw 1 (Yes/No).
flaw2: Rater’s determination of whether the team successfully identified reasoning flaw 2 (Yes/No).
flaw3: Rater’s determination of whether the team successfully identified reasoning flaw 3 (Yes/No).
flaw4: Rater’s determination of whether the team successfully identified reasoning flaw 4 (Yes/No).