ReasoningWAgg: Aggregation Method: ReasoningWAgg
In aggreCAT: Mathematically Aggregating Expert Judgments

ReasoningWAgg

R Documentation

Aggregation Method: ReasoningWAgg

Description

Calculate one of several types of linear-weighted best estimates using supplementary participant reasoning data to create weights.

Usage

ReasoningWAgg(
  expert_judgements,
  reasons = NULL,
  type = "ReasonWAgg",
  name = NULL,
  beta_transform = FALSE,
  beta_param = c(6, 6),
  placeholder = FALSE,
  percent_toggle = FALSE,
  flag_loarmean = FALSE,
  round_2_filter = TRUE
)

Arguments

`expert_judgements`	A dataframe in the format of data_ratings.
`reasons`	A dataframe in the form of data_supp_reasons
`type`	One of `"ReasonWAgg"`, `"ReasonWAgg2"`.
`name`	Name for aggregation method. Defaults to `type` unless specified.
`beta_transform`	Toggle switch to extremise confidence scores with the beta distribution. Defaults to `FALSE`.
`beta_param`	Length two vector of alpha and beta parameters of the beta distribution. Defaults to `c(6,6)`.
`placeholder`	Toggle the output of the aggregation method to impute placeholder data.
`percent_toggle`	Change the values to probabilities. Default is `FALSE`.
`flag_loarmean`	A toggle to impute LOArMean instead of ArMean when no participants have a reasoning weight for a specific claim (defaults `FALSE`).
`round_2_filter`	Note that the IDEA protocol results in both a Round 1 and Round 2 set of probabilities for each claim. Unless otherwise specified, we will assume that the final Round 2 responses (after discussion) are being referred to.

Details

Weighted by the breadth of reasoning provided to support the individuals’ estimate.

type may be one of the following:

\loadmathjax

ReasonWAgg: Weighted by the number of supporting reasons

Giving greater weight to best estimates that are accompanied by a greater number of supporting reasons may be beneficial. We will consider \mjeqnw\_reason_i,cascii to be the number of unique reasons provided by that individual \mjeqniascii in support of their estimate for claim \mjeqncascii.

\mjdeqn\hat

p_c(ReasonWAgg) = \sum_i=1^N \tildew\_reason_i,cB_i,cascii

See Hanea et al. (2021) for an example of reason coding.

ReasonWAgg2: Incorporates both the number of reasons and their diversity across claims.

The claim diversity component of this score is calculated per individual from all claims they assessed. We assume each individual answers at least two claims. If an individual has assessed only one claim, there weighting for that claim is equivalent to "ReasonWAgg".

We will consider \mjeqnw\_varReason_i,cascii to be the weighted "number of unique reasons" provided by participant \mjeqniascii in support of their estimate for claim \mjeqncascii. Assume there are \mjeqnRascii total unique reasons any participant can use to justify their numerical answers. Then, for each participant \mjeqniascii we can construct a matrix \mjeqn\mathbfCR_iascii with \mjeqnRascii columns, each corresponding to a unique reason, \mjeqnrascii, and \mjeqnCascii rows, where \mjeqnCascii is the number of claims assessed by that participant. Each element of this matrix \mjeqn\mathbfCR_i(r,c)ascii can be either 1 or 0. \mjeqn\mathbfCR_i(r,c) = 1ascii if reason \mjeqnR_rascii was used to justify the estimates assessed for \mjeqncascii, and \mjeqn\mathbfCR_i(r,c) = 0ascii if reason \mjeqnR_rascii was not mentioned when assessing claim \mjeqncascii. The more frequently that a participant uses a given reason reduces the amount it contributes to the weight assigned to that participant.

\mjdeqn

w\_varReason_i,c =\sum_r=1^R \mathbfCR_i(c,r) \cdot (1 - \frac\sum_c=1^C \mathbfCR_i(c,r)C)ascii

\mjdeqn\hat

p_c(ReasonWAgg2) = \sum_i=1^N \tildew\_varReason_i,cB_i,cascii

Value

A tibble of confidence scores cs for each paper_id.

Note

When flag_loarmean is set to TRUE, two additional columns will be returned; method_applied (a character variable describing the method actually applied with values of either LoArMean or ReasonWAgg) and no_reason_score (a logical variable describing whether no reasoning scores were supplied for any user for the given claim, where TRUE indicates no reasoning scores supplied and FALSE indicates that at least one participant for that claim had a reasoning score greater than 0).

named method_applied (with values LoArMean or ReasonWAgg), and no_reason_score, a logical variable describing whether or not there were no reasoning scores for that claim.