BayesianWAgg: Aggregation Method: BayesianWAgg
In aggreCAT: Mathematically Aggregating Expert Judgments

BayesianWAgg

R Documentation

Aggregation Method: BayesianWAgg

Description

Bayesian aggregation methods with either uninformative or informative prior distributions.

JAGS Install

For instructions on installing JAGS onto your system visit https://gist.github.com/dennisprangle/e26923fae7477566510757ab3341f54c

Usage

BayesianWAgg(
  expert_judgements,
  type = "BayTriVar",
  priors = NULL,
  name = NULL,
  placeholder = FALSE,
  percent_toggle = FALSE,
  round_2_filter = TRUE
)

Arguments

`expert_judgements`	A dataframe in the format of data_ratings.
`type`	One of `"BayTriVar"`, or `"BayPRIORsAgg"`.
`priors`	(Optional) A dataframe of priors in the format of data_supp_priors, required for `type` `BayPRIORsAgg`.
`name`	Name for aggregation method. Defaults to `type` unless specified.
`placeholder`	Toggle the output of the aggregation method to impute placeholder data.
`percent_toggle`	Change the values to probabilities. Default is `FALSE`.
`round_2_filter`	Note that the IDEA protocol results in both a Round 1 and Round 2 set of probabilities for each claim. Unless otherwise specified, we will assume that the final Round 2 responses (after discussion) are being referred to.

Details

type may be one of the following:

BayTriVar: The Bayesian Triple-Variability Method, fit with JAGS.

\loadmathjax

Three kinds of variability around best estimates are considered:

generic claim variability: variation across individuals within a claim
generic participant variability: variation within an individual across claims
claim - participant specific uncertainty (operationalised by bounds): informed by interval widths given by individual \mjeqniascii for claim \mjeqncascii.

The model takes the log odds transformed individual best estimates as input (data), uses a normal likelihood function and derives a posterior distribution for the probability of replication.

\mjdeqn

log( \fracB_i,c1-B_i,c) \sim N(\mu_c, \sigma_i,c),ascii

where \mjeqn\mu_cascii denotes the mean estimated probability of replication for claim \mjeqncascii, and \mjeqn\sigma_i,cascii denotes the standard deviation of the estimated probability of replication for claim \mjeqncascii and individual \mjeqniascii (on the logit scale). Parameter \mjeqn\sigma_i,cascii is calculated as: \mjdeqn\sigma_i,c = (U_i,c - L_i,c + 0.01) \times \sqrt\sigma_i^2+\sigma_c^2ascii with \mjeqn\sigma_iascii denoting the standard deviation of estimated probabilities of replication for individual \mjeqniascii and \mjeqn\sigma_cascii denoting the standard deviation of the estimated probability of replication for claim \mjeqncascii.

The uninformative priors for specifying this Bayesian model are \mjeqn\mu_c \sim N(0,\ 3)ascii, \mjeqn\sigma_i \sim U(0,\ 10)ascii and \mjeqn\sigma_c \sim U(0,\ 10)ascii. After obtaining the median of the posterior distribution of \mjeqn\mu_cascii, we can back transform to obtain \mjeqn\hatp_cascii:

\mjdeqn\hat

p_c\left( BayTriVar \right) = \frace^\mu_c1+e^\mu_cascii

BayPRIORsAgg: Priors derived from predictive models, updated with best estimates.

This method uses Bayesian updating to update a prior probability of replication estimated from a predictive model with an aggregate of the individuals’ best estimates for any given claim. Methodology is the same as type "BayTriVar" except an informative prior is used for \mjeqn\mu_cascii. Conceptually the parameters of the prior distribution of \mjeqn\mu_cascii are informed by the PRIORS model (Gould et al. 2021) which is a multilevel logistic regression model that predicts the probability of replication using attributes of the original study. However, any model providing predictions of the probability of replication can be used to generate the required priors.

Value

A tibble of confidence scores cs for each paper_id.

Warning

Both BayTriVar and BayPRIORsAgg methods require a minimum of two claims for which judgements are supplied to expert_judgements. This is due to the mathematical definition of these aggregators: BayesianWAgg calculates the variance in best estimates across multiple claims as well as the variance in best estimates across claims per individual. Thus when only one claim is provided in expert_judgements, the variance is 0, hence more than one claim is required for the successful execution of both Bayesian methods.