Acknowledgement. Large language models supplied by Anthropic and OpenAI were used in preparing this vignette, to assist with checking code, testing the veracity of the approaches described, and on occasion editing the text. Responsibility for the content rests with the authors.
knitr::opts_chunk$set(collapse = TRUE, comment = "#>", warning = FALSE, message = FALSE, fig.width = 7, fig.height = 5)
This vignette shows the Bayesian posterior summaries in r4lineups for lineup fairness and confidence-based eyewitness measures.
Use these functions when you want credible intervals or direct probability
statements such as (P(E' < 4.5 \mid data)), (P(F < 6 \mid data)), or
(P(DR > 1 \mid data)). The examples use small posterior samples so that the
vignette runs quickly during package checks; increase S for final analyses.
Most readers of this package will have been trained in the frequentist tradition, so this section sets out what is different here. Nothing in what follows requires accepting a philosophical position; it is mainly about what question the interval answers.
The question you probably want to ask. Having measured a six-person lineup and obtained an effective size of 4.8, a natural question is: what is the probability that this lineup's effective size is below 4.5, a tolerance chosen before looking at the data? (Why not "below 6" is explained below: for effective size that question has the same answer, 1, whatever the data.) A frequentist confidence interval cannot answer that. It is a statement about the procedure — that intervals constructed this way contain the true value 95% of the time — not about this lineup. The true effective size either is or is not below 4.5, and frequentist probability does not attach to it.
A Bayesian credible interval does answer it. "There is a 95% probability that E′ lies between 4.1 and 5.6, given these data" is a statement about the quantity, and it is the statement most people believe they are making when they report a confidence interval.
How that is achieved. By treating the unknown quantity as having a distribution. Three steps:
alpha = 0.5 is the Jeffreys prior for
proportions, which is close to saying "no lineup member is favoured, and I am
not confident about that either".Why the functions return draws rather than formulas. Effective size is
(1/\sum p_i^2) — a nonlinear function of the choice probabilities. There is
no tidy closed form for its posterior. Instead the functions sample S draws
from the posterior of the probabilities and transform each draw. The resulting
collection of S values is the posterior of E′, and any summary is computed
from it. This is why S is important: too few draws and the summaries are noisy.
Note that these are independent Monte Carlo draws, not MCMC. Monte Carlo
means summarising a distribution from random draws taken from it.
Markov chain Monte Carlo (MCMC) is the method used when a posterior cannot be
sampled directly: it builds a chain of draws, each depending on the last, that
only gradually settles into the posterior, so its draws are correlated and the
chain has to be checked for convergence. Here the posteriors have standard forms
and can be sampled directly, so every draw is independent and there is no chain
to check (see Technical Details).
What you gain in practice, beyond the interpretation:
What it does not give you. It is not a way of getting a decisive answer from thin data. A weak prior and a small sample give a wide posterior.
library(r4lineups)
The Bayesian fairness functions use mock-witness lineup choices:
The Bayesian calibration function uses the standard confidence-based data frame
with target_present, identification, and confidence. The SDT comparison
uses summary counts: hits, misses, false alarms, and correct rejections for two
conditions.
esize_T_bayes() fits a Dirichlet-multinomial model to the lineup choice counts
and transforms each posterior draw of the choice-probability vector to Tredoux's
effective size (E').
The Dirichlet distribution is a distribution over sets of probabilities that sum to 1, here the probabilities (p_1, \ldots, p_k) that a mock witness chooses each of the (k) lineup members. It is the multi-category generalisation of the beta distribution, which describes a single probability. It has one parameter per member, (\alpha_i), which acts like a count of choices imagined before seeing the data. With mock-witness counts (n_1, \ldots, n_k), the posterior is again Dirichlet, with parameters (n_i + \alpha_i): the prior counts are simply added to the observed ones. That is why the prior has little influence once there are a few dozen real choices, and why the posterior can be sampled exactly.
data(nortje2012) lineup_vec <- nortje2012$lineup_1 k <- max(lineup_vec, na.rm = TRUE) lineup_tab <- table(factor(lineup_vec, levels = seq_len(k))) # An illustrative tolerance. In a real analysis choose and justify it in # advance; the software cannot supply one. tolerance <- 0.75 * k set.seed(1) esize_post <- esize_T_bayes( lineup_tab, alpha = 0.5, S = 1000, threshold = tolerance ) esize_post
plot(esize_post)
Why the threshold is not the nominal size. For any choice probabilities
over (k) members, (E' = 1/\sum p_i^2 \le k), with equality only when all
(k) probabilities are exactly equal. A continuous posterior gives that single
point probability zero, so (P(E' < k \mid data) = 1) for any data at all,
including perfectly balanced counts; esize_T_bayes() warns if asked for it.
Useful summaries are the credible interval for (E'), the interval for the
deficit (k - E'), or the probability below a tolerance chosen in advance:
deficit <- k - esize_post$E_draws knitr::kable( data.frame( Summary = c(sprintf("95%% interval for E' (k = %d)", k), "95% interval for the deficit k - E'", sprintf("P(E' < %.1f | data)", tolerance)), Value = c(sprintf("[%.2f, %.2f]", esize_post$credible_interval[["lower"]], esize_post$credible_interval[["upper"]]), sprintf("[%.2f, %.2f]", quantile(deficit, .025), quantile(deficit, .975)), sprintf("%.3f", esize_post$threshold_probs$P_below)) ), row.names = FALSE, caption = "Posterior summaries of effective size, nortje2012 lineup 1." )
Functional size does not have this problem: it is not bounded above by (k), so (P(F < k \mid data)) is informative.
func_size_bayes() models the suspect-selection rate with a beta-binomial
posterior and transforms posterior draws to functional size.
set.seed(2) func_post <- func_size_bayes( lineup_vec, target_pos = 3, alpha = 0.5, S = 1000, threshold = k ) func_post
plot(func_post)
diag_ratio_T_bayes() models target-present and target-absent suspect-selection
rates with independent beta-binomial posteriors and returns the posterior of the
diagnosticity ratio. The example pairs the first two mock-witness lineups in
nortje2012 (suspect positions 3 and 6) as if they were a target-present and a
target-absent lineup, only to show the computation; they are not such a pair, so
the result is not a finding about diagnosticity.
tp_vec <- nortje2012$lineup_1 ta_vec <- nortje2012$lineup_2 set.seed(3) diag_post <- diag_ratio_T_bayes( lineup_pres = tp_vec, lineup_abs = ta_vec, pos_pres = 3, pos_abs = 6, k1 = max(tp_vec, na.rm = TRUE), k2 = max(ta_vec, na.rm = TRUE), alpha = 0.5, S = 1000, threshold = 1 ) diag_post
plot(diag_post)
The threshold 1 is useful because (DR > 1) indicates that suspect
identifications are more common in the target-present lineup than in the
target-absent lineup.
The Bayesian calibration model conditions on bin frequencies and treats
correctness within each bin as exchangeable Bernoulli observations. Fixed
target-present and target-absent trial totals do not generally imply that
conditional binomial likelihood. Interpret these intervals as working-model
summaries for that assumption, rather than automatically exact inference for
a fixed-arm experiment. Filler-derived fractional counts add another
approximation; see ?calibration_bayes.
calibration_bayes() places beta posteriors on accuracy within confidence bins
and propagates uncertainty into calibration (C), over/underconfidence
(O/U), and normalized resolution (NRI).
data(lineup_example) set.seed(4) cal_post <- calibration_bayes( lineup_example, confidence_bins = c(0, 60, 80, 100), alpha = 0.5, S = 1000 ) cal_post
plot(cal_post, metric = "C") plot(cal_post, metric = "NRI")
sdt_compare() compares two conditions using summary SDT counts. It draws from
posterior hit and false-alarm rates, converts each draw to (d'), and reports
the posterior for (\Delta d' = d'_A - d'_B).
set.seed(5) sdt_post <- sdt_compare( hits_A = 45, misses_A = 55, fas_A = 12, crs_A = 88, hits_B = 38, misses_B = 62, fas_B = 10, crs_B = 90, label_A = "Procedure A", label_B = "Procedure B", alpha = 0.5, S = 1000 ) sdt_post
plot(sdt_post)
These Bayesian functions complement, rather than replace, the frequentist and bootstrap functions elsewhere in the package:
For substantive reports, run enough posterior draws for stable summaries and
check sensitivity to the prior concentration, for example alpha = 0.1, 0.5,
and 1.
sensitivity <- do.call(rbind, lapply(c(0.1, 0.5, 1.0), function(a) { set.seed(11) post <- esize_T_bayes(lineup_tab, alpha = a, S = 2000) data.frame( `Prior alpha` = a, `Posterior median` = round(post$posterior_median, 3), `Lower` = round(post$credible_interval[["lower"]], 3), `Upper` = round(post$credible_interval[["upper"]], 3), check.names = FALSE ) })) knitr::kable( sensitivity, row.names = FALSE, caption = "Sensitivity of the E' posterior to prior concentration, nortje2012 lineup 1 (nominal size 8)." )
If the three rows agree closely, as they generally will once there are a few dozen choices, the prior is doing little and the result is driven by the data. Disagreement is a signal that the sample is too small for the data to dominate, and should be reported rather than resolved by picking a prior.
One definition of E′. Tredoux (1998) defined effective size through an
index of diversity, (I = 1 - \sum_i p_i^2), transformed back to a count of
members: (E' = 1/(1 - I)). Substituting (I) gives (E' = 1/\sum_i p_i^2), the
reciprocal of Simpson's concentration index, also known as the inverse Simpson
or inverse Herfindahl index. The two forms are algebraically identical; they
differ only in notation, and this package uses (1/\sum_i p_i^2) throughout.
What differs between functions is the input. esize_T() substitutes the
observed proportions (o_i/N); esize_T_bayes() applies the same formula to
each posterior draw of the true probabilities. The posterior median therefore
need not equal the classical point estimate, particularly in small samples, and
that gap reflects estimation, not a different definition. The bias-corrected
estimate from esize_T(..., unbiased = TRUE) is a third estimator of the same
quantity.
| Function | Model | Transformation applied to each draw |
|---|---|---|
| esize_T_bayes() | Dirichlet-multinomial on the k choice counts | (E' = 1/\sum_i p_i^2) |
| func_size_bayes() | Beta-binomial on suspect selections | (F = 1/p_{\text{suspect}}) |
| diag_ratio_T_bayes() | Independent beta-binomials for the two lineups | (DR = p_{\text{TP}} / p_{\text{TA}}) |
| calibration_bayes() | Beta posterior on accuracy within each confidence bin | C, O/U and NRI |
| sdt_compare() | Beta posteriors on hit and false-alarm rates | (\Delta d' = d'_A - d'_B) |
In each case the posterior of the transformed quantity is obtained by transforming draws, not by approximating the transformation. This is important for quantities like (DR), a ratio whose sampling distribution is markedly skewed: the posterior inherits that skew correctly, where a symmetric normal-theory interval would not.
alpha is the concentration parameter of the Dirichlet or beta prior, applied
symmetrically. alpha = 0.5 is the Jeffreys prior, alpha = 1 is uniform, and
smaller values place more weight near the boundaries. All are weak; with more
than a few dozen observations the likelihood dominates, which the sensitivity
check above is there to confirm.
The posteriors here are conjugate: a Dirichlet prior with multinomial counts
gives a Dirichlet posterior, and a beta prior with binomial counts gives a beta
posterior. Both are standard distributions with known forms, so the functions
draw from the exact posterior directly (Dirichlet draws by normalising
independent gamma draws, beta draws with rbeta()). Each draw is independent of
the others.
MCMC is needed only when a posterior cannot be sampled like this, typically because its normalising constant cannot be computed. A Markov chain is then constructed whose long-run distribution is the posterior; successive draws are correlated, early draws must be discarded as burn-in, and convergence has to be checked (for example with R-hat and the effective sample size). None of that applies here. (E'), (F) and the other summaries are fixed transformations of each draw, so the transformed values are also independent draws from their exact posteriors, even though those posteriors have no closed form.
The practical consequences are that there is no burn-in, thinning or convergence
diagnostic; the effective sample size is simply S; and Monte Carlo error falls
as (1/\sqrt{S}). The Monte Carlo standard error of a tail probability (p),
for example, is about (\sqrt{p(1-p)/S}): 0.016 for (p = 0.5) at
S = 1000, and 0.005 at S = 10000. S = 1000 is adequate for a median, but
tail probabilities and interval endpoints need more; use 10,000 or more where a
reported number depends on them. This simplicity rests on conjugacy:
non-conjugate priors or hierarchical extensions would bring MCMC back.
alpha, so before any choices are seen no member is expected to
draw more choices than another. That is a statement of prior ignorance, not a
claim that the foils are equally good in fact. The prior counts are added to
the observed choices, so with even a moderate number of mock witnesses the
observed differences between members dominate, and the posterior can show
that a lineup is very unequal. The symmetric prior does pull small samples
slightly towards equal probabilities, and so towards a larger E′, which is one
reason to report the sensitivity check above.diag_ratio_T_bayes() treats the two lineups as independent samples.sdt_compare() assumes equal-variance signal detection, as the classical
(d') does.::: {.hanging-indent}
Tredoux, C. G. (1998). Statistical inference on measures of lineup fairness. Law and Human Behavior, 22(2), 217-237.
:::
?esize_T_bayes, ?func_size_bayes, ?diag_ratio_T_bayes,
?calibration_bayes, ?sdt_compareAny scripts or data that you put into this service are public.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.