View source: R/simulate_lineup_data.R
| simulate_power_analysis | R Documentation |
Estimates statistical power to detect a difference in discriminability between two lineup conditions, by simulating both conditions at a range of sample sizes and recording how often the difference is detected.
simulate_power_analysis(
sample_sizes,
d_prime,
d_prime_2 = 0,
n_simulations = 1000,
test_function = NULL,
alpha = 0.05,
lineup_size = 6,
n_bootstrap = 200,
seed = NULL,
verbose = TRUE,
d_prime_null = NULL,
...
)
sample_sizes |
Integer vector of sizes to test (e.g., c(50, 100, 200)). A size n means n target-present and n target-absent witnesses in each of the two simulated conditions: 2n witnesses per condition and 4n in the whole simulated experiment. |
d_prime |
Numeric. Discriminability of the condition of interest. |
d_prime_2 |
Numeric. Generating discriminability of the second
(comparison) condition. It is not the null hypothesis: the test asks
whether the two conditions differ. The default, 0, is a condition with no
memory of the culprit, so it asks how often the procedure can be shown to
discriminate better than chance. For a comparison of two working
procedures, set it to a realistic value (for example |
n_simulations |
Integer. Number of simulated studies per sample size
(default = 1000). Power is a proportion estimated from these, so its own
standard error is roughly |
test_function |
Function of two dataframes returning a p-value. Defaults
to a bootstrap comparison of partial AUC via |
alpha |
Numeric. Significance level (default = 0.05) |
lineup_size |
Integer. Lineup size for both conditions (default = 6) |
n_bootstrap |
Integer. Bootstrap resamples used by the default test
(default = 200). This is the dominant cost: the default test is run
|
seed |
Optional integer seed for reproducibility. |
verbose |
Logical. Report progress (default = TRUE). |
d_prime_null |
Deprecated name for |
... |
Additional arguments passed to simulate_lineup_data(), for example
|
Power is the probability that a study of a given size detects an effect that is really there. Estimating it by simulation means specifying the effect, generating many studies of that size, applying the analysis you intend to use, and counting how often it reaches significance.
That requires a second condition to compare against, which
d_prime_2 supplies; the null hypothesis is that the two conditions do
not differ. Simulating a single condition and computing a statistic from it is
not a power analysis, because no hypothesis is being tested.
Each simulated study samples witnesses for one lineup set (one fixed foil
structure) and gives every witness the same d' and criterion. The
power is therefore for generalising over witnesses to these materials.
Generalising over culprits and lineups as well needs sets sampled as units,
and with few sets the effective sample size for that question is closer to
the number of sets (Wells & Windschitl, 1999; Westfall, Kenny, & Judd,
2014).
Power is estimated from n_simulations studies, so it has Monte Carlo
error, and each default test rests on n_bootstrap resamples. An
estimated power of 1 (standard error 0) means only that the true power is
high: by the rule of three, with 30 simulations it is above about
1 - 3/30 = 0.90 with 95% confidence.
A dataframe of class power_analysis with columns:
sample_size: n, the number of target-present (and, separately, target-absent) witnesses per condition
n_total: Witnesses in the whole simulated experiment, 4n
power: Proportion of simulated studies reaching significance
power_se: Standard error of that proportion
mean_effect: Mean observed difference in pAUC
sd_effect: Standard deviation of the observed difference
n_valid: Simulations that produced a usable test
n_failed: Attempted simulations without a usable test
n_attempted: All attempted simulations
Power and its Monte Carlo standard error are conditional on usable tests. Failures are retained in the attempted count and trigger a warning; compare failure rates as well as conditional power when assessing a design.
Wells, G. L., & Windschitl, P. D. (1999). Stimulus sampling and social psychological experimentation. Personality and Social Psychology Bulletin, 25, 1115–1125.
Westfall, J., Kenny, D. A., & Judd, C. M. (2014). Statistical power and optimal design in experiments in which samples of participants respond to samples of stimuli. Journal of Experimental Psychology: General, 143, 2020–2045.
compare_pauc, simulate_lineup_data
# Power to show discriminability better than chance
power_res <- simulate_power_analysis(
sample_sizes = c(50, 100),
d_prime = 1.5,
c_criterion = 1.23,
n_simulations = 8,
n_bootstrap = 50,
conf_levels = 5,
seed = 123
)
print(power_res)
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.