replicateTTests: Simulate Early Stopping Experiments
In safestats: Safe Anytime-Valid Inference

replicateTTests

R Documentation

Simulate Early Stopping Experiments

Description

Simulate multiple data sets to show the effects of optional testing for safe (and frequentist) tests.

Usage

replicateTTests(
  nPlan,
  deltaTrue,
  muGlobal = 0,
  sigmaTrue = 1,
  paired = FALSE,
  alternative = c("twoSided", "greater", "less"),
  lowN = 3,
  nSim = 1000L,
  alpha = 0.05,
  beta = 0.2,
  safeOptioStop = TRUE,
  parameter = NULL,
  freqOptioStop = FALSE,
  nPlanFreq = NULL,
  logging = TRUE,
  seed = NULL,
  pb = TRUE,
  ...
)

Arguments

`nPlan`	vector of max length 2 representing the planned sample sizes.
`deltaTrue`	numeric, the value of the true standardised effect size (test-relevant parameter).
`muGlobal`	numeric, the true global mean of a paired or two-sample t-test. Its value should not matter for the test. This parameter is treated as a nuisance.
`sigmaTrue`	numeric > 0,the true standard deviation of the data. Its value should not matter for the test.This parameter treated is treated as a nuisance.
`paired`	logical, if `TRUE` then paired t-test.
`alternative`	a character string specifying the alternative hypothesis must be one of "twoSided" (default), "greater" or "less".
`lowN`	integer that defines the smallest n of our search space for n.
`nSim`	the number of replications, that is, experiments with max samples nPlan.
`alpha`	numeric in (0, 1) that specifies the tolerable type I error control –independent of n– that the designed test has to adhere to. Note that it also defines the rejection rule e10 > 1/alpha.
`beta`	numeric in (0, 1) that specifies the tolerable type II error control necessary to calculate both the sample sizes and deltaS, which defines the test. Note that 1-beta defines the power.
`safeOptioStop`	logical, `TRUE` implies that optional stopping simulation is performed for the safe test.
`parameter`	numeric, the safe test defining parameter, i.e., deltaS (use designSafeT to find this).
`freqOptioStop`	logical, `TRUE` implies that optional stopping simulation is performed for the frequentist test.
`nPlanFreq`	the frequentist sample size(s) to plan for. Acquired from `designFreqT()`.
`logging`	logical, if `TRUE`, then return the simulated data.
`seed`	To set the seed for the simulated data.
`pb`	logical, if `TRUE`, then show progress bar.
`...`	further arguments to be passed to or from methods.

Value

Returns an object of class "safeTSim". An object of class "safeTSim" is a list containing at least the following components:

nPlan: the planned sample size(s).
deltaTrue: the value of the true standardised effect size (test-relevant parameter) provided by the user.
muGlobal: the true global mean of a paired or two-sample t-test (nuisance parameter) provided by the user.
paired: if TRUE then paired t-test.
alternative: any of "twoSided", "greater", "less" provided by the user.
lowN: the smallest number of samples (first group) at which monitoring of the tests begins.
nSim: the number of replications of the experiment.
alpha: the tolerable type I error provided by the user.
beta: the tolerable type II error provided by the user.
testType: any of "oneSample", "paired", "twoSample" provided by the user.
parameter: the parameter (point prior) used in the safe test derived from the design. Acquired from designSafeT().
nPlanFreq: the frequentist planned sample size(s). Acquired from designFreqT()
safeSim: list with the simulation results of the safe test under optional stopping.
freqSim: list with the simulation results of the frequentist test under optional stopping.

Examples


# Design safe test
alpha <- 0.05
beta <- 0.20
designObj <- designSafeT(1, alpha=alpha, beta=beta)

# Design frequentist test
freqObj <- designFreqT(1, alpha=alpha, beta=beta)

# Simulate under the alternative with deltaTrue=deltaMin
simResults <- replicateTTests(nPlan=designObj$nPlan, deltaTrue=1, parameter=designObj$parameter,
                              nPlanFreq=freqObj$nPlan, beta=beta, nSim=250)

# Should be about 1-beta
simResults$safeSim$powerAtN1Plan

# This is higher due to optional stopping
simResults$safeSim$powerOptioStop

# Optional stopping allows us to do better than n1PlanFreq once in a while
simResults$safeSim$probLeqN1PlanFreq
graphics::hist(simResults$safeSim$allN, main="Histogram of stopping times", xlab="n1",
               breaks=seq.int(designObj$nPlan[1]))

# Simulate under the alternative with deltaTrue > deltaMin
simResults <- replicateTTests(nPlan=designObj$nPlan, deltaTrue=1.5, parameter=designObj$parameter,
                              nPlanFreq=freqObj$nPlan, beta=beta, nSim=250)

# Should be larger than 1-beta
simResults$safeSim$powerAtN1Plan

# This is even higher due to optional stopping
simResults$safeSim$powerOptioStop

# Optional stopping allows us to do better than n1PlanFreq once in a while
simResults$safeSim$probLeqN1PlanFreq
graphics::hist(simResults$safeSim$allN, main="Histogram of stopping times", xlab="n1",
               breaks=seq.int(designObj$nPlan[1]))

# Under the null deltaTrue=0
simResults <- replicateTTests(nPlan=designObj$nPlan, deltaTrue=0, parameter=designObj$parameter,
                              nPlanFreq=freqObj$nPlan, freqOptioStop=TRUE, beta=beta, nSim=250)

# Should be lower than alpha, because if the null is true, P(S > 1/alpha) < alpha for all n
simResults$safeSim$powerAtN1Plan

# This is a bit higher due to optional stopping, but if the null is true,
# then still P(S > 1/alpha) < alpha for all n
simResults$safeSim$powerOptioStop

# Should be lowr than alpha, as the experiment is performed as was planned
simResults$freqSim$powerAtN1Plan

# This is larger than alpha, due to optional stopping.
simResults$freqSim$powerOptioStop
simResults$freqSim$powerOptioStop > alpha

safestats documentation built on Nov. 24, 2022, 5:07 p.m.