T1EpSceptical: Compute type-I error rate of the sceptical p-value

In florafauna/RStest: Design and Analysis of Replication Studies

Description

The type-I error rate of the sceptical p-value is computed for a specified level of replication success, the relative variance, and the alternative hypothesis.

Usage

T1EpSceptical(
  level,
  c,
  alternative = c("one.sided", "two.sided", "greater", "less"),
  type = c("golden", "nominal", "liberal", "controlled")
)

Arguments

level	Numeric vector of levels of replication success.
c	Numeric vector of variance ratios of the original and replication effect estimates. This is usually the ratio of the sample size of the replication study to the sample size of the original study.
alternative	Either "one.sided" (one.sided), "two.sided", "greater", or "less". If "one.sided", the type-I error rate is computed based on a one-sided assessment of replication success in the direction of the original effect estimate. If "two.sided", the type-I error rate is computed based on a two-sided assessment of replication success regardless of the direction of the original and replication effect estimate. If "greater" or "less", the type-I error rate is computed based on a one-sided assessment of replication success in the pre-specified direction of the original and replication effect estimate.
type	Type of recalibration. Can be either "golden" (default), "nominal" (no recalibration), "liberal", or "controlled". "golden" ensures that for an original study just significant at the specified level, replication success is only possible if the replication effect estimate is at least as large as the original one. See levelSceptical for details about recalibration types.

Details

T1EpSceptical is the vectorized version of .T1EpSceptical_. Vectorize is used to vectorize the function.

Value

The type-I error rate.

Author(s)

Samuel Pawel, Leonhard Held

References

Held, L. (2020). The harmonic mean chi-squared test to substantiate scientific findings. Journal of the Royal Statistical Society: Series C (Applied Statistics), 69, 697-708. doi: 10.1111/rssc.12410

Held, L., Micheloud, C., Pawel, S. (2021). The assessment of replication success based on relative effect size. https://arxiv.org/abs/2009.07782

See Also

pSceptical, levelSceptical, PPpSceptical

Examples

## compare type-I error rate for different levels of replication success
levels <- c("nominal" = levelSceptical(level = 0.025, type = "nominal"),
            "liberal" = levelSceptical(level = 0.025, type = "liberal"),
            "controlled" = levelSceptical(level = 0.025, type = "controlled"),
            "golden" = levelSceptical(level = 0.025, type = "golden"))
c <- seq(0.2, 5, by = 0.05)
t1 <- sapply(X = levels, FUN = function(l) {
  T1EpSceptical(level = l, c = c, alternative = "one.sided", type = "nominal")
})
matplot(x = c, y = t1*100, type = "l", lty = 1, lwd = 2, las = 1, log = "x",
        xlab = bquote(italic(c)), ylab = "Type-I error (%)", xlim = c(0.2, 5))
legend("topright", legend = names(levels), lty = 1, lwd = 2, col = seq_along(levels))

## check that one.sided controlled level controls type-I error rate for c = 1 
## at alpha = 0.05*0.025 = 0.00125
T1EpSceptical(level = levelSceptical(level = 0.025, alternative = "one.sided", 
                                     type = "controlled"), 
              c = 1, alternative = "one.sided",  type = "nominal")

florafauna/RStest documentation built on Dec. 20, 2021, 8:44 a.m.