In dsrobertson/onlineFDR: Online error rate control
knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>"
)
options(tinytex.verbose = TRUE)
library(onlineFDR)
sample.df <- data.frame(
id = c('A15432', 'B90969', 'C18705', 'B49731', 'E99902',
'C38292', 'A30619', 'D46627', 'E29198', 'A41418',
'D51456', 'C88669', 'E03673', 'A63155', 'B66033'),
date = as.Date(c(rep("2014-12-01",3),
rep("2015-09-21",5),
rep("2016-05-19",2),
"2016-11-12",
rep("2017-03-27",4))),
pval = c(2.90e-14, 0.06743, 0.01514, 0.08174, 0.00171,
3.61e-05, 0.79149, 0.27201, 0.28295, 7.59e-08,
0.69274, 0.30443, 0.000487, 0.72342, 0.54757))
set.seed(1)
FDR Control
Consider a sequence of hypotheses $H_1, H_2, H_3, \ldots$ that arrive
sequentially in a stream, with corresponding $p$-values
$(p_1, p_2, p_3, \ldots)$. A testing procedure provides a sequence of adjusted
significance thresholds $\alpha_i$, with corresponding decision rule:
[
R_i = \left{\begin{array}{ccc}
1 & \text{if } p_i \leq \alpha_i & (\text{reject } H_i)\
0 & \text{otherwise } & (\text{accept } H_i)
\end{array}\right}
]
In online testing, the significance thresholds can only be functions of
the prior decisions, i.e. $\alpha_i = \alpha_i(R_1, R_2, \ldots, R_{i-1})$.
Javanmard and Montanari (2015, 2018) proposed two procedures for online control.
The first is LOND, which stands for (significance) Levels based On Number of
Discoveries. The second is LORD, which stands for (significance) Levels based On
Recent Discovery. LORD was subsequently extended by Ramdas et al. (2017).
Ramdas et al. (2018) also proposed the SAFFRON procedure, which provides an
adaptive method of online FDR control, which includes a variant of
Alpha-investing. Finally, Tian & Ramdas (2019) proposed the ADDIS procedure
as an improvement of SAFFRON in the presence of conservative nulls.
LOND {#LOND}
The LOND procedure controls the FDR for independent or positively dependent
(PRDS) $p$-values. Given an overall significance level $\alpha$, we choose a
sequence of non-negative numbers $\beta = (\beta_i){i \in \mathbb{N}}$ such
that they sum to $\alpha$. The values of the adjusted significance thresholds
$\alpha_i$ are chosen as follows: [ \alpha_i = \beta_i (D(i-1) + 1) ] where
$D(n) = \sum{i=1}^n R_i$ denotes the number of discoveries (i.e. rejections) in
the first $n$ hypotheses tested.
LOND can be adjusted to also control FDR under arbitrarily dependent $p$-values.
To do so, it is modified with $\tilde{\beta}i = \beta_i/H(i)$ in place of
$\beta_i$, where $H(i) = \sum{j=1}^i \frac{1}{j}$ is the $i$-th harmonic
number. Note that this leads to a substantial loss in power compared to the
unadjusted LOND procedure. The correction factor is similar to the classical one
used by Benjamini and Yekutieli (2001), except that in this case the $i$-th
hypothesis among $N$ is penalised by a factor of $H(i)$ to give consistent
results across time (as compared to a factor $H(N)$ for the Benjamini and
Yekutieli method).
The default sequence of $\beta$ is given by
[\beta_j = C \alpha \frac{\log(\max(j, 2))}{j e^{\sqrt{\log j}}}] where
$C \approx 0.07720838$, as proposed by Javanmard and Montanari (2018)
equation 31.
LORD {#LORD}
The LORD procedure controls the FDR for independent $p$-values. We first fix a
sequence of non-negative numbers $\gamma = (\gamma_i){i \in \mathbb{N}}$ such
that $\gamma_i \geq \gamma_j$ for
$i \leq j$ and $\sum{i=1}^{\infty} \gamma_i = 1$. At each time $i$, let
$\tau_i$ be the last time a discovery was made before $i$: [
\tau_i = \max \left{ l \in {1, \ldots, i-1} : R_l = 1\right}
]
LORD depends on constants $w_0$ and $b_0$, where $w_0 \geq 0$ represents the
initial 'wealth' of the procedure and $b_0 > 0$ represents the 'payout' for
rejecting a hypothesis. We require $w_0+b_0 \leq \alpha$ for FDR control to
hold.
Javanmard and Montanari (2018) presented three different versions of LORD, which
have different definitions of the adjusted significance thresholds $\alpha_i$.
Versions 1 and 2 have since been superseded by the LORD++ procedure of
Ramdas et al. (2017), so we do not describe them here.
-
LORD++: The significance thresholds for LORD++ are chosen as follows: [
\alpha_i = \gamma_i w_0 + (\alpha - w_0) \gamma_{i-\tau_1} +
\alpha \sum_{j : \tau_j < i, \tau_j \neq \tau_1} \gamma_{i - \tau_j}
]
-
LORD 3: The significance thresholds depend on the time of the last
discovery time and the wealth accumulated at that time, with
[
\alpha_i = \gamma_{i - \tau_i} W(\tau_i)
]
where $\tau_1 = 0$. Here ${W(j)}_{j \geq 0}$ represents the 'wealth' available
at time $j$, and is defined recursively:
[
\begin{aligned}
W(0) &= w_0 \
W(j) &= W(j-1) - \alpha_{j-1} + b_0 R_j
\end{aligned}
]
- D-LORD: This is equivalent to the LORD++ procedure with discarding. Given
a discarding threshold $\tau \in (0,1)$ and initial wealth $w_0 \leq \tau\alpha$
the significance thresholds are chosen as follows: [
\alpha_t = \min{\tau, \tilde{\alpha}t}
] where [
\tilde{\alpha}_t = w_0 \gamma{S^t} +
(\tau\alpha - w_0)\gamma_{S^t - \kappa_1^} +
\tau\alpha \sum_{j \geq 2} \gamma_{S^t - \kappa_j^}
] and [
\kappa_j = \min{i \in [t-1] : \sum_{k \leq i}
1 {p_k \leq \alpha_k} \geq j}, \;
\kappa_j^* = \sum_{i \leq \kappa_j} 1 {p_i \leq \tau }, \;
S^t = \sum_{i < t} 1 {p_i \leq \tau }
]
LORD++ is an instance of a monotone rule, and provably controls the FDR for
independent p-values provided $w_0 \leq \alpha$. LORD 3 is a non-monotone rule,
and FDR control is only demonstrated empirically. In some scenarios with large
$N$, LORD 3 will have a slightly higher power than LORD++ (see Robertson et
al., 2018), but since it is a non-monotone rule we would recommend using LORD++
(which is the default), especially since it also has a provable guarantee of FDR
control.
In all versions, the default sequence of $\gamma$ is
given by [\gamma_j = C \frac{\log(\max(j, 2))}{j e^{\sqrt{\log j}}}]
where $C \approx 0.07720838$, as proposed by Javanmard and Montanari (2018)
equation 31.
Javanmard and Montanari (2018) also proposed an adjusted version of LORD that
is valid for arbitrarily dependent p-values. Similarly to LORD 3, the adjusted
significance thresholds are set equal to [ \alpha_i = \xi_i W(\tau_i)] where
(assuming $w_0 \leq b_0$),
$\sum_{j=1}^{\infty} \xi_i (1 + \log(j)) \leq \alpha / b_0$
The default sequence of $\xi$ is given by
[ \xi_j = \frac{C \alpha }{b_0 j \log(\max(j, 2))^3}]
where $C \approx 0.139307$.
Note that allowing for dependent p-values can lead to a substantial loss in
power compared with the LORD procedures described above.
SAFFRON {#SAFFRON}
The SAFFRON procedure controls the FDR for independent p-values, and was
proposed by Ramdas et al. (2018). The algorithm is based on an estimate of the
proportion of true null hypotheses. More precisely, SAFFRON sets the adjusted
test levels based on an estimate of the amount of alpha-wealth that is allocated
to testing the true null hypotheses.
SAFFRON depends on constants $w_0$ and $\lambda$, where $w_0$ satisfies
$0 \leq w_0 \leq \alpha$ and represents the initial 'wealth' of the
procedure, and $\lambda \in (0,1)$ represents the threshold for a 'candidate'
hypothesis. A 'candidate' refers to p-values smaller than $\lambda$, since
SAFFRON will never reject a p-value larger than $\lambda$. These candidates can
be thought of as the hypotheses that are a-priori more likely to be non-null.
The SAFFRON procedure runs as follows:
-
At each time $t$, define the number of candidates after the $j$-th rejection
as [ C_{j+} = C_{j+}(t) = \sum_{i = \tau_j + 1}^{t-1} C_i]
where $C_t = 1{p_t \leq \lambda }$ is the indicator for candidacy.
-
SAFFRON starts with $\alpha_1 = \min{(1 - \lambda)\gamma_1 w_0, \lambda}$. Subsequent
test levels are chosen as $\alpha_t = \min{ \lambda, \tilde{\alpha}t}$, where
[
\tilde{\alpha}_t = (1 - \lambda) [w_0 \gamma{t-C_{0+}} +
(\alpha - w_0)\gamma_{t-\tau_1-C_{1+}} +
\alpha \sum_{j \geq 2}
\gamma_{t - \tau_j- C_{j+}}]
]
The default sequence of $\gamma$ for SAFFRON is
given by $\gamma_j \propto j^{-1.6}$.
Alpha-investing {#AlphaInvesting}
Ramdas et al. (2018) proposed a variant of the Alpha-investing algorithm of
Foster and Stine (2008) that guarantees FDR control for independent p-values.
This procedure uses SAFFRON's update rule with the constant $\lambda$
replaced by a sequence $\lambda_i = \alpha_i$. This is also equivalent to using
the ADDIS algorithm (see below) with $\tau = 1$ and $\lambda_i = \alpha_i$.
ADDIS {#ADDIS}
The ADDIS procedure controls the FDR for independent p-values, and was proposed
by Tian & Ramdas (2019). The algorithm compensates for the power loss of SAFFRON
with conservative nulls, by including both adaptivity in the fraction of null
hypotheses (like SAFFRON) and the conservativeness of nulls (unlike SAFFRON).
ADDIS depends on constants $w_0, \lambda$ and $\tau$. $w_0$ represents the
initial `wealth' of the procedure and satisfies $0 \leq w_0 \leq \alpha$. $\tau \in (0,1]$ represents the threshold for a hypothesis to be
selected for testing: p-values greater than $\tau$ are implicitly 'discarded' by
the procedure. Finally, $\lambda \in [0,\tau)$ sets the threshold for a p-value to
be a candidate for rejection: ADDIS will never reject a p-value larger than $\lambda$.
The significance thresholds for ADDIS are chosen as follows: [
\alpha_t = \min{\lambda, \tilde{\alpha}t}
] where [
\tilde{\alpha}_t = (\tau - \lambda)[w_0 \gamma{S^t-C_{0+}} +
(\alpha - w_0)\gamma_{S^t - \kappa_1^-C_{1+}} +
\alpha \sum_{j \geq 2} \gamma_{S^t - \kappa_j^ - C_{j+}}
] and [
\kappa_j = \min{i \in [t-1] : \sum_{k \leq i}
1 {p_k \leq \alpha_k} \geq j}, \;
\kappa_j^* = \sum_{i \leq \kappa_j} 1 {p_i \leq \tau }, \;
S^t = \sum_{i < t} 1 {p_i \leq \tau }, \;
C_{j+} = \sum_{i = \kappa_j + 1}^{t-1} 1{p_i \leq \lambda}
]
The default sequence of $\gamma$ for ADDIS is the same as for SAFFRON given
here.
FWER Control
Alpha-spending {#Alpha-spending}
The Alpha-spending procedure controls the FWER for a potentially infinite stream
of p-values using a Bonferroni-like test. Given an overall significance level
$\alpha$, the significance thresholds are chosen as
[\alpha_i = \alpha \gamma_i]
where $\sum_{i=1}^{\infty} \gamma_i = 1$ and $\gamma_i \geq 0$. The procedure
strongly controls the FWER for arbitrarily dependent p-values.
Note that the procedure also controls the generalised familywise error rate
(k-FWER) for $k > 1$ if $\alpha$ is replaced by $\min(1,k\alpha)$.
The default sequence of $\gamma$ is the same as that for $\xi$ for LORD given
here.
Online Fallback {#onlineFallback}
The online fallback procedure of Tian & Ramdas (2019b) provides a uniformly more
powerful method than Alpha-spending, by saving the significance level of a
previous rejection. More specifically, online fallback tests hypothesis $H_i$ at
level [\alpha_i = \alpha \gamma_i + R_{i-1} \alpha_{i-1}] where $R_i = 1{p_i
\leq \alpha_i}$ denotes a rejected hypothesis. The procedure strongly controls
the FWER for arbitrarily dependent p-values.
The default sequence of $\gamma$ is the same as that for $\xi$ for LORD given
here.
ADDIS-spending {#ADDIS-spending}
The ADDIS-spending procedure strongly controls the FWER for independent
p-values, and was proposed by Tian & Ramdas (2021). The procedure compensates
for the power loss of Alpha-spending, by including both adapativity in the
fraction of null hypotheses and the conservativeness of nulls.
ADDIS depends on constants $\lambda$ and $\tau$, where $\lambda < \tau$. Here
$\tau \in (0,1)$ represents the threshold for a hypothesis to be selected for
testing: p-values greater than $\tau$ are implicitly `discarded' by the
procedure, while $\lambda \in (0,1)$ sets the threshold for a p-value to be a
candidate for rejection: ADDIS-spending will never reject a p-value larger than
$\lambda$.
Note that the procedure controls the generalised familywise error rate (k-FWER)
for $k > 1$ if $\alpha$ is replaced by $\min(1,k\alpha)$. Tian and Ramdas
(2019b) also presented a version for handling local dependence, see the Section
on Asynchronous testing below.
The default sequence of $\gamma$ for ADDIS-spending is the same as for SAFFRON
given here.
Accounting for dependent p-values
As noted above, the LORD, SAFFRON, ADDIS and ADDIS-spending procedures assume
independent p-values, while the LOND procedure is also valid under positive
dependencies (like the Benjamini-Hochberg method, see below). Adjusted
versions of LOND and LORD available for arbitrarily dependent p-values.
Alpha-spending and online fallback also control the FWER and FDR for arbitrarily
dependent p-values.
By way of comparison, the usual Benjamini-Hochberg method for controlling
the FDR assumes that the p-values are positively dependent (PRDS). As an
example, the PRDS is satisfied for multivariate normal test statistics with a
positive correlation matrix). See Benjamini & Yekutieli (2001) for further
technical details.
dsrobertson/onlineFDR documentation built on April 21, 2023, 8:17 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
knitr::opts_chunk$set( collapse = TRUE, comment = "#>" ) options(tinytex.verbose = TRUE) library(onlineFDR) sample.df <- data.frame( id = c('A15432', 'B90969', 'C18705', 'B49731', 'E99902', 'C38292', 'A30619', 'D46627', 'E29198', 'A41418', 'D51456', 'C88669', 'E03673', 'A63155', 'B66033'), date = as.Date(c(rep("2014-12-01",3), rep("2015-09-21",5), rep("2016-05-19",2), "2016-11-12", rep("2017-03-27",4))), pval = c(2.90e-14, 0.06743, 0.01514, 0.08174, 0.00171, 3.61e-05, 0.79149, 0.27201, 0.28295, 7.59e-08, 0.69274, 0.30443, 0.000487, 0.72342, 0.54757)) set.seed(1)
FDR Control
Consider a sequence of hypotheses $H_1, H_2, H_3, \ldots$ that arrive sequentially in a stream, with corresponding $p$-values $(p_1, p_2, p_3, \ldots)$. A testing procedure provides a sequence of adjusted significance thresholds $\alpha_i$, with corresponding decision rule: [ R_i = \left{\begin{array}{ccc} 1 & \text{if } p_i \leq \alpha_i & (\text{reject } H_i)\ 0 & \text{otherwise } & (\text{accept } H_i) \end{array}\right} ]
In online testing, the significance thresholds can only be functions of the prior decisions, i.e. $\alpha_i = \alpha_i(R_1, R_2, \ldots, R_{i-1})$.
Javanmard and Montanari (2015, 2018) proposed two procedures for online control. The first is LOND, which stands for (significance) Levels based On Number of Discoveries. The second is LORD, which stands for (significance) Levels based On Recent Discovery. LORD was subsequently extended by Ramdas et al. (2017). Ramdas et al. (2018) also proposed the SAFFRON procedure, which provides an adaptive method of online FDR control, which includes a variant of Alpha-investing. Finally, Tian & Ramdas (2019) proposed the ADDIS procedure as an improvement of SAFFRON in the presence of conservative nulls.
LOND {#LOND}
The LOND procedure controls the FDR for independent or positively dependent (PRDS) $p$-values. Given an overall significance level $\alpha$, we choose a sequence of non-negative numbers $\beta = (\beta_i){i \in \mathbb{N}}$ such that they sum to $\alpha$. The values of the adjusted significance thresholds $\alpha_i$ are chosen as follows: [ \alpha_i = \beta_i (D(i-1) + 1) ] where $D(n) = \sum{i=1}^n R_i$ denotes the number of discoveries (i.e. rejections) in the first $n$ hypotheses tested.
LOND can be adjusted to also control FDR under arbitrarily dependent $p$-values. To do so, it is modified with $\tilde{\beta}i = \beta_i/H(i)$ in place of $\beta_i$, where $H(i) = \sum{j=1}^i \frac{1}{j}$ is the $i$-th harmonic number. Note that this leads to a substantial loss in power compared to the unadjusted LOND procedure. The correction factor is similar to the classical one used by Benjamini and Yekutieli (2001), except that in this case the $i$-th hypothesis among $N$ is penalised by a factor of $H(i)$ to give consistent results across time (as compared to a factor $H(N)$ for the Benjamini and Yekutieli method).
The default sequence of $\beta$ is given by [\beta_j = C \alpha \frac{\log(\max(j, 2))}{j e^{\sqrt{\log j}}}] where $C \approx 0.07720838$, as proposed by Javanmard and Montanari (2018) equation 31.
LORD {#LORD}
The LORD procedure controls the FDR for independent $p$-values. We first fix a sequence of non-negative numbers $\gamma = (\gamma_i){i \in \mathbb{N}}$ such that $\gamma_i \geq \gamma_j$ for $i \leq j$ and $\sum{i=1}^{\infty} \gamma_i = 1$. At each time $i$, let $\tau_i$ be the last time a discovery was made before $i$: [ \tau_i = \max \left{ l \in {1, \ldots, i-1} : R_l = 1\right} ]
LORD depends on constants $w_0$ and $b_0$, where $w_0 \geq 0$ represents the initial 'wealth' of the procedure and $b_0 > 0$ represents the 'payout' for rejecting a hypothesis. We require $w_0+b_0 \leq \alpha$ for FDR control to hold.
Javanmard and Montanari (2018) presented three different versions of LORD, which have different definitions of the adjusted significance thresholds $\alpha_i$. Versions 1 and 2 have since been superseded by the LORD++ procedure of Ramdas et al. (2017), so we do not describe them here.
-
LORD++: The significance thresholds for LORD++ are chosen as follows: [ \alpha_i = \gamma_i w_0 + (\alpha - w_0) \gamma_{i-\tau_1} + \alpha \sum_{j : \tau_j < i, \tau_j \neq \tau_1} \gamma_{i - \tau_j} ]
-
LORD 3: The significance thresholds depend on the time of the last discovery time and the wealth accumulated at that time, with [ \alpha_i = \gamma_{i - \tau_i} W(\tau_i) ] where $\tau_1 = 0$. Here ${W(j)}_{j \geq 0}$ represents the 'wealth' available at time $j$, and is defined recursively:
[ \begin{aligned} W(0) &= w_0 \ W(j) &= W(j-1) - \alpha_{j-1} + b_0 R_j \end{aligned} ]
- D-LORD: This is equivalent to the LORD++ procedure with discarding. Given a discarding threshold $\tau \in (0,1)$ and initial wealth $w_0 \leq \tau\alpha$ the significance thresholds are chosen as follows: [ \alpha_t = \min{\tau, \tilde{\alpha}t} ] where [ \tilde{\alpha}_t = w_0 \gamma{S^t} + (\tau\alpha - w_0)\gamma_{S^t - \kappa_1^} + \tau\alpha \sum_{j \geq 2} \gamma_{S^t - \kappa_j^} ] and [ \kappa_j = \min{i \in [t-1] : \sum_{k \leq i} 1 {p_k \leq \alpha_k} \geq j}, \; \kappa_j^* = \sum_{i \leq \kappa_j} 1 {p_i \leq \tau }, \; S^t = \sum_{i < t} 1 {p_i \leq \tau } ]
LORD++ is an instance of a monotone rule, and provably controls the FDR for independent p-values provided $w_0 \leq \alpha$. LORD 3 is a non-monotone rule, and FDR control is only demonstrated empirically. In some scenarios with large $N$, LORD 3 will have a slightly higher power than LORD++ (see Robertson et al., 2018), but since it is a non-monotone rule we would recommend using LORD++ (which is the default), especially since it also has a provable guarantee of FDR control.
In all versions, the default sequence of $\gamma$ is given by [\gamma_j = C \frac{\log(\max(j, 2))}{j e^{\sqrt{\log j}}}] where $C \approx 0.07720838$, as proposed by Javanmard and Montanari (2018) equation 31.
Javanmard and Montanari (2018) also proposed an adjusted version of LORD that is valid for arbitrarily dependent p-values. Similarly to LORD 3, the adjusted significance thresholds are set equal to [ \alpha_i = \xi_i W(\tau_i)] where (assuming $w_0 \leq b_0$), $\sum_{j=1}^{\infty} \xi_i (1 + \log(j)) \leq \alpha / b_0$
The default sequence of $\xi$ is given by [ \xi_j = \frac{C \alpha }{b_0 j \log(\max(j, 2))^3}] where $C \approx 0.139307$.
Note that allowing for dependent p-values can lead to a substantial loss in power compared with the LORD procedures described above.
SAFFRON {#SAFFRON}
The SAFFRON procedure controls the FDR for independent p-values, and was proposed by Ramdas et al. (2018). The algorithm is based on an estimate of the proportion of true null hypotheses. More precisely, SAFFRON sets the adjusted test levels based on an estimate of the amount of alpha-wealth that is allocated to testing the true null hypotheses.
SAFFRON depends on constants $w_0$ and $\lambda$, where $w_0$ satisfies $0 \leq w_0 \leq \alpha$ and represents the initial 'wealth' of the procedure, and $\lambda \in (0,1)$ represents the threshold for a 'candidate' hypothesis. A 'candidate' refers to p-values smaller than $\lambda$, since SAFFRON will never reject a p-value larger than $\lambda$. These candidates can be thought of as the hypotheses that are a-priori more likely to be non-null.
The SAFFRON procedure runs as follows:
-
At each time $t$, define the number of candidates after the $j$-th rejection as [ C_{j+} = C_{j+}(t) = \sum_{i = \tau_j + 1}^{t-1} C_i] where $C_t = 1{p_t \leq \lambda }$ is the indicator for candidacy.
-
SAFFRON starts with $\alpha_1 = \min{(1 - \lambda)\gamma_1 w_0, \lambda}$. Subsequent test levels are chosen as $\alpha_t = \min{ \lambda, \tilde{\alpha}t}$, where [ \tilde{\alpha}_t = (1 - \lambda) [w_0 \gamma{t-C_{0+}} + (\alpha - w_0)\gamma_{t-\tau_1-C_{1+}} + \alpha \sum_{j \geq 2} \gamma_{t - \tau_j- C_{j+}}] ]
The default sequence of $\gamma$ for SAFFRON is given by $\gamma_j \propto j^{-1.6}$.
Alpha-investing {#AlphaInvesting}
Ramdas et al. (2018) proposed a variant of the Alpha-investing algorithm of Foster and Stine (2008) that guarantees FDR control for independent p-values. This procedure uses SAFFRON's update rule with the constant $\lambda$ replaced by a sequence $\lambda_i = \alpha_i$. This is also equivalent to using the ADDIS algorithm (see below) with $\tau = 1$ and $\lambda_i = \alpha_i$.
ADDIS {#ADDIS}
The ADDIS procedure controls the FDR for independent p-values, and was proposed by Tian & Ramdas (2019). The algorithm compensates for the power loss of SAFFRON with conservative nulls, by including both adaptivity in the fraction of null hypotheses (like SAFFRON) and the conservativeness of nulls (unlike SAFFRON).
ADDIS depends on constants $w_0, \lambda$ and $\tau$. $w_0$ represents the initial `wealth' of the procedure and satisfies $0 \leq w_0 \leq \alpha$. $\tau \in (0,1]$ represents the threshold for a hypothesis to be selected for testing: p-values greater than $\tau$ are implicitly 'discarded' by the procedure. Finally, $\lambda \in [0,\tau)$ sets the threshold for a p-value to be a candidate for rejection: ADDIS will never reject a p-value larger than $\lambda$.
The significance thresholds for ADDIS are chosen as follows: [ \alpha_t = \min{\lambda, \tilde{\alpha}t} ] where [ \tilde{\alpha}_t = (\tau - \lambda)[w_0 \gamma{S^t-C_{0+}} + (\alpha - w_0)\gamma_{S^t - \kappa_1^-C_{1+}} + \alpha \sum_{j \geq 2} \gamma_{S^t - \kappa_j^ - C_{j+}} ] and [ \kappa_j = \min{i \in [t-1] : \sum_{k \leq i} 1 {p_k \leq \alpha_k} \geq j}, \; \kappa_j^* = \sum_{i \leq \kappa_j} 1 {p_i \leq \tau }, \; S^t = \sum_{i < t} 1 {p_i \leq \tau }, \; C_{j+} = \sum_{i = \kappa_j + 1}^{t-1} 1{p_i \leq \lambda} ]
The default sequence of $\gamma$ for ADDIS is the same as for SAFFRON given here.
FWER Control
Alpha-spending {#Alpha-spending}
The Alpha-spending procedure controls the FWER for a potentially infinite stream of p-values using a Bonferroni-like test. Given an overall significance level $\alpha$, the significance thresholds are chosen as [\alpha_i = \alpha \gamma_i] where $\sum_{i=1}^{\infty} \gamma_i = 1$ and $\gamma_i \geq 0$. The procedure strongly controls the FWER for arbitrarily dependent p-values.
Note that the procedure also controls the generalised familywise error rate (k-FWER) for $k > 1$ if $\alpha$ is replaced by $\min(1,k\alpha)$.
The default sequence of $\gamma$ is the same as that for $\xi$ for LORD given here.
Online Fallback {#onlineFallback}
The online fallback procedure of Tian & Ramdas (2019b) provides a uniformly more powerful method than Alpha-spending, by saving the significance level of a previous rejection. More specifically, online fallback tests hypothesis $H_i$ at level [\alpha_i = \alpha \gamma_i + R_{i-1} \alpha_{i-1}] where $R_i = 1{p_i \leq \alpha_i}$ denotes a rejected hypothesis. The procedure strongly controls the FWER for arbitrarily dependent p-values.
The default sequence of $\gamma$ is the same as that for $\xi$ for LORD given here.
ADDIS-spending {#ADDIS-spending}
The ADDIS-spending procedure strongly controls the FWER for independent p-values, and was proposed by Tian & Ramdas (2021). The procedure compensates for the power loss of Alpha-spending, by including both adapativity in the fraction of null hypotheses and the conservativeness of nulls.
ADDIS depends on constants $\lambda$ and $\tau$, where $\lambda < \tau$. Here $\tau \in (0,1)$ represents the threshold for a hypothesis to be selected for testing: p-values greater than $\tau$ are implicitly `discarded' by the procedure, while $\lambda \in (0,1)$ sets the threshold for a p-value to be a candidate for rejection: ADDIS-spending will never reject a p-value larger than $\lambda$.
Note that the procedure controls the generalised familywise error rate (k-FWER) for $k > 1$ if $\alpha$ is replaced by $\min(1,k\alpha)$. Tian and Ramdas (2019b) also presented a version for handling local dependence, see the Section on Asynchronous testing below.
The default sequence of $\gamma$ for ADDIS-spending is the same as for SAFFRON given here.
Accounting for dependent p-values
As noted above, the LORD, SAFFRON, ADDIS and ADDIS-spending procedures assume independent p-values, while the LOND procedure is also valid under positive dependencies (like the Benjamini-Hochberg method, see below). Adjusted versions of LOND and LORD available for arbitrarily dependent p-values. Alpha-spending and online fallback also control the FWER and FDR for arbitrarily dependent p-values.
By way of comparison, the usual Benjamini-Hochberg method for controlling the FDR assumes that the p-values are positively dependent (PRDS). As an example, the PRDS is satisfied for multivariate normal test statistics with a positive correlation matrix). See Benjamini & Yekutieli (2001) for further technical details.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.