Description Usage Arguments Details Value Author(s) References See Also Examples
Function for conducting a Bayesian A/B test (i.e., test between two proportions).
1 2 3 4 5 6 7 8 9 10 
data 
list or data frame with the data. This list (data frame) needs to
contain the following elements: 
prior_par 
list with prior parameters. This list needs to contain the
following elements: 
prior_prob 
named vector with prior probabilities for the four
hypotheses 
nsamples 
determines the number of importance samples for obtaining the
log marginal likelihood for 
is_df 
degrees of freedom of the multivariate t importance sampling
proposal density. The default is 
posterior 
Boolean which indicates whether posterior samples should be
returned. The default is 
y 
integer vector of length 2 containing the number of "successes" in the control and experimental conditon 
n 
integer vector of length 2 containing the number of trials in the control and experimental conditon 
The implemented Bayesian A/B test is based on the following model by Kass and Vaidyanathan (1992, section 3):
log(p1/(1  p1)) = β  ψ/2
log(p2/(1  p2)) = β + ψ/2
y1 ~ Binomial(n1, p1)
y2 ~ Binomial(n2, p2).
"H0"
states that ψ = 0,
"H1"
states that ψ != 0, "H+"
states that ψ
> 0, and "H"
states that ψ < 0. Normal priors are
assigned to the two parameters ψ (i.e., the testrelevant log odds
ratio) and β (i.e., the grand mean of the log odds which is a
nuisance parameter). Log marginal likelihoods for "H0"
and
"H1"
are obtained via Laplace approximations (see Kass &
Vaidyanathan, 1992) which work well even for very small sample sizes. For
the onesided hypotheses "H+"
and "H"
the log marginal
likelihoods are obtained based on importance sampling which uses as a
proposal a multivariate t distribution with location and scale matrix
obtained via a Laplace approximation to the (logtransformed) posterior. If
posterior = TRUE
, posterior samples are obtained using importance
sampling.
returns an object of class "ab"
with components:
input
: a list with the input arguments.
post
: a
list with parameter posterior samples for the three hypotheses "H1"
,
"H+"
(in the output called "Hplus"
), and "H"
(in the
output called "Hminus"
). Only contains samples if posterior =
TRUE
.
laplace
: a list with the approximate parameter
posterior mode and variance/covariance matrix for each hypothesis obtained
via a Laplace approximation.
method
: character that indicates
the method that has been used to obtain the results. The default is
"logis"
(importance sampling with multivariate t proposal based on
a Laplace approximation to the log transformed posterior). If this method
fails (for the onesided hypotheses), method "issn"
is used (i.e.,
importance sampling is used to obtain unconstrained samples, then a
skewnormal distribution is fitted to the samples to obtain the results for
the onesided hypotheses). If method = "issn"
, posterior samples
can only be obtained for "H1"
.
logml
: a list with the
estimated log marginal likelihoods for the hypotheses "H0"
(i.e.,
"logml0"
), "H1"
(i.e., "logml1"
), "H+"
(i.e.,
"logmlplus"
), and "H"
(i.e., "logmlminus"
).
post_prob
: a named vector with the posterior probabilities of the
four hypotheses "H1"
, "H+"
, "H"
, and "H0"
.
logbf
: a list with the log Bayes factor in favor of
"H1"
over "H0"
, the log Bayes factor in favor of "H+"
over "H0"
, and the log Bayes factor in favor of "H"
over
"H0"
.
bf
: a list with the Bayes factor in favor of
"H1"
over "H0"
(i.e., "bf10"
), the Bayes factor in
favor of "H+"
over "H0"
(i.e., "bfplus0"
), and the
Bayes factor in favor of "H"
over "H0"
(i.e.,
"bfminus0"
).
Quentin F. Gronau
Kass, R. E., & Vaidyanathan, S. K. (1992). Approximate Bayes factors and orthogonal parameters, with application to testing equality of two binomial proportions. Journal of the Royal Statistical Society, Series B, 54, 129144. doi: 10.1111/j.25176161.1992.tb01868.x
Gronau, Q. F., Raj K. N., A., & Wagenmakers, E.J. (2021). Informed Bayesian Inference for the A/B Test. Journal of Statistical Software, 100. doi: 10.18637/jss.v100.i17
elicit_prior
allows the user to elicit a prior based
on providing quantiles for either the log odds ratio, the odds ratio, the
relative risk, or the absolute risk. The resulting prior is always
translated to the corresponding normal prior on the log odds ratio. The
plot_prior
function allows the user to visualize the prior
distribution. The simulate_priors
function produces samples
from the prior distribution. The prior and posterior probabilities of the
hypotheses can be visualized using the prob_wheel
function.
Parameter posteriors can be visualized using the
plot_posterior
function. The plot_sequential
function allows the user to sequentially plot the posterior probabilities
of the hypotheses (only possible if the data
object contains vectors
with the cumulative "successes"/trials).
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24  # synthetic data
data < list(y1 = 10, n1 = 28, y2 = 14, n2 = 26)
# Bayesian A/B test with default settings
ab < ab_test(data = data)
print(ab)
# different prior parameter settings
prior_par < list(mu_psi = 0.2, sigma_psi = 0.8,
mu_beta = 0, sigma_beta = 0.7)
ab2 < ab_test(data = data, prior_par = prior_par)
print(ab2)
# different prior probabilities
prior_prob < c(.1, .3, .2, .4)
names(prior_prob) < c("H1", "H+", "H", "H0")
ab3 < ab_test(data = data, prior_prob = prior_prob)
print(ab3)
# also possible to obtain posterior samples
ab4 < ab_test(data = data, posterior = TRUE)
# plot parameter posterior
plot_posterior(x = ab4, what = "logor")

Bayesian A/B Test Results:
Bayes Factors:
BF10: 0.976286
BF+0: 1.744964
BF0: 0.2299104
Prior Probabilities Hypotheses:
H+: 0.25
H: 0.25
H0: 0.5
Posterior Probabilities Hypotheses:
H+: 0.439
H: 0.0578
H0: 0.5032
Bayesian A/B Test Results:
Bayes Factors:
BF10: 1.212806
BF+0: 1.825308
BF0: 0.3165151
Prior Probabilities Hypotheses:
H+: 0.25
H: 0.25
H0: 0.5
Posterior Probabilities Hypotheses:
H+: 0.4407
H: 0.0764
H0: 0.4829
Bayesian A/B Test Results:
Bayes Factors:
BF10: 0.976286
BF+0: 1.72398
BF0: 0.2372397
Prior Probabilities Hypotheses:
H1: 0.1
H+: 0.3
H: 0.2
H0: 0.4
Posterior Probabilities Hypotheses:
H1: 0.0919
H+: 0.4869
H: 0.0447
H0: 0.3766
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.