ss.power.it returns the necessary per-group sample size
to achieve a desired level of statistical power for a planned study using
an independent t-test, based on information obtained from a previous study.
The effect from the previous study can be corrected for publication bias
and/or uncertainty to provide a sample size that will achieve more accurate
statistical power for a planned study, when compared to approaches that use
a sample effect size at face value or rely on sample size only. The bias
and uncertainty adjusted previous study noncentrality parameter is also
returned, which can be transformed to various effect size metrics.
Observed t-value from a previous study used to plan sample size for a planned study
Per group sample size (if equal) or the two group sample sizes of the previous study (enter either a single number or a vector of length 2)
Total sample size of the previous study, assumed equal across groups if specified
Alpha-level α for the previous study or the assumed statistical significance necessary for publishing in the field; to assume no publication bias, a value of 1 can be entered
Alpha-level (α) assumed for the planned study
Desired level of assurance, or the long run proportion of times that the planned study power will reach or surpass desired level (assurance > .5 corrects for uncertainty; assurance < .5 not recommended)
Desired level of statistical power for the planned study
Value used in the iterative scheme to determine the noncentrality parameter necessary for sample size planning (0 < step < .01) (users should not generally need to change this value; smaller values lead to more accurate sample size planning results, but unnecessarily small values will add unnecessary computational time)
Researchers often use the sample effect size from a prior study as an estimate of the likely size of an expected future effect in sample size planning. However, sample effect size estimates should not usually be used at face value to plan sample size, due to both publication bias and uncertainty.
The approach implemented in
ss.power.it uses the observed
t-value and sample size from a previous study to correct the
noncentrality parameter associated with the effect of interest for
publication bias and/or uncertainty. This new estimated noncentrality
parameter is then used to calculate the necessary per-group sample size to
achieve the desired level of power in the planned study.
The approach uses a likelihood function of a truncated non-central F distribution, where the truncation occurs due to small effect sizes being unobserved due to publication bias. The numerator of the likelihood function is simply the density of a noncentral F distribution. The denominator is the power of the test, which serves to truncate the distribution. In the two-group case, this formula reduces to the density of a truncated noncentral t-distribution.(See Taylor & Muller, 1996, Equation 2.1. and Anderson & Maxwell, 2017, for more details.)
Assurance is the proportion of times that power will be at or above the desired level, if the experiment were to be reproduced many times. For example, assurance = .5 means that power will be above the desired level half of the time, but below the desired level the other half of the time. Selecting assurance = .5 (selecting the noncentrality parameter at the 50th percentile of the likelihood distribution) results in a median-unbiased estimate of the population noncentrality parameter and does not corrects for uncertainty. In order to correct for uncertainty, assurance > .5 can be selected, which corresponds to selecting the noncentrality parameter associated with the (1 - assurance) quantile of the likelihood distribution.
If the previous study of interest has not been subjected to publication
bias (e.g., a pilot study),
alpha.prior can be set to 1 to indicate
no publication bias. Alternative α-levels can also be
accommodated to represent differing amounts of publication bias. For
alpha.prior=.20 would reflect less severe
publication bias than the default of .05. In essence, setting
alpha.prior at .20 assumes that studies with p-values less
than .20 are published, whereas those with larger p-values are not.
In some cases, the corrected noncentrality parameter for a given level of
assurance will be estimated to be zero. This is an indication that, at the
desired level of assurance, the previous study's effect cannot be
accurately estimated due to high levels of uncertainty and bias. When this
happens, subsequent sample size planning is not possible with the chosen
specifications. Two alternatives are recommended. First, users can select a
lower value of assurance (e.g. .8 instead of .95). Second, users can reduce
the influence of publciation bias by setting
alpha.prior at a value
greater than .05. It is possible to correct for uncertainty only by setting
alpha.prior=1 and choosing the desired level of assurance. We
encourage users to make the adjustments as minimal as possible.
ss.power.it assumes that the planned study will have equal n.
Unequal n in the previous study is handled in the following way for the
independent-t. If the user enters an odd value for N, no information is
available on the exact group sizes. The function calculates n by dividing N
by 2 and both rounding up and rounding down the result, thus assuming equal
n. The suggested sample size for the planned study is calculated using both
of these values of n, and the function returns the larger of these two
suggestions, to be conservative. If the user enters a vector for n with two
different values, specific information is available on the exact group
sizes. n is calcualted as the harmonic mean of these two values (a measure
of effective sample size). Again, this is rounded both up and down. The
suggested sample size for the planned study is calculated using both of
these values of n, and the function returns the larger of these two
suggestions, to be conservative. The adjusted noncentrality parameter
that is output is the lower of the two possibilities, again, to be
conservative. When the individual group sizes of an unequal-n previous study
are known, we highly encourage entering these explicitly, especially if the
sample sizes are quite discrepant, as this allows for the greatest precision
in estimating an appropriate planned study n.
Suggested per-group sample size for planned study Publication bias and uncertainty- adjusted prior study noncentrality parameter
Anderson, S. F., & Maxwell, S. E. (2017). Addressing the 'replication crisis': Using original studies to design replication studies with appropriate statistical power. Multivariate Behavioral Research, 52, 305-322.
Anderson, S. F., Kelley, K., & Maxwell, S. E. (2017). Sample size planning for more accurate statistical power: A method correcting sample effect sizes for uncertainty and publication bias. Psychological Science, 28, 1547-1562.
Taylor, D. J., & Muller, K. E. (1996). Bias in linear model power and sample size calculation due to estimating noncentrality. Communications in Statistics: Theory and Methods, 25, 1595-1610.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.