Effect Size from Test Statistics"

In effectsize: Indices of Effect Size

library(knitr)
library(effectsize)

knitr::opts_chunk$set(comment = ">")
options(digits = 2)
options(knitr.kable.NA = "")

set.seed(747)

.eval_if_requireNamespace <- function(...) {
  pkgs <- c(...)
  knitr::opts_chunk$get("eval") && all(sapply(pkgs, requireNamespace, quietly = TRUE))
}

Introduction

In many real world applications there are no straightforward ways of obtaining standardized effect sizes. However, it is possible to get approximations of most of the effect size indices ($d$, $r$, $\eta^2_p$...) with the use of test statistics. These conversions are based on the idea that test statistics are a function of effect size and sample size. Thus information about samples size (or more often of degrees of freedom) is used to reverse-engineer indices of effect size from test statistics. This idea and these functions also power our Effect Sizes From Test Statistics shiny app.

The measures discussed here are, in one way or another, signal to noise ratios, with the "noise" representing the unaccounted variance in the outcome variable^[Note that for generalized linear models (Poisson, Logistic...), where the outcome is never on an arbitrary scale, estimates themselves are indices of effect size! Thus this vignette is relevant only to general linear models.].

The indices are:

• Percent variance explained ($\eta^2_p$, $\omega^2_p$, $\epsilon^2_p$).

• Measure of association ($r$).

• Measure of difference ($d$).

(Partial) Percent Variance Explained

These measures represent the ratio of $Signal^2 / (Signal^2 + Noise^2)$, with the "noise" having all other "signals" partial-ed out (be they of other fixed or random effects). The most popular of these indices is $\eta^2_p$ (Eta; which is equivalent to $R^2$).

The conversion of the $F$- or $t$-statistic is based on @friedman1982simplified.

Let's look at an example:

library(afex)

data(md_12.1)

aov_fit <- aov_car(rt ~ angle * noise + Error(id / (angle * noise)),
  data = md_12.1,
  anova_table = list(correction = "none", es = "pes")
)
aov_fit

Let's compare the $\eta^2_p$ (the pes column) obtained here with ones recovered from F_to_eta2():

library(effectsize)
options(es.use_symbols = TRUE) # get nice symbols when printing! (On Windows, requires R >= 4.2.0)

F_to_eta2(
  f = c(40.72, 33.77, 45.31),
  df = c(2, 1, 2),
  df_error = c(18, 9, 18)
)

They are identical!^[Note that these are partial percent variance explained, and so their sum can be larger than 1.] (except for the fact that F_to_eta2() also provides confidence intervals^[Confidence intervals for all indices are estimated using the non-centrality parameter method; These methods search for a the best non-central parameter of the non-central $F$/$t$ distribution for the desired tail-probabilities, and then convert these ncps to the corresponding effect sizes.] :)

In this case we were able to easily obtain the effect size (thanks to afex!), but in other cases it might not be as easy, and using estimates based on test statistic offers a good approximation.

For example:

In Simple Effect and Contrast Analysis

library(emmeans)

joint_tests(aov_fit, by = "noise")

F_to_eta2(
  f = c(8, 51),
  df = 2,
  df_error = 9
)

We can also use t_to_eta2() for contrast analysis:

pairs(emmeans(aov_fit, ~ angle))

t_to_eta2(
  t = c(-6.2, -8.2, -3.2),
  df_error = 9
)

In Linear Mixed Models

library(lmerTest)

fit_lmm <- lmer(Reaction ~ Days + (Days | Subject), sleepstudy)

anova(fit_lmm)

F_to_eta2(45.9, 1, 17)

We can also use t_to_eta2() for the slope of Days (which in this case gives the same result).

parameters::model_parameters(fit_lmm, effects = "fixed", ci_method = "satterthwaite")

t_to_eta2(6.77, df_error = 17)

Bias-Corrected Indices

Alongside $\eta^2_p$ there are also the less biased $\omega_p^2$ (Omega) and $\epsilon^2_p$ (Epsilon; sometimes called $\text{Adj. }\eta^2_p$, which is equivalent to $R^2_{adj}$; @albers2018power, @mordkoff2019simple).

F_to_eta2(45.9, 1, 17)
F_to_epsilon2(45.9, 1, 17)
F_to_omega2(45.9, 1, 17)

Measure of Association

Similar to $\eta^2_p$, $r$ is a signal to noise ratio, and is in fact equal to $\sqrt{\eta^2_p}$ (so it's really a partial $r$). It is often used instead of $\eta^2_p$ when discussing the strength of association (but I suspect people use it instead of $\eta^2_p$ because it gives a bigger number, which looks better).

For Slopes

parameters::model_parameters(fit_lmm, effects = "fixed", ci_method = "satterthwaite")

t_to_r(6.77, df_error = 17)

In a fixed-effect linear model, this returns the partial correlation. Compare:

fit_lm <- lm(rating ~ complaints + critical, data = attitude)

parameters::model_parameters(fit_lm)

t_to_r(
  t = c(7.46, 0.01),
  df_error = 27
)

to:

correlation::correlation(attitude,
  select = "rating",
  select2 = c("complaints", "critical"),
  partial = TRUE
)

In Contrast Analysis

This measure is also sometimes used in contrast analysis, where it is called the point bi-serial correlation - $r_{pb}$ [@cohen1965some; @rosnow2000contrasts]:

pairs(emmeans(aov_fit, ~ angle))

t_to_r(
  t = c(-6.2, -8.2, -3.2),
  df_error = 9
)

Measures of Difference

These indices represent $Signal/Noise$ with the "signal" representing the difference between two means. This is akin to Cohen's $d$, and is a close approximation when comparing two groups of equal size [@wolf1986meta; @rosnow2000contrasts].

These can be useful in contrast analyses.

Between-Subject Contrasts

m <- lm(breaks ~ tension, data = warpbreaks)

em_tension <- emmeans(m, ~tension)
pairs(em_tension)

t_to_d(
  t = c(2.53, 3.72, 1.20),
  df_error = 51
)

However, these are merely approximations of a true Cohen's d. It is advised to directly estimate Cohen's d, whenever possible. For example, here with emmeans::eff_size():

eff_size(em_tension, sigma = sigma(m), edf = df.residual(m))

Within-Subject Contrasts

pairs(emmeans(aov_fit, ~ angle))

t_to_d(
  t = c(-6.2, -8.2, -3.3),
  df_error = 9,
  paired = TRUE
)

(Note set paired = TRUE to not over estimate the size of the effect; @rosenthal1991meta; @rosnow2000contrasts)

References

effectsize documentation built on Sept. 14, 2023, 5:07 p.m.