In setgree/ResultsStandardizeR: Some functions that Seth Green & John-Henry Pezzuto wrote to assist with a forthcoming meta-analysis

Standardizing Effect Sizes across Studies: A Walkthrough.

Meta-analysis is a tool for calculating the average effect of an intervention across multiple studies. R provides extensive resources for conducting meta-analyses (for an overview, see this CRAN page, this published article on the package metaplus, or this documentation on the metafor package.) The ResultsStandardizeR package provides tools for the first two steps in a meta-analysis, which are 1) estimating the effect size of each study and 2) estimating the standard error of each effect size.

The 3 steps for estimating effect size and standard error are:

1. Identify the effect type and raw (unstandardized) effect size
2. calculate the pooled standard deviation.
3. Calculate the effect size and standard error.

The remainder of this vignette will walk through these steps for Clunies-Ross and O'Meara (1989).

Difference-in-Differences example: Clunies-Ross and O'Meara (1989)

Clunies-Ross and O'Meara (1989) study how contact with intellectually disabled students affects the atttitudes of Australian fourth graders towards intellectually disabled persons. The intervention took place at two schools; each treatment group and each control group had 15 students, for 60 in total.

The following table (taken from p. 279) presents the results of the intervention:

We will use the "total" scale to calculate effects of the ntervention from pre-test to second post-test.

The unstandardized difference-in-differences is:

$DID = (111.1 - 92.7 ) - (102.9 - 99.4) = 14.9$

store this result in R:

cr_es <- (111.1 - 92.7 ) - (102.9 - 99.4) 

The pooled standard deviation is calculated via the following equation (taken from Morris (2008):)

$$ SD_{pre}=\sqrt{ \frac{(n_t - 1)SD_{pre,T}^2 + (n_c - 1)SD_{pre,C}^2} {n_t + n_c - 2} } $$

This comes out to:

sd_pool <- function(sd_t, sd_c, n_t, n_c){
   sqrt(
  (
    (n_t - 1) * (sd_t^2) + (n_c - 1) * (sd_c^2)
    ) /
    (n_t + n_c - 2)
  )
}
  cr_sd <- sd_pool(sd_t = 34.67, sd_c = 28.28, n_t = 15, n_c = 15) # 31.63675

(Alternatively, Glass's $\Delta$ can be calculated by dividing the unstandardized effect size by the standard deviation of the control group alone. Paluck, Green, and Green (forthcoming) uses this method.)

To calculate the standardized effect size and the standard error, we employ the following equations:

$$d = \frac{(M_{post,T}-M_{Pre,T})-(M_{post,C}-M_{pre,C})}{SD_{pre}}$$ To calculate standard error, we impute the results of that calculation into the following equations:

$$V_d = h_g \frac{n_t + n_c}{n_t n_c} + \frac{d ^ 2}{2 (n_t + n_c)}$$ in which $h_g$ is a correction factor for biases arising in small samples: $$h_g = 1 - \frac{3}{4(n_t + n_c - 2 ) - 1} $$

Standard error is the square root of variance: $$se_d = \sqrt{V_d}$$ In code, the ResultsStandardizeR::ResultsStandardizeR function provides all of these results.

ResultsStandardizeR <- function(eff_type, u_s_d, pop_sd, n_t, n_c){
  # difference in differences
  if (eff_type == "d_i_d"){
    d <- round(u_s_d / pop_sd, digits = 3)
  }
  # difference in means
  else if (eff_type == "d_i_m"){
    d <- round(u_s_d / pop_sd, digits = 3)
  }
  # standardized regression coefficient
  else if (eff_type == "st_reg_coef"){
    d <- u_s_d
  }

  # regression coefficient
  else if (eff_type == "reg_coef"){
    d <- round(u_s_d / pop_sd, digits = 3)
  }

  # t test
  else if (eff_type == "t_test"){
    d <- round(u_s_d * sqrt( (n_t + n_c ) / (n_t * n_c) ), digits = 3)
  }
  # f_test
  else if (eff_type == "f_test"){
    d <- round(sqrt( ( u_s_d * (n_t + n_c) ) / (n_t * n_c) ), digits = 3)
  }
  # Next, compute variance of the estimated effect size

  ust_var_d <- (((n_t + n_c)
                 / (n_t * n_c))
                +
                  ((d ^ 2) / (2 * (n_t + n_c)) )
  )
  # Apply hedge's g correction
  hedge_g <- 1 - (3
                  /
                    (4 * (n_t + n_c - 2 ) - 1))

  var_d <- round((hedge_g ^ 2) * ust_var_d, digits = 3)

  # standard error is the square root of variance
  st_err_g <- round(sqrt(var_d), digits = 3)

  # print everything out
  results <- c(d, var_d, st_err_g)

  col_names <- c("Standardized Effect (Cohen's D)",
                 "Variance of D", "Standard Error of D")

  results_table <- data.frame(col_names, results)

  return(results_table)
}

Filling in the equation with information taken from Clunies-Ross and O'Meara, we get:

cr_d <- ResultsStandardizeR(eff_type = "d_i_d", u_s_d = cr_es, pop_sd = cr_sd, n_t = 15, n_c = 15)

Which yields the following results:

1 Standardized Effect (Cohen's D) = 0.471

2 Variance of D = 0.130

3 Standard Error of D = 0.361

The math is slightly different for t-tests, F-tests, regression coefficients, standardized regression coefficients, or difference in means, but the general idea is the same, and ResultsStandardizeR works for all of them.

Use these equations for a study, rinse, and repeat until you have effect sizes for each.

setgree/ResultsStandardizeR documentation built on June 2, 2020, 11:48 a.m.