empirical_ES: Empirical effect sizes based on latent trait estimates

Description Usage Arguments DIF DBF/DTF Author(s) References Examples

View source: R/empirical_ES.R

Description

Computes effect size measures of differential item functioning and differential test/bundle functioning based on expected scores from Meade (2010). Item parameters from both reference and focal group are used in conjunction with focal group empirical theta estimates (and an assumed normally distributed theta) to compute expected scores.

Usage

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
empirical_ES(
  mod,
  Theta.focal = NULL,
  focal_items = 1L:extract.mirt(mod, "nitems"),
  DIF = TRUE,
  npts = 61,
  theta_lim = c(-6, 6),
  plot = FALSE,
  type = "b",
  par.strip.text = list(cex = 0.7),
  par.settings = list(strip.background = list(col = "#9ECAE1"), strip.border = list(col
    = "black")),
  ...
)

Arguments

mod

a multipleGroup object which estimated only 2 groups. The first group in this object is assumed to be the reference group by default (i.e., ref.group = 1), which conforms to the invariance arguments in multipleGroup

Theta.focal

an optional matrix of Theta values from the focal group to be evaluated. If not supplied the default values to fscores will be used in conjunction with the ... arguments passed

focal_items

a numeric vector indicating which items to include the tests. The default uses all of the items. Selecting fewer items will result in tests of 'differential bundle functioning' when DIF = FALSE

DIF

logical; return a data.frame of item-level imputation properties? If FALSE, only DBF and DTF statistics will be reported

npts

number of points to use in the integration. Default is 61

theta_lim

lower and upper limits of the latent trait (theta) to be evaluated, and is used in conjunction with npts

plot

logical; plot expected scores of items/test where expected scores are computed using focal group thetas and both focal and reference group item parameters

type

type of objects to draw in lattice; default plots both points and lines

par.strip.text

plotting argument passed to lattice

par.settings

plotting argument passed to lattice

...

additional arguments to be passed to fscores and xyplot

DIF

The default DIF = TRUE produces several effect sizes indices at the item level. Signed indices allow DIF favoring the focal group at one point on the theta distribution to cancel DIF favoring the reference group at another point on the theta distribution. Unsigned indices take the absolute value before summing or averaging, thus not allowing cancellation of DIF across theta.

SIDS

Signed Item Difference in the Sample. The average difference in expected scores across the focal sample using both focal and reference group item parameters.

UIDS

Unsigned Item Difference in the Sample. Same as SIDS except absolute value of expected scores is taken prior to averaging across the sample.

D-Max

The maximum difference in expected scores in the sample.

ESSD

Expected Score Standardized Difference. Cohen's D for difference in expected scores.

SIDN

Signed Item Difference in a Normal distribution. Identical to SIDS but averaged across a normal distribution rather than the sample.

UIDN

Unsigned Item Difference in a Normal distribution. Identical to UIDS but averaged across a normal distribution rather than the sample.

DBF/DTF

DIF = FALSE produces a series of test/bundle-level indices that are based on item-level indices.

STDS

Signed Test Differences in the Sample. The sum of the SIDS across items.

UTDS

Unsigned Test Differences in the Sample. The sum of the UIDS across items.

Stark's DTFR

Stark's version of STDS using a normal distribution rather than sample estimated thetas.

UDTFR

Unsigned Expected Test Scores Differences in the Sample. The difference in observed summed scale scores expected, on average, across a hypothetical focal group with a normally distributed theta, had DF been uniform in nature for all items

UETSDS

Unsigned Expected Test Score Differences in the Sample. The hypothetical difference expected scale scores that would have been present if scale-level DF had been uniform across respondents (i.e., always favoring the focal group).

UETSDN

Identical to UETSDS but computed using a normal distribution.

Test D-Max

Maximum expected test score differences in the sample.

ETSSD

Expected Test Score Standardized Difference. Cohen's D for expected test scores.

Author(s)

Adam Meade and Phil Chalmers rphilip.chalmers@gmail.com

References

Chalmers, R., P. (2012). mirt: A Multidimensional Item Response Theory Package for the R Environment. Journal of Statistical Software, 48(6), 1-29. doi: 10.18637/jss.v048.i06

Meade, A. W. (2010). A taxonomy of effect size measures for the differential functioning of items and scales. Journal of Applied Psychology, 95, 728-743.

Examples

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
## Not run: 

#no DIF
set.seed(12345)
a <- matrix(abs(rnorm(15,1,.3)), ncol=1)
d <- matrix(rnorm(15,0,.7),ncol=1)
itemtype <- rep('2PL', nrow(a))
N <- 1000
dataset1 <- simdata(a, d, N, itemtype)
dataset2 <- simdata(a, d, N, itemtype, mu = .1, sigma = matrix(1.5))
dat <- rbind(dataset1, dataset2)

# ensure 'Ref' is the first group (and therefore reference group during estimation)
group <- factor(c(rep('Ref', N), rep('Focal', N)), levels = c('Ref', 'Focal'))

mod <- multipleGroup(dat, 1, group = group,
   invariance = c(colnames(dat)[1:5], 'free_means', 'free_var'))
coef(mod, simplify=TRUE)

empirical_ES(mod)
empirical_ES(mod, DIF=FALSE)
empirical_ES(mod, DIF=FALSE, focal_items = 10:15)

empirical_ES(mod, plot=TRUE)
empirical_ES(mod, plot=TRUE, DIF=FALSE)

###---------------------------------------------
# DIF
set.seed(12345)
a1 <- a2 <- matrix(abs(rnorm(15,1,.3)), ncol=1)
d1 <- d2 <- matrix(rnorm(15,0,.7),ncol=1)
a2[10:15,] <- a2[10:15,] + rnorm(6, 0, .3)
d2[10:15,] <- d2[10:15,] + rnorm(6, 0, .3)
itemtype <- rep('dich', nrow(a1))
N <- 1000
dataset1 <- simdata(a1, d1, N, itemtype)
dataset2 <- simdata(a2, d2, N, itemtype, mu = .1, sigma = matrix(1.5))
dat <- rbind(dataset1, dataset2)
group <- factor(c(rep('Ref', N), rep('Focal', N)), levels = c('Ref', 'Focal'))

mod <- multipleGroup(dat, 1, group = group,
   invariance = c(colnames(dat)[1:5], 'free_means', 'free_var'))
coef(mod, simplify=TRUE)

empirical_ES(mod)
empirical_ES(mod, DIF = FALSE)
empirical_ES(mod, plot=TRUE)
empirical_ES(mod, plot=TRUE, DIF=FALSE)


## End(Not run)

Example output

Loading required package: stats4
Loading required package: lattice

Iteration: 1, Log-Lik: -18088.494, Max-Change: 0.37971
Iteration: 2, Log-Lik: -17927.412, Max-Change: 0.16094
Iteration: 3, Log-Lik: -17911.502, Max-Change: 0.08104
Iteration: 4, Log-Lik: -17909.155, Max-Change: 0.04069
Iteration: 5, Log-Lik: -17908.578, Max-Change: 0.02045
Iteration: 6, Log-Lik: -17908.347, Max-Change: 0.01025
Iteration: 7, Log-Lik: -17908.154, Max-Change: 0.01089
Iteration: 8, Log-Lik: -17908.062, Max-Change: 0.00772
Iteration: 9, Log-Lik: -17907.989, Max-Change: 0.00689
Iteration: 10, Log-Lik: -17907.872, Max-Change: 0.02108
Iteration: 11, Log-Lik: -17907.717, Max-Change: 0.00753
Iteration: 12, Log-Lik: -17907.692, Max-Change: 0.00530
Iteration: 13, Log-Lik: -17907.654, Max-Change: 0.00948
Iteration: 14, Log-Lik: -17907.631, Max-Change: 0.00490
Iteration: 15, Log-Lik: -17907.619, Max-Change: 0.00406
Iteration: 16, Log-Lik: -17907.604, Max-Change: 0.01225
Iteration: 17, Log-Lik: -17907.575, Max-Change: 0.00435
Iteration: 18, Log-Lik: -17907.570, Max-Change: 0.00309
Iteration: 19, Log-Lik: -17907.563, Max-Change: 0.00506
Iteration: 20, Log-Lik: -17907.558, Max-Change: 0.00278
Iteration: 21, Log-Lik: -17907.555, Max-Change: 0.00236
Iteration: 22, Log-Lik: -17907.553, Max-Change: 0.00721
Iteration: 23, Log-Lik: -17907.545, Max-Change: 0.00252
Iteration: 24, Log-Lik: -17907.543, Max-Change: 0.00178
Iteration: 25, Log-Lik: -17907.542, Max-Change: 0.00282
Iteration: 26, Log-Lik: -17907.540, Max-Change: 0.00159
Iteration: 27, Log-Lik: -17907.539, Max-Change: 0.00136
Iteration: 28, Log-Lik: -17907.539, Max-Change: 0.00418
Iteration: 29, Log-Lik: -17907.536, Max-Change: 0.00145
Iteration: 30, Log-Lik: -17907.536, Max-Change: 0.00103
Iteration: 31, Log-Lik: -17907.535, Max-Change: 0.00161
Iteration: 32, Log-Lik: -17907.535, Max-Change: 0.00091
Iteration: 33, Log-Lik: -17907.534, Max-Change: 0.00078
Iteration: 34, Log-Lik: -17907.534, Max-Change: 0.00242
Iteration: 35, Log-Lik: -17907.533, Max-Change: 0.00082
Iteration: 36, Log-Lik: -17907.533, Max-Change: 0.00060
Iteration: 37, Log-Lik: -17907.533, Max-Change: 0.00096
Iteration: 38, Log-Lik: -17907.533, Max-Change: 0.00052
Iteration: 39, Log-Lik: -17907.533, Max-Change: 0.00045
Iteration: 40, Log-Lik: -17907.533, Max-Change: 0.00136
Iteration: 41, Log-Lik: -17907.532, Max-Change: 0.00047
Iteration: 42, Log-Lik: -17907.532, Max-Change: 0.00033
Iteration: 43, Log-Lik: -17907.532, Max-Change: 0.00052
Iteration: 44, Log-Lik: -17907.532, Max-Change: 0.00030
Iteration: 45, Log-Lik: -17907.532, Max-Change: 0.00025
Iteration: 46, Log-Lik: -17907.532, Max-Change: 0.00077
Iteration: 47, Log-Lik: -17907.532, Max-Change: 0.00027
Iteration: 48, Log-Lik: -17907.532, Max-Change: 0.00019
Iteration: 49, Log-Lik: -17907.532, Max-Change: 0.00031
Iteration: 50, Log-Lik: -17907.532, Max-Change: 0.00017
Iteration: 51, Log-Lik: -17907.532, Max-Change: 0.00015
Iteration: 52, Log-Lik: -17907.532, Max-Change: 0.00044
Iteration: 53, Log-Lik: -17907.532, Max-Change: 0.00015
Iteration: 54, Log-Lik: -17907.532, Max-Change: 0.00011
Iteration: 55, Log-Lik: -17907.532, Max-Change: 0.00018
Iteration: 56, Log-Lik: -17907.532, Max-Change: 0.00009
$Ref
$items
           a1      d g u
Item_1  1.085  0.518 0 1
Item_2  1.182 -0.652 0 1
Item_3  1.040 -0.284 0 1
Item_4  0.869  0.885 0 1
Item_5  1.063  0.144 0 1
Item_6  0.567  0.683 0 1
Item_7  1.273  1.001 0 1
Item_8  0.924 -0.330 0 1
Item_9  0.890 -1.059 0 1
Item_10 0.721 -1.082 0 1
Item_11 0.832  1.188 0 1
Item_12 1.478 -0.252 0 1
Item_13 1.288  0.445 0 1
Item_14 1.034  0.452 0 1
Item_15 0.864 -0.062 0 1

$means
F1 
 0 

$cov
   F1
F1  1


$Focal
$items
           a1      d g u
Item_1  1.085  0.518 0 1
Item_2  1.182 -0.652 0 1
Item_3  1.040 -0.284 0 1
Item_4  0.869  0.885 0 1
Item_5  1.063  0.144 0 1
Item_6  0.363  0.585 0 1
Item_7  1.083  0.915 0 1
Item_8  0.997 -0.471 0 1
Item_9  0.840 -1.041 0 1
Item_10 0.647 -1.181 0 1
Item_11 1.009  1.199 0 1
Item_12 1.278 -0.240 0 1
Item_13 1.201  0.366 0 1
Item_14 1.219  0.403 0 1
Item_15 0.696 -0.103 0 1

$means
   F1 
0.106 

$cov
      F1
F1 1.697




          SIDS  UIDS   SIDN  UIDN   ESSD theta.of.max.D  max.D mean.ES.foc
item.1   0.000 0.000  0.000 0.000  0.000         -1.786  0.000       0.618
item.2   0.000 0.000  0.000 0.000  0.000         -1.786  0.000       0.408
item.3   0.000 0.000  0.000 0.000  0.000         -1.786  0.000       0.468
item.4   0.000 0.000  0.000 0.000  0.000         -1.786  0.000       0.691
item.5   0.000 0.000  0.000 0.000  0.000         -1.786  0.000       0.550
item.6  -0.018 0.042 -0.020 0.037 -0.148         -2.560  0.098       0.645
item.7  -0.004 0.023 -0.007 0.021 -0.018         -1.935  0.047       0.686
item.8  -0.024 0.024 -0.026 0.026 -0.106         -0.533 -0.037       0.431
item.9  -0.001 0.007  0.000 0.006 -0.005          2.571 -0.020       0.313
item.10 -0.024 0.025 -0.023 0.023 -0.163          2.571 -0.065       0.271
item.11 -0.009 0.024 -0.006 0.021 -0.048         -2.357 -0.081       0.736
item.12 -0.003 0.023 -0.003 0.022 -0.012          1.229 -0.036       0.484
item.13 -0.012 0.015 -0.014 0.015 -0.045          0.618 -0.024       0.589
item.14 -0.012 0.027 -0.012 0.025 -0.048         -1.426 -0.056       0.595
item.15 -0.013 0.031 -0.013 0.028 -0.064          1.901 -0.057       0.494
        mean.ES.ref
item.1        0.618
item.2        0.408
item.3        0.468
item.4        0.691
item.5        0.550
item.6        0.663
item.7        0.690
item.8        0.455
item.9        0.314
item.10       0.295
item.11       0.744
item.12       0.487
item.13       0.601
item.14       0.607
item.15       0.507


          Effect Size       Value
1                STDS -0.12008160
2                UTDS  0.24123472
3              UETSDS  0.13069317
4               ETSSD -0.03693662
5         Starks.DTFR -0.12339419
6               UDTFR  0.22581019
7              UETSDN  0.12941914
8 theta.of.max.test.D  1.73548473
9           Test.Dmax -0.23003594


          Effect Size       Value
1                STDS -0.01762183
2                UTDS  0.04212819
3              UETSDS  0.04212819
4               ETSSD -0.01368588
5         Starks.DTFR -0.01980182
6               UDTFR  0.03720931
7              UETSDN  0.03720931
8 theta.of.max.test.D -2.56041489
9           Test.Dmax  0.09832391





Iteration: 1, Log-Lik: -17936.586, Max-Change: 0.73422
Iteration: 2, Log-Lik: -17615.416, Max-Change: 0.25313
Iteration: 3, Log-Lik: -17593.858, Max-Change: 0.08898
Iteration: 4, Log-Lik: -17591.007, Max-Change: 0.04354
Iteration: 5, Log-Lik: -17590.327, Max-Change: 0.02234
Iteration: 6, Log-Lik: -17590.052, Max-Change: 0.01229
Iteration: 7, Log-Lik: -17589.799, Max-Change: 0.00947
Iteration: 8, Log-Lik: -17589.685, Max-Change: 0.00934
Iteration: 9, Log-Lik: -17589.595, Max-Change: 0.00863
Iteration: 10, Log-Lik: -17589.380, Max-Change: 0.02047
Iteration: 11, Log-Lik: -17589.235, Max-Change: 0.00420
Iteration: 12, Log-Lik: -17589.211, Max-Change: 0.00434
Iteration: 13, Log-Lik: -17589.158, Max-Change: 0.01041
Iteration: 14, Log-Lik: -17589.119, Max-Change: 0.00203
Iteration: 15, Log-Lik: -17589.113, Max-Change: 0.00212
Iteration: 16, Log-Lik: -17589.099, Max-Change: 0.00520
Iteration: 17, Log-Lik: -17589.088, Max-Change: 0.00152
Iteration: 18, Log-Lik: -17589.086, Max-Change: 0.00110
Iteration: 19, Log-Lik: -17589.082, Max-Change: 0.00260
Iteration: 20, Log-Lik: -17589.079, Max-Change: 0.00110
Iteration: 21, Log-Lik: -17589.079, Max-Change: 0.00083
Iteration: 22, Log-Lik: -17589.077, Max-Change: 0.00229
Iteration: 23, Log-Lik: -17589.076, Max-Change: 0.00088
Iteration: 24, Log-Lik: -17589.076, Max-Change: 0.00063
Iteration: 25, Log-Lik: -17589.075, Max-Change: 0.00131
Iteration: 26, Log-Lik: -17589.075, Max-Change: 0.00061
Iteration: 27, Log-Lik: -17589.074, Max-Change: 0.00048
Iteration: 28, Log-Lik: -17589.074, Max-Change: 0.00145
Iteration: 29, Log-Lik: -17589.074, Max-Change: 0.00052
Iteration: 30, Log-Lik: -17589.074, Max-Change: 0.00037
Iteration: 31, Log-Lik: -17589.074, Max-Change: 0.00064
Iteration: 32, Log-Lik: -17589.074, Max-Change: 0.00034
Iteration: 33, Log-Lik: -17589.074, Max-Change: 0.00028
Iteration: 34, Log-Lik: -17589.073, Max-Change: 0.00086
Iteration: 35, Log-Lik: -17589.073, Max-Change: 0.00031
Iteration: 36, Log-Lik: -17589.073, Max-Change: 0.00022
Iteration: 37, Log-Lik: -17589.073, Max-Change: 0.00032
Iteration: 38, Log-Lik: -17589.073, Max-Change: 0.00020
Iteration: 39, Log-Lik: -17589.073, Max-Change: 0.00016
Iteration: 40, Log-Lik: -17589.073, Max-Change: 0.00050
Iteration: 41, Log-Lik: -17589.073, Max-Change: 0.00019
Iteration: 42, Log-Lik: -17589.073, Max-Change: 0.00013
Iteration: 43, Log-Lik: -17589.073, Max-Change: 0.00017
Iteration: 44, Log-Lik: -17589.073, Max-Change: 0.00011
Iteration: 45, Log-Lik: -17589.073, Max-Change: 0.00010
$Ref
$items
           a1      d g u
Item_1  1.202  0.566 0 1
Item_2  1.163 -0.626 0 1
Item_3  0.965 -0.316 0 1
Item_4  0.819  0.872 0 1
Item_5  1.165  0.161 0 1
Item_6  0.533  0.631 0 1
Item_7  1.147  1.022 0 1
Item_8  1.064 -0.319 0 1
Item_9  0.889 -1.003 0 1
Item_10 0.756 -1.098 0 1
Item_11 1.024  1.394 0 1
Item_12 1.485 -0.268 0 1
Item_13 1.280  0.402 0 1
Item_14 1.009  0.490 0 1
Item_15 0.745 -0.136 0 1

$means
F1 
 0 

$cov
   F1
F1  1


$Focal
$items
           a1      d g u
Item_1  1.202  0.566 0 1
Item_2  1.163 -0.626 0 1
Item_3  0.965 -0.316 0 1
Item_4  0.819  0.872 0 1
Item_5  1.165  0.161 0 1
Item_6  0.326  0.556 0 1
Item_7  1.085  0.947 0 1
Item_8  0.987 -0.529 0 1
Item_9  0.786 -1.009 0 1
Item_10 0.908 -1.274 0 1
Item_11 1.437  0.628 0 1
Item_12 1.947  0.153 0 1
Item_13 1.477  0.315 0 1
Item_14 1.313  0.623 0 1
Item_15 0.919 -0.878 0 1

$means
   F1 
0.123 

$cov
      F1
F1 1.788




          SIDS  UIDS   SIDN  UIDN   ESSD theta.of.max.D  max.D mean.ES.foc
item.1   0.000 0.000  0.000 0.000  0.000          1.153  0.000       0.624
item.2   0.000 0.000  0.000 0.000  0.000          1.153  0.000       0.417
item.3   0.000 0.000  0.000 0.000  0.000          1.153  0.000       0.464
item.4   0.000 0.000  0.000 0.000  0.000          1.153  0.000       0.691
item.5   0.000 0.000  0.000 0.000  0.000          1.153  0.000       0.556
item.6  -0.014 0.045 -0.017 0.037 -0.120         -2.463  0.103       0.640
item.7  -0.009 0.010 -0.010 0.011 -0.037          0.312 -0.016       0.691
item.8  -0.043 0.043 -0.046 0.046 -0.174          0.940 -0.066       0.424
item.9  -0.010 0.018 -0.008 0.015 -0.054          2.421 -0.050       0.319
item.10 -0.016 0.025 -0.020 0.025 -0.089          2.622  0.043       0.284
item.11 -0.136 0.136 -0.138 0.138 -0.545         -1.274 -0.291       0.627
item.12  0.064 0.070  0.075 0.079  0.195          0.536  0.139       0.555
item.13 -0.016 0.026 -0.017 0.025 -0.055         -1.037 -0.055       0.579
item.14  0.014 0.042  0.019 0.039  0.055         -1.774 -0.061       0.630
item.15 -0.137 0.137 -0.147 0.147 -0.663         -0.086 -0.173       0.355
        mean.ES.ref
item.1        0.624
item.2        0.417
item.3        0.464
item.4        0.691
item.5        0.556
item.6        0.654
item.7        0.699
item.8        0.467
item.9        0.329
item.10       0.300
item.11       0.763
item.12       0.490
item.13       0.595
item.14       0.616
item.15       0.492


          Effect Size       Value
1                STDS -0.30204034
2                UTDS  0.55171507
3              UETSDS  0.30204034
4               ETSSD -0.08665228
5         Starks.DTFR -0.30918670
6               UDTFR  0.56373602
7              UETSDN  0.30918918
8 theta.of.max.test.D -1.05111208
9           Test.Dmax -0.54821798

mirt documentation built on June 29, 2021, 1:06 a.m.