item.noninvar: Effect Size Measure of Measurement Non-Invariance, dMACS

In misty: Miscellaneous Functions 'T. Yanagida'

Effect Size Measure of Measurement Non-Invariance, dMACS

Description

This function computes the effect size measure dMACS by Nye and Drasgow (2011) and the signed dMACS by Nye et al. (2019) for evaluating the magnitude and the direction of between-group and longitudinal measurement non-invariance or non-equivalence for continuous and ordered categorical items and also computes the expected bias in the mean and variance of the total score.

Usage

item.noninvar(data = NULL, ..., object = NULL, model = NULL, group = NULL,
              ref = NULL, pooled = TRUE, signed = FALSE, cluster = NULL,
              long = FALSE, ordered = FALSE, rescov = NULL, rescov.long = TRUE,
              ident = c("marker", "var", "effect"),
              estimator = c("ML", "MLM", "MLMV", "MLMVS", "MLF", "MLR",
                            "GLS", "WLS", "DWLS", "WLSM", "WLSMV",
                            "ULS", "ULSM", "ULSMV", "DLS", "PML"),
              missing = c("listwise", "pairwise", "fiml", "two.stage",
                          "robust.two.stage", "doubly.robust"),
              print = c("all", "summary", "dmacs", "bias"),
              digits = 3, as.na = NULL, write = NULL, append = TRUE,
              check = TRUE, output = TRUE)

Arguments

data	a data frame. If model = NULL, confirmatory factor analysis based on a measurement model with one factor labeled f comprising all variables in the data frame specified in data for evaluating between-group measurement non-invariance for the grouping variable specified in the argument group is conducted. Longitudinal measurement non-invariance evaluation can only be conducted by specifying the model using the argument model. Note that the cluster variable is excluded from data when specifying cluster. If model is specified, the data frame needs to contain all variables used in the argument model and the cluster variable when specifying the name of the cluster variable in the argument cluster.
...	an expression indicating the variable names in data, e.g., item.noninvar(dat, x1, x2, x2, group = "group"). Note that the operators +, -, ~, :, ::, and ! can also be used to select variables, see 'Details' in the df.subset function.
object	an object of class lavaan, i.e., a fitted latent variable model. Between-group measurement non-invariance is evaluated when specifying a fitted multiple-group model, while longitudinal measurement non-invariance is evaluated when specifying a fitted single-group model with at least two latent variables each representing a factor at different time points.
model	a character vector specifying a measurement model with one factor, or a list of character vectors for specifying a measurement model with more than one factor for evaluating between-group measurement non-invariance when long = FALSE or a list of character vectors for specifying a measurement model with one factor for each time of measurement for evaluating longitudinal measurement non-invariance when specifying long = TRUE. For example, model = c("x1", "x2", "x3", "x4") for specifying a measurement model with one factor labeled f comprising four indicators, or model = list(factor1 = c("x1", "x2", "x3", "x4"), factor2 = c("x5", "x6", "x7", "x8")) for specifying a measurement model with two latent factors labeled factor1 and factor2 each comprising four indicators for evaluating between-group measurement non-invariance, or model = list(time1 = c("ax1", "ax2", "ax3", "ax4"), time2 = c("bx1", "bx2", "bx3", "bx4"), time3 = c("cx1", "cx2", "cx3", "cx4")) for specifying a longitudinal measurement model with three time points comprising four indicators at each time point. This function cannot evaluate longitudinal measurement invariance for a measurement model with more than one factor. Note that the name of each list element is used to label factors, i.e., all list elements need to be named, otherwise factors are labeled with "f1", "f2", "f3" when long = FALSE and with "t1", "t2", "t3" when long = TRUE and so on.
group	either a character string indicating the variable name of the grouping variable in the data frame specified in data or a vector representing the groups for conducting multiple-group analysis to evaluate between-group measurement non-invariance. Note that when specifying a grouping variable with more than two groups, the function compares each group with the reference group specified in the argument ref.
ref	a numeric value or character string indicating the name of the reference group or reference time point. By default, the the first group or time point is used as reference.
pooled	logical: if TRUE (default), the item-level pooled standard deviation is used in the denominator of the effect size measure, if FALSE the item-level standard deviation of the reference group or reference time point is used.
signed	logical: if TRUE, the signed dMACS is computed that incorporates the unsquared differences between groups or time points illustrating the direction of the differences and allowing effects in opposite directions to cancel out (see Nye et al., 2019).
cluster	either a character string indicating the variable name of the cluster variable in data, or a vector representing the nested grouping structure (i.e., group or cluster variable). Note that this option is not available when evaluating measurement invariance for ordered categorical indicators by specifying ordered = TRUE).
long	logical: if TRUE, longitudinal measurement non-invariance evaluation is conducted. The longitudinal measurement model is specified by using the argument model. Note that this function can only deal with a measurement model with one factor at each time point when investigating longitudinal measurement non-invariance. Moreover, this function can only evaluate either between-group or longitudinal measurement non-invariance, but not both at the same time.
ordered	logical: if TRUE, all indicator variables of the measurement model are treated as ordered categorical variables. Note that the function only supports delta parameterization. Also note that all indicators variables need to have the same number of response categories. Accordingly, zero cell counts are not allowed, e.g., zero observations for a response category of an indicator within a group when investigating between-group measurement non-invariance or zero observations for a response category of an indicator at a time point when investigating longitudinal measurement non-invariance.
rescov	a character vector or a list of character vectors for specifying residual covariances, e.g., rescov = c("x1", "x2") for specifying a residual covariance between items x1 and x2, or rescov = list(c("x1", "x2"), c("x3", "x4")) for specifying residual covariances between items x1 and x2, and items x3 and x4.
rescov.long	logical: if TRUE (default), residual covariances between parallel indicators are estimated across time when evaluating longitudinal measurement non-invariance (long = TRUE), i.e., residual variances of the same indicators that are measured at different time points are correlated across all possible time points. Note that residual covariances should be estimated even if the parameter estimates are statistically not significant since indicator-specific systematic variance is likely to correlate with itself over time (Little, 2013, p. 164).
ident	a character string indicating the method used for identifying and scaling latent variables, i.e., "marker" for the marker variable method fixing the first factor loading of the latent variable to 1 and fixing the first intercept to 0, "var" (default) for the fixed variance method fixing the variance of the latent variable to 1 and the latent mean to 0, or "effect" for the effects-coding method using equality constraints so that the average of the factor loading of the latent variable equals 1 and the sum of intercepts equals 0. Note that measurement non-invariance evaluation for ordered categorical indicators can only be conducted based on the fixed variance method ("var").
estimator	a character string indicating the estimator to be used (see 'Details' in the help page of the item.cfa() function). By default, "MLR" is used for CFA models with continuous indicators and "WLSMV" is used for CFA models with ordered categorical indicators. Note that the estimators "ML", "MLM", "MLMV", "MLMVS", "MLF" and "MLR" are not available when ordered = TRUE.
missing	a character string indicating how to deal with missing data, i.e., "listwise" for listwise deletion, "pairwise" for pairwise deletion, "fiml" for full information maximum likelihood method, "two.stage" for two-stage maximum likelihood method, "robust.two.stage" for robust two-stage maximum likelihood method, and "doubly-robust" for doubly-robust method (see 'Details' in the help page of theitem.cfa() function). By default, "fiml" is used for CFA models with continuous indicators and "listwise" is used for CFA models with ordered categorical indicators given that "fiml" is not available for a limited-information estimator used to estimate the CFA model with ordered categorical indicators.
print	a character string or character vector indicating which results to show on the console, i.e. "all" for all results, "summary" for a summary of the specification, "dmacs" for the effect sizes measure dMACS, and "bias" for the expected bias in the mean and variance of the total score. By default, a summary of the specification and the effect size measure dMACS are printed.
digits	an integer value indicating the number of decimal places to be used for displaying results.
as.na	a numeric vector indicating user-defined missing values, i.e., these values are converted to NA before conducting the analysis. Note that as.na() function is only applied to data but not to group or cluster.
write	a character string naming a file for writing the output into either a text file with file extension ".txt" (e.g., "Output.txt") or Excel file with file extension ".xlsx" (e.g., "Output.xlsx"). If the file name does not contain any file extension, an Excel file will be written.
append	logical: if TRUE (default), output will be appended to an existing text file with extension .txt specified in write, if FALSE existing text file will be overwritten.
check	logical: if TRUE (default), argument specification is checked and convergence and model identification checks are conducted for all estimated models.
output	logical: if TRUE (default), output is shown.

Details

Nye and Drasgow (2011) introduced the effect size measure d_{MACS} (Mean and Covariance Structure) for evaluating measurement non-invariance at the item level on a standardized metric similar to Cohen's d (1988) or Glass's (1976) measures:

Effect Size Measure d_{MACS}

d_{MACS} (Nye & Drasgow, 2011) ranging [0, \infty] is based on the predicted response \hat{X}_{iR} to an item i for an individual in the reference group (or reference time point) R and the corresponding response \hat{X}_{iF} for an individual in the focal group (or focal time point) F:

\hat{X}_{iR} = \tau_{iR} + \lambda_{iR}\xi

\hat{X}_{iF} = \tau_{iF} + \lambda_{iF}\xi

where \tau_{iR} and \tau_{iF} are the interepts, \lambda_{iR} and \lambda_{iF} are the factor loadings of item i in the reference and focal group, and \xi is the score on the latent variable.

The effect size evaluating the magnitude of measurement non-invariance is a weighted average difference in predicted responses in standardized metric defined as:

d_{MACS} = \frac{1}{SD_{iP}} \sqrt{\int (\hat{X}_{iR} - \hat{X}_{iF}|\xi)^2 f_F(\xi)d\xi }

where SD_{iP} is the pooled within-group standard deviation of item i across reference and focal group given by

SD_{iP} = \frac{(N_R - 1)SD_R + (N_F - 1)SD_F}{(N_R - 1) + (N_F - 1)}

Note that f_F(\xi) is the distribution of the latent trait \xi in the focal group, which is assumed to have a normal distribution with a mean and variance estimated from the latent factor in the focal group.

Effect Size Measure d_{MACS\_Signed}

d_{MACS\_Signed} (Nye et al., 2019) ranging [-\infty, \infty] incorporates the unsquared differences between predicted response to an item i between two groups:

d_{MACS\_Signed} = \frac{1}{SD_{iP}} \int (\hat{X}_{iR} - \hat{X}_{iF}|\xi) f_F(\xi)d\xi

Note that d_{MACS\_Signed} provides complementary information to the unsigned version by (1) capturing the direction of the difference and (2) allowing cancellation of effects in opposite direction.

Guidelines for Interpreting Effect Size Measure d_{MACS} and d_{MACS\_Signed}

The effect size measures d_{MACS} and d_{MACS\_Signed} represent the differences in both the factor loadings and the intercepts across two groups and can be interpreted based on following guidelines (see Nye et al., 2019):

• Effect Size Measure d_{MACS}:

  • Results of a simulation study provided benchmarks for interpreting d_{MACS}:

    • Small effect: 0.20

    • Medium effect: 0.40

    • Large effect: 0.70

  • The simulation study operationalized effect sizes empirically based on a literature review of journals in organizational behavior and entrepreneurship:

    • Difference in standardized factor loadings: 0.10 (small), 0.20 (medium), and 0.30 (large)

    • Difference in intercept: 0.25 (small), 0.50 (medium), and 0.75 (large)

  • Results also showed that when the sample size (n_g = 250) and/or the number of items (k = 8) were small, d_{MACS} can become greater than 0.20 due to poorly estimated model parameters.

  Note that d_{MACS} does not provide information about the direction of the effect, i.e., it is unclear which group the item is biased against.

• Effect Size Measure d_{MACS\_Signed}:

  • Results of a simulation study provided benchmarks for interpreting d_{MACS\_Signed}:

    • Small effect: |0.40|

    • Medium effect: |0.60|

    • Large effect: |0.80|

  • Simulation study investigated the practical importance of non-invariance when no true latent mean difference exist between groups, i.e., false positive results due to non-invariance:

    • d_{MACS\_Signed} = |0.40| in one out of eight items results in Cohen's d ~ 0.13 and a .13 probability for finding statistically significant differences due to non-invariance.

    • d_{MACS\_Signed} = |0.60| in one out of eight items results in Cohen's d ~ 0.26 and a .85 probability for finding statistically significant differences due to non-invariance.

    • d_{MACS\_Signed} = |0.80| in one out of eight items results in Cohen's d ~ 0.26 and a 1.00 probability for finding statistically significant differences due to non-invariance.

Practical Consequences of Measurement Non-Invariance

The practical consequences of non-invariance can be investigated by computing the amount of the observed mean and variance difference of a scale between groups that can be attributed to non-invariance:

\Delta mean(x_s) = \sum_{1}^{n}\int (\hat{X}_{iR} - \hat{X}_{iF}|\xi) f_F(\xi)d\xi

\Delta var(x_s) = 2C_i\lambda_{iR} \phi_F + C_i^2\phi_F

where X_s is the scale score, \lambda_{iR} is the factor loading of item i in the reference group, C_i is the difference between the factor loading for item i in the reference and focal groups, and \phi_F is the variance of the latent factor in the focal group. According to these formula, two items with high d_{MACS\_Signed} in opposite directions can have no impact on \Delta mean(x_s) and \Delta var(x_s) due to the cancellation effect.

Note that with fewer items in the scale, the practical importance of a single item with high d_{MACS\_Signed} would increase, while a single non-invariant item in a longer measure would have less practical importance. For example, the practical importance of a single non-invariant item with a d_{MACS\_Signed} = 0.40 in a 30-item measure would correspond to a Cohen’s d value of 0.05 and a .14 probability of a statistically significant mean differences in the absence of a true differences between groups. That is, the same cutoffs used for an 8-item measure might not apply to a 30-item measure. Moreover, a measure with more than one non-invariant item would have a greater chance of distorting research outcomes.

In summary, the findings in Nye et al. (2019) suggest that a more nuanced interpretation of the effect size of non-invariance may be required. Accordingly, Lai et al. (2025) notet that cutoff values should be used with caution as the interpretation of the magnitude of non-invariance should be based on many other factors, such as the construct of interest, the grouping variables, the main usage of the instrument, the context of the measurement, and so on.

Value

Returns an object of class misty.object, which is a list with following entries:

call	function call
type	type of analysis
data	data frame including all variables used in the analysis, i.e., indicators for the factor, grouping variable and cluster variable
args	specification of function arguments
model	model specification for the for the configural invariance model
model.fit	fitted lavaan object of the configural invariance model
check	list with the results of the convergence and model identification check for the configural invariance model
result	list with result tables, i.e., summary for the summary of the specification and noninvar for the dMACS effect size measure, expected bias in the mean total score, and expected bias in the variance of the total score

Note

This function is based on modified copies of the functions dmacs_summary dmacs_summary_single, item_dmacs, expected_value, delta_mean_item and delta_var from the dmacs package by David Dueber.

Author(s)

Takuya Yanagida

References

Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Lawrence Erlbaum.

Dueber D (2026). dmacs: Measurement Nonequivalence Effect Size Calculator. R package version 0.1.0.9002. https://github.com/ddueber/dmacs

Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5, 3-8.

Lai, M. H. C., Zhang, Y., Ozcan, M., Tse, W. W. Y., & Miles, A. (2025). fMACS: Generalizing dMACS effect size for measurement noninvariance with multiple groups and multiple grouping variables. Structural Equation Modeling: A Multidisciplinary Journal, 32(4), 638-646. https://doi.org/10.1080/10705511.2025.2484812

Nye, C. D., Bradburn, J., Olenick, J., Bialko, C., & Drasgow, F. (2019). How big are my effects? Examining the magnitude of the effect sizes in studies of measurement equivalence. Organizational Research Methods, 22(3), 678–709. https://doi.org/ 10.1177/1094428118761122

Nye, C., & Drasgow, F. (2011). Effect size indices for analyses of measurement equivalence: Understanding the practical importance of differences between groups. Journal of Applied Psychology, 96(5), 966-980.

See Also

item.invar, item.cfa, multilevel.invar

Examples

## Not run: 
# Load data set "HolzingerSwineford1939" in the lavaan package
data("HolzingerSwineford1939", package = "lavaan")

#----------------------------------------------------------------------------
# Between-Group Measurement Non-Invariance: Continuous Indicators

#..................
# Measurement model with one factor

# Example 1a: Model specification using the argument '...'
item.noninvar(HolzingerSwineford1939, x1, x2, x3, x4, group = "school")

# Example 1b: Alternative model specification without using the argument '...'
item.noninvar(HolzingerSwineford1939[, c("x1", "x2", "x3", "x4")],
              group = HolzingerSwineford1939$school)

# Example 1c: Alternative model specification without using the argument '...'
item.noninvar(HolzingerSwineford1939[, c("x1", "x2", "x3", "x4", "school")], group = "school")

# Example 1d: Alternative model specification using the argument 'model'
item.noninvar(HolzingerSwineford1939, model = c("x1", "x2", "x3", "x4"), group = "school")

# Example 1e: Estimate model and specify the 'object' argument
model <- 'f =~ x1 + x2 + x3 + x4'

fit <- cfa(model, data = HolzingerSwineford1939, group = "school", std.lv = TRUE)
item.noninvar(object = fit)

#..................
# Measurement model with two factors

# Example 2a: Model specification using the argument 'model'
item.noninvar(HolzingerSwineford1939,
              model = list(c("x1", "x2", "x3", "x4"), c("x5", "x6", "x7", "x8")),
              group = "school")

# Example 2b: Model specification using the argument 'model'
model <- 'f1 =~ x1 + x2 + x3 + x4
          f2 =~ x5 + x6 + x7 + x8'

#..................
# Signed dMACS and reference group

# Example 3a: Signed dMACS
item.noninvar(HolzingerSwineford1939, x1, x2, x3, x4, group = "school", signed = TRUE)

# Example 3b: Specify reference group and use SD of the reference group
item.noninvar(HolzingerSwineford1939, x1, x2, x3, x4, group = "school",
              ref = "Pasteur", pooled = FALSE)

#..................
# Residual covariances

# Example 4a: One residual covariance
item.noninvar(HolzingerSwineford1939, model = c("x1", "x2", "x3", "x4"),
              rescov = c("x3", "x4"), group = "school")

# Example 4b: Two residual covariances
item.noninvar(HolzingerSwineford1939, model = c("x1", "x2", "x3", "x4"),
              rescov = list(c("x1", "x4"), c("x3", "x4")), group = "school")

#..................
# Print argument

# Example 5: Request all results
item.noninvar(HolzingerSwineford1939, model = c("x1", "x2", "x3", "x4"),
              group = "school", print = "all")

#----------------------------------------------------------------------------
# Longitudinal Measurement Non-Invariance: Continuous Indicators

# Example 6: Two time points with three indicators at each time point
item.noninvar(HolzingerSwineford1939,
              model = list(c("x1", "x2", "x3"), c("x5", "x6", "x7")), long = TRUE)

#----------------------------------------------------------------------------
# Between-Group Measurement Non-Invariance: Ordered Categorical Indicators
#
# Note that the example analysis for ordered categorical indicators cannot be
# conduct as the data set 'data' is not available.

# Example 7: Two groups
item.noninvar(data, item1, item2, item3, item4, group = "two.group", ordered = TRUE)

#----------------------------------------------------------------------------
# Longitudinal Measurement Non-Invariance: Ordered Categorical Indicators

# Example 8: Two Time Points
item.noninvar(data, model = list(c("aitem1", "aitem2", "aitem3"),
                                 c("bitem1", "bitem2", "bitem3")),
           long = TRUE, ordered = TRUE)
#------------------------------------------------
# Write Results

# Example 9a: Write Results into a text file
item.noninvar(HolzingerSwineford1939, model = c("x1", "x2", "x3", "x4"),
              group = "school", print = "all", write = "Non-Invariance.txt", output = FALSE)

# Example 9b: Write Results into a Excel file
item.noninvar(HolzingerSwineford1939, model = c("x1", "x2", "x3", "x4"),
              group = "school", print = "all", write = "Non-Invariance.xlsx", output = FALSE)

## End(Not run)

misty documentation built on March 6, 2026, 9:08 a.m.