multilevel.omega: Multilevel Composite Reliability

In misty: Miscellaneous Functions 'T. Yanagida'

Multilevel Composite Reliability

Description

This function computes point estimate and Monte Carlo confidence interval for the multilevel composite reliability defined by Lai (2021) for a (1) within-cluster construct, (2) shared cluster-level construct, and (3) individual and configural cluster construct by calling the cfa function in the R package lavaan. By default, the function prints level-specific multilevel composite reliability indices for an individual and configural cluster construct with 95% Monte Carlo confidence interval based on Huber-White standard errors.

Usage

multilevel.omega(data, ..., cluster, rescov = NULL,
                 const = c("within", "shared", "config"), fix.resid = NULL,
                 se = c("none", "standard", "robust.huber.white"),
                 optim.method = c("nlminb", "em"), missing = c("listwise", "fiml"),
                 nrep = 100000, seed = NULL, conf.level = 0.95,
                 print = c("all", "omega", "item"), digits = 2, r.digits = 3,
                 as.na = NULL, write = NULL, append = TRUE, check = TRUE,
                 output = TRUE)

Arguments

data	a data frame. Multilevel confirmatory factor analysis based on a measurement model with one factor at the Within level and one factor at the Between level comprising all variables in the data frame is conducted. Note that the cluster variable specified in cluster is excluded from data when specifying the argument cluster using the variable name of the cluster variable.
...	an expression indicating the variable names in data, e.g., multilevel.omega(dat, x1, x2, x3, cluster = "cluster"). Note that the operators +, -, ~, :, ::, and ! can also be used to select variables, see 'Details' in the df.subset function.
cluster	either a character string indicating the variable name of the cluster variable in data, or a vector representing the nested grouping structure (i.e., group or cluster variable).
rescov	a character vector or a list of character vectors for specifying residual covariances at the Within level, e.g. rescov = c("x1", "x2") for specifying a residual covariance between indicators x1 and x2 at the Within level or rescov = list(c("x1", "x2"), c("x3", "x4")) for specifying residual covariances between indicators x1 and x2, and indicators x3 and x4 at the Within level. Note that residual covariances at the Between level cannot be specified using this function.
const	a character string indicating the type of construct(s), i.e., "within" for within-cluster constructs, "shared" for shared cluster-level constructs, and "config" (default) for individual and configural constructs.
fix.resid	a character vector for specifying residual variances to be fixed at 0 at the Between level, e.g., fix.resid = c("x1", "x3") to fix residual variances of indicators x1 and x2 at the Between level at 0. Note that it is also possible to specify fix.resid = "all" which fixes all residual variances at the Between level at 0 in line with the strong factorial measurement invariance assumption across cluster.
se	a character string indicating the standard errors used for computing Monte Carlo confidence intervals, i.e., "none" for no standard errors, "standard" for conventional standard error based on inverting the expected observed or first.order information matrix, and "robust.huber.white" for the 'MLR' (aka pseudo ML, Huber-White) approach. Note that se = "none" saves computation time by not computing standard errors and Monte Carlo confidence intervals.
optim.method	a character string indicating the optimizer, i.e., "nlminb" (default) for the unconstrained and bounds-constrained quasi-Newton method optimizer and "em" for the Expectation Maximization (EM) algorithm.
missing	a character string indicating how to deal with missing data, i.e., "listwise" for listwise deletion or "fiml" (default) for full information maximum likelihood (FIML) method.
nrep	an integer value indicating the number of Monte Carlo repetitions for computing confidence intervals.
seed	a numeric value specifying the seed of the random number generator for computing the Monte Carlo confidence interval.
conf.level	a numeric value between 0 and 1 indicating the confidence level of the interval.
print	a character vector indicating which results to show, i.e. "all" (default) for all results "omega" for the composite reliability omega and "item" for item statistics.
digits	an integer value indicating the number of decimal places to be used for displaying mean, standard deviation, minimum, maximum, skewness, and kurtosis.
r.digits	an integer value indicating the number of decimal places to be used for displaying multilevel coefficient alpha, ICC(1), and standardized factor loadings.
as.na	a numeric vector indicating user-defined missing values, i.e. these values are converted to NA before conducting the analysis. Note that as.na() function is only applied to data but not to cluster.
write	a character string naming a file for writing the output into either a text file with file extension ".txt" (e.g., "Output.txt") or Excel file with file extension ".xlsx" (e.g., "Output.xlsx"). If the file name does not contain any file extension, an Excel file will be written.
append	logical: if TRUE (default), output will be appended to an existing text file with extension .txt specified in write, if FALSE existing text file will be overwritten.
check	logical: if TRUE (default), argument specification, convergence and model identification is checked.
output	logical: if TRUE (default), output is shown.

Details

Geldhof et al. (2014) introduced a multilevel confirmatory factor analysis approach for estimating score reliability at each level. This approach uses the estimated factor loadings, factor variances, and residual variances from both the within-level and between-level model to compute the within-level reliability \tilde{\omega}^w and the between-level reliability \tilde{\omega}^b based on McDonald's \omega. However, Lai (2021) pointed out two limitations of this approach. First, it does not account for different conceptualizations of constructs in multilevel data, where constructs may have different meaning (Stapleton et al., 2016). Second, the between-level reliability \tilde{\omega}^b in Geldhof et al. (2014) is a measure of reliability of the latent cluster means, which ignores the sampling error in the observed cluster means (see Lüdtke et al., 2011). Consequently, \tilde{\omega}^b substantially overestimates the true reliability of observed between-level composite scores. To address these limitations, Lai (2021) proposed multilevel reliability coefficients tailored for different types of constructs to estimate the reliability of the observed scores at each level, i.e., (1) within-cluster construct, (2) shared cluster-level construct, and (3) individual and configural constructs:

Within-Cluster Construct

The latent variable representing the within-cluster construct is only meaningful at the within level. For example, students' popularity among their peers within the same classroom based on sociometric ratings. In this case any variation across classrooms should be irrelevant to the construct. Thus, a saturated between-level model is specified. The reliability for a within-level composite, Z^w_{ij}, measuring a within-cluster construct is

\omega^w = \frac{ (\sum^p_{k = 1}\lambda^w_k)^2\phi^w} {(\sum^p_{k = 1}\lambda^w_k)^2\phi^w + \mathbf{1}^{\prime}\mathbf{\Theta}^w\mathbf{1}}

Note that this formula is essentially the same as the formula for the individual construct of an individual and configural construct, except that the factor loadings are specific to the within level as the between-level model is saturated.

Shared Cluster-Level Construct

The latent variable representing the shared construct is only meaningful at the between level. For example, safety of a neighborhood, effectiveness of a teacher, or organizational climate are measured by multiple informants at the within level. In these cases, any variation within clusters should be irrelevant to the construct. Thus, a saturated within-level model is specified. The reliability for a between-level composite, Z^b_{j}, measuring a shared construct is

\omega^b = \frac{(\sum^p_{k = 1}\lambda^b_k)^2\phi^b} {(\sum^p_{k = 1}\lambda^b_k)^2\phi^b + \mathbf{1}^{\prime}\mathbf{\Theta}^b\mathbf{1} + \mathbf{1}^{\prime}\mathbf{\sum}^w\mathbf{1}/\tilde{n}}

where \mathbf{\sum}^w is the variance covariance matrix of the saturated within-level model and \tilde{n} is the harmonic mean of cluster sizes \tilde{n} = \frac{1}{\sum^p_{k = 1}\frac{1}{n_j}}.

Individual and Configural Construct

The individual construct is defined at the within level (e.g., individual student achievement), whereas the configural construct consists of cluster averages of individual constructs (e.g., mean student achievement of a classroom). Note that the factor loadings need to be constrained to be equal across levels, i.e., \mathbf{\lambda}^w = \mathbf{\lambda}^b so that the latent variables at the within level, \eta^w, and between level, \eta^b, are on the same metric.

Individual Construct

The reliability for a within-level composite, Z^w_{ij}, measuring an individual construct is

\omega^w = \frac{ (\sum^p_{k = 1}\lambda_k)^2\phi^w} {(\sum^p_{k = 1}\lambda_k)^2\phi^w + \mathbf{1}^{\prime}\mathbf{\Theta}^w\mathbf{1}}

Configural Construct

The reliability for a between-level composite, Z^b_{j}, measuring a configural construct is

\omega^b = \frac{(\sum^p_{k = 1}\lambda_k)^2\phi^b}{(\sum^p_{k = 1}\lambda_k)^2(\phi^b + \phi^w / \tilde{n}) + \mathbf{1}^{\prime}\mathbf{\Theta}^b\mathbf{1} + \mathbf{1}^{\prime}\mathbf{\Theta}^w\mathbf{1}/\tilde{n}}

Composite Measuring an Individual and Configural Construct

The reliability for an overall composite, Z_{ij}, measuring an individual construct capturing the population variance of both the within-level and the between-level components of the true score and the errors is

\omega^{2l} = \frac{(\sum^p_{k = 1}\lambda_k)^2(\phi^w + \phi^b)}{(\sum^p_{k = 1}\lambda_k)^2(\phi^w + \phi^b) + \mathbf{1}^{\prime}\mathbf{\Theta}^b\mathbf{1} + \mathbf{1}^{\prime}\mathbf{\Theta}^w\mathbf{1}}

Note that \omega^{2l} is simply the ratio of true variance to total variance of the observed scores. In practice, \omega^{2l} is not the most interesting coefficient because researchers are mainly interested in distinguishing within- and between effects (Castro-Alvarez et al., 2026).

Note that the multilevel factor model assumes p items measuring one latent construct at the within level (within-cluster construct), between level (shared cluster-level construct), or within- and between level (individual and configural construct). Local independence is assumed, \Theta^w_j = diag[\theta^w_{11} ... \theta^w_{pp}] and \Theta^b_j = diag[\theta^b_{11} ... \theta^b_{pp}], but composite reliability can be computed when this assumption is violated by specifying the rescov argument. The model assumes equal factor loadings across clusters, \mathbf{\lambda}^w_j = \mathbf{\lambda}^w, and homogeneity of error covariances across cluster, \mathbf{\Theta}^w_j = \mathbf{\Theta}^w. In case of an individual and configural construct, the model also assumes cross-level measurement invariance, i.e., \mathbf{\lambda}^w = \mathbf{\lambda}^b.

Value

call	function call
type	type of analysis
data	data frame specified in data including the group variable specified in cluster
args	specification of function arguments
model	specified model
model.fit	fitted lavaan object (mod.fit)
check	results of the convergence and model identification check
result	list with result tables, i.e., omega for the coefficient omega including Monte Carlo confidence interval and itemstat for descriptive statistics

Note

The function uses the functions lavInspect, lavTech, and lavNames, provided in the R package lavaan by Yves Rosseel (2012). The internal function .internal.mvrnorm is a copy of the mvrnorm function in the package MASS by Venables and Ripley (2002).

Author(s)

Takuya Yanagida [e-mail]

References

Castro-Alvarez, S., Bringmann, L. F., Back, J., & Liu, S. (2026). The many reliabilities of psychological dynamics: An overview of statistical approaches to estimate the internal consistency reliability of intensive longitudinal data. Psychological Methods, 31(2), 281–296. https://doi.org/10.1037/met0000778

Geldhof, G. J., Preacher, K. J., & Zyphur, M. J. (2014). Reliability estimation in a multilevel confirmatory factor analysis framework. Psychological Methods, 19, 72-91. http://dx.doi.org/10.1037/a0032138

Lai, M. H. C. (2021). Composite reliability of multilevel data: It’s about observed scores and construct meanings. Psychological Methods, 26(1), 90–102. https://doi.org/10.1037/met0000287

Lüdtke, O., Marsh, H. W., Robitzsch, A., & Trautwein, U. (2011). A 2 2 taxonomy of multilevel latent contextual models: Accuracy bias tradeoffs in full and partial error correction models. Psychological Methods, 16, 444-467. http://dx.doi.org/10.1037/a0024376

Rosseel, Y. (2012). lavaan: An R Package for Structural Equation Modeling. Journal of Statistical Software, 48, 1-36. https://doi.org/10.18637/jss.v048.i02

Stapleton, L. M., Yang, J. S., & Hancock, G. R. (2016). Construct meaning in multilevel settings. Journal of Educational and Behavioral Statistics, 41(5), 481–520. https://doi.org/10.3102/1076998616646200

Venables, W. N., Ripley, B. D. (2002).Modern Applied Statistics with S (4th ed.). Springer. https://www.stats.ox.ac.uk/pub/MASS4/.

See Also

multilevel.alpha, item.omega, multilevel.cfa, multilevel.fit, multilevel.invar, multilevel.cor, multilevel.descript

Examples

## Not run: 

# Load data set "Demo.twolevel" in the lavaan package
data("Demo.twolevel", package = "lavaan")

#————————————————————————————————————————————————————————————————————————————
# Cluster Variable Specification

# Example 1a: Specification using the argument '...'
multilevel.omega(Demo.twolevel, y1:y4, cluster = "cluster")

# Example 1b: Alternative specification with cluster variable 'cluster' in 'data'
multilevel.omega(Demo.twolevel[, c("y1", "y2", "y3", "y4", "cluster")], cluster = "cluster")

# Example 1b: Alternative specification with cluster variable 'cluster' not in 'data'
multilevel.omega(Demo.twolevel[, c("y1", "y2", "y3", "y4")], cluster = Demo.twolevel$cluster)

#————————————————————————————————————————————————————————————————————————————
# Type of Construct

# Example 2a: Within-Cluster Construct
multilevel.omega(Demo.twolevel[, c("y1", "y2", "y3", "y4")],
                 cluster = Demo.twolevel$cluster, const = "within")

# Example 2b: Shared Cluster-Level Construct
multilevel.omega(Demo.twolevel, y1, y2, y3, y4, cluster = "cluster", const = "shared")

# Example 2c: Configural Construct
multilevel.omega(Demo.twolevel, y1, y2, y3, y4, cluster = "cluster", const = "config")

#————————————————————————————————————————————————————————————————————————————
# Residual (Co)Variances

# Example 3a: Residual covariance between "y4" and "y5" at the Within level
multilevel.omega(Demo.twolevel, y1, y2, y3, y4, cluster = "cluster", const = "config",
                 rescov = c("y3", "y4"))

# Example 3b: Residual variances of 'y1' at the Between level fixed at 0
multilevel.omega(Demo.twolevel, y1, y2, y3, y4, cluster = "cluster", const = "config",
                 fix.resid = c("y1", "y2"))

#————————————————————————————————————————————————————————————————————————————
# Arguments 'se' and 'print'

# Example 4a: No confidence intervals to speed up computation
 multilevel.omega(Demo.twolevel, y1:y4, cluster = "cluster", se = "none")

# Example 4b: Omega and item statistics
multilevel.omega(Demo.twolevel, y1:y4, cluster = "cluster", print = "all")

#————————————————————————————————————————————————————————————————————————————
# Write Results

# Example 5a: Write results into a text file
multilevel.omega(Demo.twolevel[, c("y1", "y2", "y3", "y4")],
                 cluster = Demo.twolevel$cluster, write = "Multilevel_Omega.txt")

# Example 5b: Write results into a Excel file
multilevel.omega(Demo.twolevel, y1, y2, y3, y4, cluster = "cluster",
                 write = "Multilevel_Omega.xlsx")

## End(Not run)

misty documentation built on Aug. 2, 2026, 9:06 a.m.