modcomp: Model Comparison

In misty: Miscellaneous Functions 'T. Yanagida'

Model Comparison

Description

This function performs model comparison by providing a table with fit indices for lavaan model objects, information criteria, and F-tests or likelihood ratio tests for models estimated by the function cfa(), sem(), growth(), lavaan() from the lavaan package, lm(), glm(), nls() from the stats package, lmer(), glmer(), glmer.nb() from the lme4 package, lme(), nlme() from the nlme package, glmmTMB() from the glmmTMB package, betareg from the betareg package, or glm.nb() and polr() from the MASS package. By default, the function provides the fit indices CFI, TLI, RMSEA, and SRMR for lavaan model objects and the information criteria AIC, CAIC, BIC, and SABIC.

Usage

modcomp(..., difftest = FALSE,
        print.fit = c("none", "deviance", "chisq", "cfi", "tli", "rmsea", "srmr"),
        fit.robust = c("standard", "scaled", "robust"),
        print.ic = c("all", "default", "none", "aic", "caic", "bic", "sabic",
                     "aicc", "hqc", "hbic", "spbic", "ibic", "sic", "icomp"),
        fit.digits = 3, ic.digits = 0, p.digits = 3, write = NULL, append = TRUE,
        check = TRUE, output = TRUE)

Arguments

...	a fitted model object or sequence of fitted model objects of class "lavaan", "lm", "glm", "nls", "lmerMod", "lmerModLmerTest", "glmerMod", "lme", "nlme", "glmmTMB", "betareg", "negbin", or "polr".
difftest	logical: if TRUE, results of the F-test, chi-square difference test, or likelihood ratio test are printed on the console. Note that the function does not provide difference tests for models fitted by using the betareg() function from the betareg package.
print.fit	a character vector indicating which fit indices to be printed on the console when specifying lavaan objects for the argument ..., i.e., "none" for no fit indices, "deviance" for the deviance (i.e., log-likelihood multiplied by -2), "chisq" for the chi-square value, "cfi" for the comparative fit index, "tli" for the Tucker-Lewis-index, "rmsea" for the root mean square error of approximation, and "srmr" for the standardized root mean squared residual. By default, all fit indices are printed on the console.
fit.robust	a character string indicating which version of the CFI, TLI, and RMSEA to show on the console when using a robust estimation method involving a scaling correction factor for model estimation in lavaan, i.e., "standard" (default when estimator is one of "ML", "MLF", "GLS", "WLS", "DWLS", or "ULS") for fit indices without any non-normality correction, "scaled" (default when estimator is one of "MLMVS", "ULSM", "ULSMV", "DLS", or "PML") for population-corrected robust fit indices with ad hoc non-normality correction, and robust (default when estimator is one of "MLM", "MLMV", "MLR", "WLSM", or "WLSMV") for sample-corrected robust fit indices based on formula provided by Li and Bentler (2006) and Brosseau-Liard and Savalei (2014).
print.ic	a character vector indicating which information criteria to be printed on the console, i.e., "all" for printing all information criteria, "default" for printing the default set of information criteria (i.e., AIC, CAIC, BIC, and SABIC), "none" for no information criteria, "aic" for the Akaike information criterion (AIC), "caic" for the Consistent Akaike's information criterion (CAIC), "bic" for the Bayesian information criterion (BIC), "sabic" for the sample-size adjusted BIC (SABIC), "aicc" for the corrected Akaike information criterion (AICc), "hqc" for the Hannan–Quinn criterion (HQC), "hbic" for the Haughton’s BIC (HBIC), "spbic" for the scaled unit-information prior BIC (SPBIC), "ibic" for the information-matrix-based BIC, "sic" for the stochastic information criterion (SIC), and "icomp" for the Bozdogan information complexity (ICOMP) criterion. The default setting is print.ic = c("aic", "caic", "bic", "sabic)} when specifying more than one model object for the argument \code{...}, otherwise the default setting is \code{print.ic = "none. Note that "spbic", "ibic", "sic", and "icomp" are only available for lavaan model objects.
fit.digits	an integer value indicating the number of decimal places to be used for displaying fit indices when comparing lavaan models.
ic.digits	an integer value indicating the number of decimal places to be used for displaying information criteria.
p.digits	an integer value indicating the number of decimal places to be used for displaying the p-value in the F-test or chi-square difference test.
write	a character string naming a file for writing the output into either a text file with file extension ".txt" (e.g., "Output.txt") or Excel file with file extension ".xlsx" (e.g., "Output.xlsx"). If the file name does not contain any file extension, an Excel file will be written.
append	logical: if TRUE (default), output will be appended to an existing text file with extension .txt specified in write, if FALSE existing text file will be overwritten.
check	logical: if TRUE (default), argument specification is checked.
output	logical: if TRUE (default), output is shown on the console.

Details

Information Criteria

Information criteria are statistical measures that attempt to balance model fit and model complexity to compare competing models for model selection. Most information criteria are based on the log-likelihood with a penalty for complexity, and typically have the following form (Preacher & Yaremych, 2023):

D + f(q, N)

where D is a function of the model's log-likelihood at convergence, whereas f is a function of the number estimated parameters (q) and the sample size (N).

• The Akaike Information Criterion (AIC; Akaike, 1973) is defined as

  \textrm{AIC} = -2LL + 2q

  The AIC is an efficient information criterion, i.e., it will asymptotically choose whichever model minimizes the mean square error of prediction (Vrieze, 2012). However, the AIC is not consistent, i.e., it is expected to pick different models at different N's (Kuha, 2004). Accordingly, AIC is expected to select more complex models as N increases, while in relatively small samples, the penalty for complexity has a greater influence and simpler models are selected (Preacher & Yaremych, 2023).

• The Consistent Akaike Information Criterion (CAIC; Bozdogan, 1987) is defined as

  \textrm{CAIC} = -2LL + q(ln(N) + 1)

  The CAIC modifies the standard AIC to be asymptotically consistent, i.e., it is expected to pick the true model as sample size increases. However, the CAIC is not considered an efficient information criterion. Compared to the BIC, the CAIC has a higher penalty for model complexity making it more consistent but also less efficient than the BIC.

• The Bayesian Information Criterion (BIC; Schwarz, 1978) is defined as

  \textrm{BIC} = -2LL + q\cdot ln(N)

  The BIC is a consistent information criterion, i.e., it will select the true model with probability approach 1 as N increases based on the assumption that (a) the true model is under consideration, (b) the true model's dimension remains fixed as N increases, and (c) the number of parameters in the true model is finite (Vrieze, 2012). Accordingly, BIC tends to select more parsimonious models than AIC and is less subject to choosing more complex models as N increase because the penalty term increases with N.

• The Sample-Size Adjusted Bayesian Information Criterion (SABIC; Sclove, 1987) is defined as

  \textrm{SABIC} = -2LL + q\cdot ln(\frac{N + 2}{24})

  The SABIC is a variant of the BIC that reduces the penalty for complex models and seems to perfom bettern than BIC when the sample size is small to moderate (Chen et al., 2017).

• The Corrected Akaike Information Criterion (AICc; Burnham & Anderson, 2003) is defined as

  \textrm{AICc} = AIC + \frac{2q(q + 1)}{N - q - 1}

  The AICc is a corrected version of the AIC for small sample sizes or when the number of parameters is large relative to the sample size. Note that as the sample size increases the AICc converges to the standard AIC.

• The Hannan–Quinn Criterion (HQC; Hannan & Quinn, 1979) is defined as

  \textrm{HQC} = -2LL + 2q\log{(\log{N})}

  The HQC imposes a penalty that is stronger than AIC but weaker than BIC in large sample as the penalty function decreases with increasing sample size and is often used to select the order of autoregressive processes.

• The Haughton Bayesian Information Criterion (HBIC; Haughton, 1988) is defined as

  \textrm{HBIC} = -2LL + q\log{\frac{N}{2\pi}}

  The HBIC performed well in model selection in simulation studies for structural equation models and had the best overall performance among the investigated information criteria along with the SPBIC (Haughton et al., 1997; Bollen et al., 2014).

• The Scaled Unit-Information Prior Bayesian Information Criterion (SPBIC; Bollen et al., 2012) is defined as

  \textrm{SPBIC}_{\textrm{Case 1}} = -2LL + q(1 - \frac{q}{\hat{\theta}^{'} \textrm{FIM} \hat{\theta}}), \textrm{or}

  \textrm{SPBIC}_{\textrm{Case 2}} = -2LL + \hat{\theta}^{'} \textrm{FIM} \hat{\theta},

  depending on whether the product of the vector of estimated model parameters (\hat{\theta}) and the observed information matrix (FIM) exceeds the number of estimated parameters (Case 1) or not (Case 2). The SPBIC performed well in model selection in a simulation study for structural equation models, had the best overall performance among the investigated information criteria along with the HBIC (Bollen et al., 2014), and exhibited a better performance along with the IBIC than BIC and HBIC when the sample size was small (Bollen et al., 2012).

• The Information Matrix-Based Bayesian Information Criterion (IBIC; Bollen et al., 2014) is defined as

  \textrm{IBIC} = -2LL + q\log{\frac{N}{2 \pi}} + \log{\det{\textrm{FIM}}}

  The IBIC performed well in model selection in a simulation study for structural equation models (Bollen et al., 2014) and exhibited a better performance along with the SPBIC than BIC and HBIC when the sample size was small (Bollen et al., 2012).

• The Stochastic Information Criterion (SIC; Rissanen, 1989) is defined as

  \textrm{SIC} = -2LL + q\log{N} + \log{\det{\textrm{FIM}}} = -2LL - \log{\det{\textrm{ACOV}}}

  The SIC performed well relative to other information criteria in two simulation studies of structural equation models applied to behavior genetic models (Markon & Krueger, 2004).

• The Information Complexity Criterion (ICOMP; Bozdogan & Haughton, 1988) is defined as

  \textrm{ICOMP} = -2LL + 2C(\hat{\Sigma}_{Model})

  where C represents a complexity measure and \hat{\Sigma}_{Model} represents the estimated covariance matrix of the parameter vector estimated by the model, i.e., inverse Fisher information matrix (see Akman, 2010). The ICOMP penalizes the covariance complexity of the model instead of the number of estimated parameters.

In practice, it may be sensible to choose information criteria that emphasize consistency or efficiency as consistency and efficiency cannot be maximized simultaneously (Claeskens & Hjort, 2008). More specifically, a criterion that emphasizes efficiency such as AIC should be used when prediction or cross-validation is important, whereas a criterion that emphasizes consistency such as BIC should be used when we want to identify a model that best approximates the truth. Note that there is no such thing as a correct model, there are models that cross-validate better than others, and there are models that better reflect the true data-generating process (Preacher & Yaremych, 2023).

Value

Returns an object of class misty.object, which is a list with following entries:

call	function call
type	type of analysis
class	object class of the models specified in the argument ...
model	models specified in the argument ...
args	specification of function arguments
result	result table

Note

The computation of AICc, HQC, HBIC, SPBIC, IBIC, SIC, and ICOMP are based on the moreFitIndices function from the semTools package by Terrence D. Jorgensen, Sunthud Pornprasertmanit, Alexander M. Schoemann, and Yves Rosseel.

Author(s)

Takuya Yanagida

References

Akaike, H. (1973). Information theory and an extension of the maximum likelihood principle. In B. N. Petrov & B. F. Csaki (Eds.), Second International Symposium on Information Theory, (pp. 267-281). Academiai Kiado.

Akman, O. (2010). Information complexity based modeling in the presence of length-biased sampling. Journal of Statistical Theory and Practice, 4(1), 45-55. https://doi.org/10.1080/15598608.2010.10411972

Bollen, K. A., Harden, J. J., Ray, S., & Zavisca, J. (2014). BIC and alternative Bayesian information criteria in the selection of structural equation models. Structural Equation Modeling: A Multidisciplinary Journal, 21(1), 1–19. https://doi.org/10.1080/10705511.2014.856691

Bollen, K. A., Ray, S., Zavisca, J., & Harden, J. J. (2012). A comparison of Bayes factor approximation methods including two new methods. Sociological Methods & Research, 41(2), 294-324. https://doi.org/10.1177/00491241124523

Burnham, K., & Anderson, D. (2003). Model selection and multimodel inference: A practical–theoretic approach. Springer.

Brosseau-Liard, P. E., & Savalei, V. (2014) Adjusting incremental fit indices for nonnormality. Multivariate Behavioral Research, 49, 460-470. https://doi.org/10.1080/00273171.2014.933697

Chen, Q., Luo, W., Palardy, G. J., Glaman, R., & McEnturff, A. (2017). The efficacy of common fit indices for enumerating classes in growth mixture models when nested data structure is ignored: A Monte Carlo study. SAGE Open, 7(1). https://doi:10.1177/2158244017700459

Claeskins, G., & Hjort, N. L. (2008). Model selection and model averaging. Cambridge University Press.

Hannan, E.J. and Quinn, B.G. (1979) The determination of the order of an autoregression. Journal of the Royal Statistical Society, 41, 190-195.

Haughton, D. M. A. (1988). On the choice of a model to fit data from an exponential family. The Annals of Statistics, 16(1), 342-355.

Haughton, D., Oud, J., & Jansen, R. (1997). Information and other criteria in structural equation model selection. Communications in Statistics, Part B - Simulation and Computation, 26(4), 1477-1516.

Kuha, J. (2004). AIC and BIC: Comparisons of assumptions and performance. Sociological Methods & Research, 33, 188-229.

Li, L., & Bentler, P. M. (2006). Robust statistical tests for evaluating the hypothesis of close fit of misspecified mean and covariance structural models. UCLA Statistics Preprint #506. University of California.

Markon, K. E., & Krueger, R. F. (2004). An empirical comparison of information-theoretic selection criteria for multivariate behavior genetic models. Behavior Genetics, 34, 593-610.

Preacher, K. K., & Yaremych, H. E. (2023). Model selection in structural equation modeling. In R. H. Hoyle (Ed.), Handbook of structural equation modeling (2nd ed., pp. 206-222). The Guilford Press.

Rissanen, J. (1989). Stochastic complexity in statistical inquiry. World Scientific.

Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6(2), 461-464.

Sclove, L. (1987). Application of model-selection criteria to some problems in multivariate analysis. Psychometrika, 52(3), 333-343.

Jorgensen, T. D., Pornprasertmanit, S., Schoemann, A. M., & Rosseel, Y. (2025). semTools: Useful tools for structural equation modeling. R package version 0.5-7. Retrieved from https://CRAN.R-project.org/package=semTools

Vrieze, S.I. (2012) Model selection and psychological theory: A discussion of the differences between the Akaike Information Criterion (AIC) and the Bayesian Information Criterion (BIC). Psychological Methods, 17, 228-243. https://doi.org/10.1037/a0027127

Examples

## Not run: 
#————————————————————————————————————————————————————————————————————————————
# lavaan Model Objects

# Load lavaan package
library(lavaan)

# Model specification
HS.model <- 'visual  =~ x1 + x2 + x3
             textual =~ x4 + x5 + x6
             speed   =~ x7 + x8 + x9'

# Model estimation
fit1 <- cfa(HS.model, data = HolzingerSwineford1939)
fit2 <- cfa(HS.model, data = HolzingerSwineford1939, orthogonal = TRUE)

# Example 1a: Model fit for model 'fit1', default setting
modcomp(fit1)

# Example 1b: Model comparison, default setting
modcomp(fit1, fit2)

# Example 1c: Model comparison, request likelihood ratio test
modcomp(fit1, fit2, difftest = TRUE)

# Example 1d: Model comparison, request default information criteria and AICc
modcomp(fit1, fit2, print.ic = c("default", "aicc"))

# Example 1e: Model comparison, request all information criteria
modcomp(fit1, fit2, print.ic = "all")

# Example 1f: Model fit indices, request all information criteria
modcomp(fit1, print.ic = "all")

#————————————————————————————————————————————————————————————————————————————
# lm Model Objects

# Model estimation
fit1 <- lm(mpg ~ cyl, data = mtcars)
fit2 <- lm(mpg ~ cyl + disp, data = mtcars)

# Example 2: Model comparison, requested F test
modcomp(fit1, fit2, difftest = TRUE)

#————————————————————————————————————————————————————————————————————————————
# Write Results

# Example 3a: Write Results into a text file
modcomp(fit1, fit2, difftest = TRUE, write = "Model_Comparison.txt")

# Example 3b: Write Results into a Excel file
modcomp(fit1, fit2, difftest = TRUE, write = "Model_Comparison.xlsx")

## End(Not run)

misty documentation built on Aug. 2, 2026, 9:06 a.m.