good.of.fit: Goodness-of-Fit Indices for Latent Probability Models

In ForceChoice: Forced-Choice Modeling Based on Item Response Theory and Cognitive Diagnostic Models

Goodness-of-Fit Indices for Latent Probability Models

Description

Computes a comprehensive suite of model-fit diagnostics for probability models used in IRT, CDM, latent class, and forced-choice applications. Indices include likelihood-based information criteria, limited-information goodness-of-fit statistics (M2, RMSEA, SRMSR), incremental/comparative fit indices (CFI, TLI), pseudo-R^2 measures, residual diagnostics, local-dependence statistics (Yen's Q3), and posterior classification diagnostics.

Usage

good.of.fit(
  par.vec,
  loglik.fun,
  prob.fun,
  response,
  npar,
  pi,
  pi.fun = NULL,
  alpha = 0.05,
  response.type = c("auto", "binary", "polytomous", "nominal", "ordinary"),
  response.K = NULL,
  bivariate.groups = NULL,
  nominal.groups = NULL,
  exclusive.groups = NULL,
  n.boot = 1000L,
  ...
)

Arguments

par.vec	Numeric vector of free model parameters.
loglik.fun	Function returning the fitted model log-likelihood.
prob.fun	Function returning conditional response probabilities at the latent support points.
response	An N \times I response matrix (binary, polytomous, or nominal coding depending on response.type).
npar	Integer; number of free parameters in the model.
pi	Numeric vector of latent support weights (summing to 1).
pi.fun	Optional function returning latent support weights given par.vec and .... If NULL, pi is used as-is.
alpha	Tail probability for the RMSEA confidence interval. Default 0.05 yields a 90% CI (two-sided 2\alpha). Use 0.025 for 95% or 0.005 for 99%.
response.type	Character string specifying the response coding: "auto" (default; detects from probability dimensions), "binary" (each column is a binary indicator), "polytomous" (stacked category probabilities, requires response.K), or "nominal" (expanded binary indicators with mutually exclusive categories within groups, requires nominal.groups).
response.K	Integer vector of length I giving the number of categories for each response column. Required when response.type = "polytomous".
bivariate.groups	Optional integer vector of length I; response column pairs within the same group are excluded from bivariate limited-information moments. Useful for forced-choice or testlet designs where within-block comparisons are structural zeros.
nominal.groups	Integer vector of length matching the number of expanded binary indicators for response.type = "nominal". Indicators sharing the same group value belong to the same original nominal item/block.
exclusive.groups	Deprecated alias for nominal.groups.
n.boot	Integer; number of bootstrap replicates for bootstrapped diagnostics. Default 1000; set to 0 to skip.
...	Additional arguments passed to loglik.fun, prob.fun, and pi.fun.

Value

An object of class "good.of.fit" (a list). Key components:

LogLik, deviance

Log-likelihood and deviance.

npar, N, nitems

Parameter count, sample size, number of items/indicators.

AIC, AICc, BIC, CAIC, SABIC, HQIC

Information criteria.

M2.value, df, p.value

M2 statistic, degrees of freedom, asymptotic p-value.

RMSEA2, RMSEA.lower, RMSEA.upper, RMSEA.CI.level, RMSEA.p.close

RMSEA estimate, confidence interval, interval level, close-fit test p-value.

SRMSR, RMSR

Standardized and raw root mean square residuals.

CFI, TLI, IFI

Incremental fit indices.

McFadden.R2, Nagelkerke.R2, CoxSnell.R2, AldrichNelson.R2, VeallZimmermann.R2

Pseudo-R^2 measures.

residuals, residuals.standardized, correlation.residuals

Residual vectors.

moments.observed, moments.predicted

First- and second-order marginal moments.

q3.mean, q3.abs.max, q3.adjusted.abs.max, q3.p95

Yen's Q3 local dependence statistics.

mean.max.posterior, median.max.posterior, min.max.posterior

Posterior classification diagnostics.

null.LogLik, null.M2.value, null.df

Null (independence) model diagnostics.

LRT, LRT.df, LRT.p

Likelihood ratio test against the null model.

Likelihood and Information Criteria

Let \ell = \log L(\hat{\boldsymbol{\psi}}) be the maximized log-likelihood, N the sample size, and p the number of free parameters (npar).

Deviance:

D = -2\ell

AIC (Akaike, 1974):

\text{AIC} = D + 2p

AICc (Hurvich & Tsai, 1989), small-sample correction:

\text{AICc} = \text{AIC} + \frac{2p(p+1)}{N - p - 1}, \quad N > p + 1

BIC (Schwarz, 1978):

\text{BIC} = D + p \log N

CAIC (Bozdogan, 1987), consistent AIC:

\text{CAIC} = D + p(\log N + 1)

SABIC (Sclove, 1987), sample-size adjusted BIC:

\text{SABIC} = D + p \log\bigl(\frac{N + 2}{24}\bigr)

HQIC (Hannan & Quinn, 1979):

\text{HQIC} = D + 2p \log(\log N), \quad N > e

Limited-Information M2 Family

The limited-information framework (Maydeu-Olivares & Joe, 2005, 2006) uses first- and second-order marginal moments rather than the full I-way contingency table. This is essential for sparse high-dimensional categorical responses where full-information tests break down.

Let \mathbf{m}(\boldsymbol{\psi}) be the vector of model-implied univariate and bivariate moments (dimension M), and let \hat{\boldsymbol{\Xi}}_2 be their asymptotic covariance matrix under the model.

M2 statistic:

M_2 = N\, \mathbf{e}'\hat{\mathbf{C}}_2\mathbf{e},

where \mathbf{e} = \mathbf{p} - \boldsymbol{\pi}(\hat{\boldsymbol{\psi}}) is the vector of marginal residuals, and \hat{\mathbf{C}}_2 = \boldsymbol{\Delta}_c(\boldsymbol{\Delta}_c' \hat{\boldsymbol{\Xi}}_2 \boldsymbol{\Delta}_c)^{-1}\boldsymbol{\Delta}_c' with \boldsymbol{\Delta}_c the orthogonal complement of the Jacobian matrix \boldsymbol{\Delta} = \partial\boldsymbol{\pi}/\partial\boldsymbol{\psi}'.

Under correct model specification, M_2 \sim \chi^2_{M - p} asymptotically.

RMSEA (Root Mean Square Error of Approximation):

\text{RMSEA} = \sqrt{\max\!\left(0,\; \frac{M_2 - df}{N \cdot df}\right)}, \qquad df = M - p

Confidence intervals are obtained via non-central \chi^2 inversion. The close-fit test evaluates H_0: \text{RMSEA} \le 0.05.

SRMSR (Standardized Root Mean Square Residual):

\text{SRMSR} = \sqrt{ \frac{1}{J} \sum_{i < j} (r_{ij}^{\text{obs}} - r_{ij}^{\text{model}})^2 },

where r_{ij} are the observed and model-implied correlation residuals, summed over J unique off-diagonal pairs.

McDonald's NCI (Non-Centrality Index):

\text{NCI} = \exp\!\left(-\frac{\max(M_2 - df, 0)}{2N}\right)

Incremental / Comparative Fit Indices

CFI (Comparative Fit Index):

\text{CFI} = 1 - \frac{\max(M_2 - df, 0)} {\max(M_2^{\text{null}} - df^{\text{null}}, M_2 - df, 0)},

where the null (independence) model has M_2^{\text{null}} with df^{\text{null}} degrees of freedom.

TLI (Tucker–Lewis Index, a.k.a. NNFI):

\text{TLI} = \frac{M_2^{\text{null}} / df^{\text{null}} - M_2 / df} {M_2^{\text{null}} / df^{\text{null}} - 1}

IFI (Incremental Fit Index, a.k.a. BFI):

\text{IFI} = \frac{M_2^{\text{null}} - M_2}{M_2^{\text{null}} - df}

Pseudo-R^2 Measures

McFadden's R^2:

R^2_{\text{McF}} = 1 - \frac{\ell}{\ell_0}

McFadden's adjusted R^2:

R^2_{\text{McF,adj}} = 1 - \frac{\ell - p}{\ell_0}

Cox–Snell R^2:

R^2_{\text{CS}} = 1 - \exp\!\left(\frac{2}{N}(\ell_0 - \ell)\right)

Nagelkerke R^2:

R^2_{\text{N}} = \frac{R^2_{\text{CS}}} {1 - \exp(2\ell_0 / N)}

Aldrich–Nelson R^2:

R^2_{\text{AN}} = \frac{G^2}{G^2 + N}, \quad G^2 = D_0 - D

Veall–Zimmermann R^2:

R^2_{\text{VZ}} = R^2_{\text{AN}} \times \frac{-2\ell_0 + N}{-2\ell_0}

where \ell_0 is the null-model (independence) log-likelihood and D_0 the corresponding deviance.

Yen's Q3 Local Dependence

For each pair of items (i, k), the residual correlation is

Q_{3,ik} = \text{Corr}(r_{ji}, r_{jk}), \qquad r_{ji} = Y_{ji} - E[Y_{ji} \mid \hat{\boldsymbol{\theta}}_j],

where residuals are computed from posterior expected scores. Adjusted Q3 values subtract the mean Q3 across all pairs to center the distribution (Christensen, Makransky, & Horton, 2017).

Heuristic Guidelines

The print.summary.good.of.fit() method reports descriptive recommended ranges based on common conventions. These are heuristics, not decision rules; fit cutoffs depend on model family, dimensionality, category sparsity, local dependence, and sample size.

• RMSEA: \le 0.05 close fit; \le 0.08 reasonable; \le 0.10 mediocre (Browne & Cudeck, 1993)

• SRMSR: \le 0.08 acceptable (Hu & Bentler, 1999)

• CFI / TLI: \ge 0.95 good; \ge 0.90 acceptable

References

Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19, 716–723.

Bozdogan, H. (1987). Model selection and Akaike's Information Criterion (AIC): The general theory and its analytical extensions. Psychometrika, 52, 345–370. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1007/BF02294361")}

Browne, M. W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K. A. Bollen & J. S. Long (Eds.), Testing structural equation models (pp. 136–162). Sage.

Christensen, K. B., Makransky, G., & Horton, M. C. (2017). Critical values for Yen's Q3. Applied Psychological Measurement, 41, 178–194. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1177/0146621616677520")}

Hannan, E. J., & Quinn, B. G. (1979). The determination of the order of an autoregression. Journal of the Royal Statistical Society: Series B, 41, 190–195.

Hu, L. T., & Bentler, P. M. (1999). Cutoff criteria for fit indexes in covariance structure analysis. Structural Equation Modeling, 6, 1–55. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1080/10705519909540118")}

Hurvich, C. M., & Tsai, C. L. (1989). Regression and time series model selection in small samples. Biometrika, 76, 297–307. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1093/biomet/76.2.297")}

MacCallum, R. C., Browne, M. W., & Sugawara, H. M. (1996). Power analysis and determination of sample size for covariance structure modeling. Psychological Methods, 1, 130–149.

Maydeu-Olivares, A., & Joe, H. (2005). Limited- and full-information estimation and goodness-of-fit testing in 2^n contingency tables: A unified framework. Journal of the American Statistical Association, 100, 1009–1020.

Maydeu-Olivares, A., & Joe, H. (2006). Limited information goodness-of-fit testing in multidimensional contingency tables. Psychometrika, 71, 713–732. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1007/s11336-005-1295-9")}

McFadden, D. (1974). Conditional logit analysis of qualitative choice behavior. In P. Zarembka (Ed.), Frontiers in econometrics (pp. 105–142). Academic Press.

Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6, 461–464. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1214/aos/1176344136")}

Sclove, S. L. (1987). Application of model-selection criteria to some problems in multivariate analysis. Psychometrika, 52, 333–343.

Yen, W. M. (1984). Effects of local item dependence on the fit and equating performance of the three-parameter logistic model. Applied Psychological Measurement, 8, 125–145. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1177/014662168400800201")}

See Also

get.fit.index for model-specific wrappers, summary.good.of.fit for formatted fit tables, print.good.of.fit for the print method.

ForceChoice documentation built on Sept. 13, 2026, 1:06 a.m.