fit_ppmc: Posterior predictive model checks for assessing model fit

In wjakethompson/measr: Bayesian Psychometric Measurement Using 'Stan'

Posterior predictive model checks for assessing model fit

Description

For models estimated with method = "mcmc", use the posterior distributions to compute expected distributions for fit statistics and compare to values in the observed data.

Usage

fit_ppmc(
  model,
  ndraws = NULL,
  probs = c(0.025, 0.975),
  return_draws = 0,
  model_fit = c("raw_score"),
  item_fit = c("conditional_prob", "odds_ratio", "pvalue"),
  force = FALSE
)

Arguments

model	A measrfit object.
ndraws	The number of posterior draws to base the checks on. Must be less than or equal to the total number of posterior draws retained in the estimated model. If NULL (the default) the total number from the estimated model is used.
probs	The percentiles to be computed by the stats::quantile() function for summarizing the posterior distributions of the specified fit statistics.
return_draws	Proportion of posterior draws for each specified fit statistic to be returned. This does not affect the calculation of the posterior predictive checks, but can be useful for visualizing the fit statistics. For example, if ndraws = 500, return_draws = 0.2, and model_fit = "raw_score", then the raw score chi-square will be computed 500 times (once for each draw) and 100 of those values (0.2 * 500) will be returned. If 0 (the default), only summaries of the posterior are returned (no individual samples).
model_fit	The posterior predictive model checks to compute for an evaluation of model-level fit. If NULL, no model-level checks are computed. See details.
item_fit	The posterior predictive model checks to compute for an evaluation of item-level fit. If NULL, no item-level checks are computed. Multiple checks can be provided in order to calculate more than one check simultaneously (e.g., item_fit = c("conditional_prob", "odds_ratio")). See details.
force	If all requested PPMCs have already been added to the model object using add_fit(), should they be recalculated. Default is FALSE.

Details

Posterior predictive model checks (PPMCs) use the posterior distribution of an estimated model to compute different statistics. This creates an expected distribution of the given statistic, if our estimated parameters are correct. We then compute the statistic in our observed data and compare the observed value to the expected distribution. Observed values that fall outside of the expected distributions indicate incompatibility between the estimated model and the observed data.

We currently support PPMCs at the model and item level. At the model level, we calculate the expected raw score distribution (model_fit = "raw_score"), as described by Thompson (2019) and Park et al. (2015).

At the item level, we can calculate the conditional probability that a respondent in each class provides a correct response (item_fit = "conditional_prob") as described by Thompson (2019) and Sinharay & Almond (2007) or the overall proportion correct for an item (item_fit = "pvalue"), as described by Thompson (2019). We can also calculate the odds ratio for each pair of items (item_fit = "odds_ratio") as described by Park et al. (2015) and Sinharay et al. (2006).

Value

A list with two elements, "model_fit" and "item_fit". If either model_fit = NULL or item_fit = NULL in the function call, this will be a one-element list, with the null criteria excluded. Each list element, is itself a list with one element for each specified PPMC containing a tibble. For example if item_fit = c("conditional_prob", "odds_ratio"), the "item_fit" element will be a list of length two, where each element is a tibble containing the results of the PPMC. All tibbles follow the same general structure:

• ⁠obs_{ppmc}⁠: The value of the relevant statistic in the observed data.

• ppmc_mean: The mean of the ndraws posterior samples calculated for the given statistic.

• Quantile columns: 1 column for each value of probs, providing the corresponding quantiles of the ndraws posterior samples calculated for the given statistic.

• samples: A list column, where each element contains a vector of length (ndraws * return_draws), representing samples from the posterior distribution of the calculated statistic. This column is excluded if return_draws = 0.

• ppp: The posterior predictive p-value. This is the proportion of posterior samples for calculated statistic that are greater than the observed value. Values very close to 0 or 1 indicate incompatibility between the fitted model and the observed data.

References

Park, J. Y., Johnson, M. S., Lee, Y-S. (2015). Posterior predictive model checks for cognitive diagnostic models. International Journal of Quantitative Research in Education, 2(3-4), 244-264. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1504/IJQRE.2015.071738")}

Sinharay, S., & Almond, R. G. (2007). Assessing fit of cognitive diagnostic models. Educational and Psychological Measurement, 67(2), 239-257. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1177/0013164406292025")}

Sinharay, S., Johnson, M. S., & Stern, H. S. (2006). Posterior predictive assessment of item response theory models. Applied Psychological Measurement, 30(4), 298-321. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.1177/0146621605285517")}

Thompson, W. J. (2019). Bayesian psychometrics for diagnostic assessments: A proof of concept (Research Report No. 19-01). University of Kansas; Accessible Teaching, Learning, and Assessment Systems. \Sexpr[results=rd]{tools:::Rd_expr_doi("10.35542/osf.io/jzqs8")}

Examples

mdm_dina <- measr_dcm(
  data = mdm_data, missing = NA, qmatrix = mdm_qmatrix,
  resp_id = "respondent", item_id = "item", type = "dina",
  method = "mcmc", seed = 63277, backend = "rstan",
  iter = 700, warmup = 500, chains = 2, refresh = 0
)

fit_ppmc(mdm_dina, model_fit = "raw_score", item_fit = NULL)

wjakethompson/measr documentation built on April 12, 2025, 9:46 p.m.