R/simulate_bias.R

Defines functions print.GPvam_simulation .gpvam_simulation_residual_factor .gpvam_simulation_boundary .gpvam_simulation_validate_data .gpvam_simulation_weights simulate_bias

Documented in print.GPvam_simulation simulate_bias

#' Response-refit simulations for a fitted GPvam model
#'
#' Quantify fixed-effect estimation error under a specified conditional generating
#' model, re-estimating covariance and persistence parameters for every response.
#'
#' @param model A converged GPvam fit created with version 3.4-0 or later.
#' @param data The original input data frame, including missing-score rows.
#'   Student, teacher and year keys, row names, observed-score pattern, fitted
#'   responses and fixed-effect design must match the supplied fit.
#' @param nsim Number of independent simulations, an integer at least 2.
#' @param beta Generating fixed-effect vector, defaulting to the fitted vector.
#'   Named vectors must have exactly the fitted names and order.
#' @param k_list Named list of contrast vectors in the fitted coefficient order.
#'   A single numeric vector is also accepted. NULL examines every coefficient.
#' @param generator Optional function taking model and replicate, returning
#'   finite numeric vectors random_effects and residual_errors. Their orders are
#'   those of model$eta.hat and model$observation, respectively. The default
#'   holds fitted random effects fixed and draws centered Gaussian residuals
#'   with the original fitted residual covariance on observed scores.
#' @param centered_errors Logical declaration that generating residuals have
#'   mean zero, default TRUE for the default generator and FALSE otherwise.
#' @param refit_control Named list with optional max.iter.EM and tol1 overrides.
#'   Defaults are the original fit's resolved values. Model structure, formula,
#'   and ML/REML choice cannot be changed. Hessians are not recomputed.
#' @param verbose Print one progress line per replicate.
#'
#' @details
#' All R-side persistence types (GP, rGP, VP, CP, ZP) and the supported G-side GP
#' model are handled. The original design, score-missingness pattern, and known
#' teacher assignments on missing-score rows are fixed. Missing scores are not
#' imputed; their known assignments can still contribute to later scores. For
#' R-side models the original within-student residual correlations are retained.
#' G-side residuals have the fitted year-specific variances; the default also
#' holds the fitted student random effects fixed, along with teacher effects.
#' These fixed predictions are shrunken; refitted covariance estimates can differ
#' systematically from the original fit as part of this conditional target.
#' R-side defaults redraw within-student dependence through residuals, whereas
#' G-side defaults hold student effects fixed, so their targets differ.
#'
#' Responses are generated as y_s = X beta + h_s + e_s, where h_s = Z_0 eta_s
#' uses the ORIGINAL fitted random-effect design. Every refit uses the same
#' formula, persistence type, student side and ML/REML setting. For VP, the
#' persistence multipliers and hence the refitted design Z_s may change.
#' With fitted GLS map W_s, Alignment is k'W_s h_s, Residual is k'W_s e_s,
#' and Total is their sum. Estimation_Error records k'(beta_hat_s - beta), and
#' the pointwise decomposition is checked numerically. Internal_Diagnostic is
#' k'W_s Z_s eta_hat_s, using the REFITTED design and effects; it is not the
#' generated latent contribution. Alignment_Teacher and Alignment_Student split
#' Alignment for G-side fits. The student term is zero for R-side fits.
#'
#' If centered_errors is TRUE, Residual_CV is k'(W_s - W_0)e_s, where W_0 uses
#' the original fitted covariance. Its expectation equals that of Residual
#' because W_0 is fixed and the declared residual mean is zero. Total_CV is
#' Alignment + Residual_CV. These paired control variates often reduce Monte
#' Carlo uncertainty, but variance reduction is not guaranteed. Raw contributions
#' are always retained. MCSE is the empirical standard deviation of the actual
#' replicate values divided by the square root of the successful replicate count.
#'
#' Simulations are sequential and use R's RNG; use set.seed(). Custom generators
#' must produce independent replicates with finite variances for this MCSE
#' interpretation. A generator error or invalid return aborts the entire call;
#' completed replicates are not returned after this contract violation. Refit
#' error and nonconvergence rows remain explicit with missing
#' contributions. After failures, raw summaries describe successful refits only;
#' ALL control-variate summary means and MCSEs are withheld, because selection
#' of successful refits can invalidate their equal-expectation argument. An
#' Ill_Conditioned flag records a covariance block with eigenvalue ratio below
#' 1e-8 or a nonpositive eigenvalue; those converged refits are retained. This flag does not detect every
#' variance boundary (in particular, a tiny scalar variance has ratio one).
#' GLS and decomposition checks use a tolerance of 1e-6 times the larger of
#' one and the absolute values being compared; discrepancies are retained in
#' fit_status, and values beyond tolerance make the replicate a failure.
#'
#' This is a conditional sensitivity experiment under chosen generating values,
#' not an estimate of unconditional population bias or a calibrated bias test.
#' It does not generate an endogenous classroom-assignment or stochastic-covariate
#' mechanism. No automatic coefficient adjustment is supplied. A custom generator
#' changes latent/error draws at a fixed design; stochastic designs require an
#' external joint generating and refitting workflow.
#'
#' The original data are required explicitly and are never recovered by evaluating
#' the stored call. Older fitted objects must be refitted once with the current
#' GPvam to obtain validated simulation metadata. Formula dependence on y is
#' rejected, as are formula variables absent from data. Use the same collation
#' locale as for the original fit; differing locales are rejected before refits.
#' Output contains coefficient-level summaries and statuses, not input
#' rows, generated response vectors, fitted objects, or original identifiers.
#'
#' @return A list of class GPvam_simulation with summary, replicates, fit_status,
#'   counts, beta, original_diagnostic, centered_errors, generator_type, and
#'   model_spec. Use qualified function names
#'   GPvam::simulate_bias and mixedbiastest::simulate_bias when both are attached.
#' @seealso \code{\link{GPvam}}, \code{\link{bias.test.custom}}
#' @examples
#' \donttest{
#' data(vam_data)
#' fit <- GPvam(vam_data, ~as.factor(year) + cont_var + 0,
#'              persistence = "CP", verbose = FALSE)
#' set.seed(20261022)
#' # Two draws illustrate the API; use more to obtain useful Monte Carlo precision.
#' sim <- simulate_bias(fit, vam_data, nsim = 2)
#' sim$summary
#' }
#' @export
simulate_bias <- function(model, data, nsim = 100, beta = NULL,
                          k_list = NULL, generator = NULL,
                          centered_errors = is.null(generator),
                          refit_control = list(), verbose = FALSE) {
  valid_vector <- function(x, size) {
    is.numeric(x) && !is.complex(x) && is.null(dim(x)) &&
      length(x) == size && all(is.finite(x))
  }
  if (!inherits(model, "GPvam") || !isTRUE(model$converged))
    stop("model must be a converged GPvam fit.", call. = FALSE)
  if (is.null(model$simulation_spec))
    stop("Refit this model with GPvam 3.4-0 or later to obtain verified simulation metadata.", call. = FALSE)
  if (!model$persistence %in% c("GP", "rGP", "VP", "CP", "ZP") ||
      !model$student.side %in% c("R", "G") ||
      (model$student.side == "G" && model$persistence != "GP"))
    stop("Unsupported persistence/student-side combination.", call. = FALSE)
  if (!valid_vector(nsim, 1L) || nsim < 2 || nsim != floor(nsim) ||
      nsim > .Machine$integer.max) stop("nsim must be an integer at least 2.", call. = FALSE)
  nsim <- as.integer(nsim)
  if (!is.null(generator) && !is.function(generator)) stop("generator must be NULL or a function.")
  for (nm in c("centered_errors", "verbose")) {
    z <- get(nm)
    if (!is.logical(z) || length(z) != 1L || is.na(z)) stop(nm, " must be TRUE or FALSE.")
  }
  .gpvam_simulation_validate_data(model, data)
  if (!is.list(refit_control) || (length(refit_control) &&
      (is.null(names(refit_control)) || anyDuplicated(names(refit_control)) ||
       any(!names(refit_control) %in% c("max.iter.EM", "tol1")))))
    stop("refit_control may contain only named max.iter.EM and tol1 values.")
  control <- model$simulation_spec[c("max.iter.EM", "tol1")]
  control[names(refit_control)] <- refit_control
  if (!valid_vector(control$max.iter.EM, 1L) || control$max.iter.EM < 1 ||
      control$max.iter.EM != floor(control$max.iter.EM) || control$max.iter.EM > .Machine$integer.max)
    stop("max.iter.EM must be a positive integer.")
  if (!valid_vector(control$tol1, 1L) || control$tol1 <= 0) stop("tol1 must be positive and finite.")
  p <- ncol(model$X); n <- nrow(model$X); q <- ncol(model$Z)
  fixed_names <- colnames(model$X)
  if (is.null(beta)) beta <- stats::coef(model)
  if (!valid_vector(beta, p) || (!is.null(names(beta)) && !identical(names(beta), fixed_names)))
    stop("beta must follow the fitted coefficient names and order.")
  if (is.null(k_list)) {
    k_list <- lapply(seq_len(p), function(i) { z <- numeric(p); z[i] <- 1; z })
    names(k_list) <- fixed_names
  } else if (is.numeric(k_list)) k_list <- list(Contrast = k_list)
  if (!is.list(k_list) || !length(k_list) || !all(vapply(k_list, valid_vector, logical(1), size = p)))
    stop("k_list must contain finite contrast vectors in the fitted coefficient order.")
  for (k in k_list) if (!is.null(names(k)) && !identical(names(k), fixed_names))
    stop("Named contrasts must follow the fitted coefficient names and order.")
  if (is.null(names(k_list))) names(k_list) <- paste0("Contrast", seq_along(k_list))
  if (anyNA(names(k_list)) || any(!nzchar(names(k_list))) || anyDuplicated(names(k_list)))
    stop("Contrast labels must be nonempty and unique.")
  K <- do.call(rbind, k_list); labels <- names(k_list)
  W0 <- K %*% .gpvam_simulation_weights(model)
  # pivot=FALSE is essential: U'U = R.inv in the original observation order.
  U <- .gpvam_simulation_residual_factor(model)
  fixed_mean <- as.numeric(model$X %*% beta)
  student_columns <- if (model$student.side == "G") seq_len(model$num.student) else integer()
  teacher_columns <- setdiff(seq_len(q), student_columns)
  original_diagnostic <- data.frame(Contrast = labels,
    Internal_Diagnostic = as.numeric(W0 %*% (model$Z %*% model$eta.hat)))
  components <- c("Alignment", "Residual", "Total", "Estimation_Error", "Internal_Diagnostic",
                  "Alignment_Teacher", "Alignment_Student", "Residual_CV", "Total_CV")
  rows <- statuses <- vector("list", nsim)
  for (i in seq_len(nsim)) {
    draws <- if (is.null(generator)) list(random_effects = as.numeric(model$eta.hat),
      residual_errors = as.numeric(solve(U, stats::rnorm(n)))) else generator(model, i)
    if (!is.list(draws) || !valid_vector(draws$random_effects, q) ||
        !valid_vector(draws$residual_errors, n))
      stop("Generator replicate ", i, " must supply finite random_effects and residual_errors in fitted order.")
    latent_teacher <- as.numeric(model$Z[, teacher_columns, drop = FALSE] %*% draws$random_effects[teacher_columns])
    latent_student <- if (length(student_columns)) as.numeric(model$Z[, student_columns, drop = FALSE] %*%
      draws$random_effects[student_columns]) else numeric(n)
    latent <- latent_teacher + latent_student
    simulated <- data
    simulated$y[model$observation] <- fixed_mean + latent + draws$residual_errors
    row <- data.frame(Replicate = i, Contrast = labels, stringsAsFactors = FALSE)
    for (component in components) row[[component]] <- NA_real_
    messages <- character(); started <- proc.time()[3]
    fit <- tryCatch(withCallingHandlers(GPvam(simulated, fixed_effects = model$fixed_effects,
      student.side = model$student.side, persistence = model$persistence,
      max.iter.EM = control$max.iter.EM, tol1 = control$tol1,
      hessian = FALSE, REML = model$REML, verbose = FALSE),
      warning = function(w) { messages <<- c(messages, conditionMessage(w)); invokeRestart("muffleWarning") },
      message = function(m) { messages <<- c(messages, conditionMessage(m)); invokeRestart("muffleMessage") }),
      error = function(e) e)
    status <- "ok"; ill_conditioned <- NA; identity_error <- NA_real_; gls_error <- NA_real_; iterations <- NA_integer_
    if (inherits(fit, "error")) { status <- "error"; messages <- c(messages, conditionMessage(fit))
    } else {
      iterations <- fit$iter
      if (!isTRUE(fit$converged)) status <- "nonconverged" else {
        calculated <- tryCatch({
          if (!identical(fit$observation, model$observation) ||
              !identical(colnames(fit$X), fixed_names) ||
              !isTRUE(all.equal(as.matrix(fit$X), as.matrix(model$X), tolerance = 1e-12)) ||
              !identical(fit$teach.effects[c("teacher_year", "teacher", "effect_year")],
                         model$teach.effects[c("teacher_year", "teacher", "effect_year")]) ||
              !identical(dim(fit$Z), dim(model$Z)))
            stop("The refit changed the observed-row, fixed-effect or teacher-effect mapping.")
          if (model$persistence != "VP" && max(abs(fit$Z - model$Z)) > 1e-12)
            stop("The refit changed a fixed random-effect design.")
          weights <- .gpvam_simulation_weights(fit)
          gls_beta <- as.numeric(weights %*% fit$y)
          gls_error <- max(abs(gls_beta - stats::coef(fit)))
          if (!is.finite(gls_error) || gls_error > 1e-6 * max(1, abs(gls_beta), abs(stats::coef(fit))))
            stop("Returned coefficients do not match GLS at the final returned covariance.")
          W <- K %*% weights
          row$Alignment_Teacher <- as.numeric(W %*% latent_teacher)
          row$Alignment_Student <- as.numeric(W %*% latent_student)
          row$Alignment <- row$Alignment_Teacher + row$Alignment_Student
          row$Residual <- as.numeric(W %*% draws$residual_errors)
          row$Total <- row$Alignment + row$Residual
          row$Estimation_Error <- as.numeric(K %*% (stats::coef(fit) - beta))
          row$Internal_Diagnostic <- as.numeric(W %*% (fit$Z %*% fit$eta.hat))
          if (centered_errors) {
            row$Residual_CV <- as.numeric((W - W0) %*% draws$residual_errors)
            row$Total_CV <- row$Alignment + row$Residual_CV
          }
          active_components <- if (centered_errors) components else setdiff(components, c("Residual_CV", "Total_CV"))
          if (any(!is.finite(as.matrix(row[, active_components, drop = FALSE]))))
            stop("Nonfinite simulated component.")
          identity_error <- max(abs(row$Total - row$Estimation_Error))
          if (!is.finite(identity_error) || identity_error > 1e-6 * max(1, abs(row$Total), abs(row$Estimation_Error)))
            stop("The per-replicate GLS decomposition failed its numerical identity check.")
          ill_conditioned <- .gpvam_simulation_boundary(fit)
          row
        }, error = function(e) e)
        if (inherits(calculated, "error")) {
          status <- "error"; messages <- c(messages, conditionMessage(calculated))
          for (component in components) row[[component]] <- NA_real_
        } else row <- calculated
      }
    }
    row$Status <- status; rows[[i]] <- row
    statuses[[i]] <- data.frame(Replicate = i, Status = status, Ill_Conditioned = ill_conditioned,
      Iterations = iterations, Seconds = unname(proc.time()[3] - started),
      Decomposition_Error = identity_error, GLS_Coefficient_Error = gls_error,
      Message = paste(unique(trimws(messages)), collapse = " | "), stringsAsFactors = FALSE)
    if (verbose) { cat("Simulation", i, "of", nsim, ":", status, "\n"); flush.console() }
  }
  replicates <- do.call(rbind, rows); fit_status <- do.call(rbind, statuses)
  success <- sum(fit_status$Status == "ok")
  summary <- do.call(rbind, lapply(labels, function(label) {
    d <- replicates[replicates$Contrast == label & replicates$Status == "ok", ]
    do.call(rbind, lapply(components, function(component) {
      values <- d[[component]]
      available <- length(values) > 0 && all(is.finite(values)) &&
        !(component %in% c("Residual_CV", "Total_CV") && success < nsim)
      data.frame(Contrast = label, Component = component,
        Mean = if (available) mean(values) else NA_real_,
        MCSE = if (available && length(values) > 1) stats::sd(values)/sqrt(length(values)) else NA_real_,
        N_Success = success, N_Failed = nsim - success, stringsAsFactors = FALSE)
    }))
  }))
  counts <- data.frame(Requested = nsim, Success = success, Failed = nsim-success,
    Ill_Conditioned = sum(fit_status$Ill_Conditioned, na.rm = TRUE),
    With_Message = sum(nzchar(fit_status$Message)), CV_Summary_Available = centered_errors && success == nsim)
  if (success < nsim) warning(nsim-success, " of ", nsim,
    " refits failed or did not converge; raw summaries describe successful refits only.",
    if (centered_errors) " CV summaries withheld." else "", call. = FALSE)
  structure(list(summary = summary, replicates = replicates, fit_status = fit_status,
    counts = counts, beta = beta, original_diagnostic = original_diagnostic,
    centered_errors = centered_errors,
    generator_type = if (is.null(generator)) "fixed predicted random effects; fitted Gaussian residuals" else "user-specified fixed-design scenario",
    model_spec = list(persistence = model$persistence, student.side = model$student.side,
      REML = model$REML, n_observations = n, n_random_effects = q,
      n_student_effects = length(student_columns), refit_control = control)), class = "GPvam_simulation")
}

.gpvam_simulation_weights <- function(model) {
  XtV <- crossprod(model$X, model$vinv)
  as.matrix(solve(forceSymmetric(XtV %*% model$X), XtV))
}

.gpvam_simulation_validate_data <- function(model, data) {
  spec <- model$simulation_spec
  if (is.null(spec$collation) || !identical(spec$collation, Sys.getlocale("LC_COLLATE")))
    stop("Use the same collation locale as the original fit, or refit with this GPvam version first.", call. = FALSE)
  if (!is.data.frame(data) || anyDuplicated(names(data)) ||
      !all(c("y", "student", "teacher", "year") %in% names(data)))
    stop("data must be the original GPvam input data frame.")
  if (!identical(data[, c("student", "teacher", "year"), drop = FALSE], spec$data_keys) ||
      !identical(rownames(data), spec$row_names) || !identical(!is.na(data$y), spec$observed))
    stop("Input student/year/teacher keys, row names or missing-score pattern differ from the fitted data.")
  if (!is.numeric(data$y) || any(!is.finite(data$y[spec$observed])) ||
      !isTRUE(all.equal(as.numeric(data$y[model$observation]), as.numeric(model$y), tolerance = 0)))
    stop("Observed responses differ from the fitted data.")
  if ("y" %in% all.vars(model$fixed_effects)) stop("Response-dependent fixed-effect formulas using y are not supported.")
  if (length(setdiff(all.vars(model$fixed_effects), names(data))))
    stop("All fixed-effect formula variables must be columns in data; external vectors are not supported.")
  d <- as.data.frame(data)
  d$year <- match(d$year, sort(unique(d$year)))
  d$teacher <- as.character(d$teacher)
  known <- !is.na(d$teacher)
  d$teacher[known] <- paste0(d$teacher[known], "(year", d$year[known], ")")
  d <- d[spec$observed, , drop = FALSE]
  # Match each engine's formula-evaluation order, including order-dependent
  # transforms. GP/rGP sort by year/teacher before forming X; G-side first
  # sorts students, which determines tied year/teacher order.
  if (model$student.side == "G") d <- d[order(d$student), , drop = FALSE]
  if (model$persistence %in% c("GP", "rGP")) {
    d <- d[order(d$year, d$teacher), , drop = FALSE]
  } else d <- d[order(d$student, d$year, d$teacher), , drop = FALSE]
  X <- sparse.model.matrix(model$fixed_effects, data = d, drop.unused.levels = TRUE)
  X <- X[, !(colSums(abs(X)) == 0), drop = FALSE]
  X <- as.matrix(X[match(rownames(model$X), rownames(X)), , drop = FALSE])
  if (!identical(colnames(X), colnames(model$X)) ||
      !identical(rownames(X), rownames(model$X)) ||
      !isTRUE(all.equal(X, as.matrix(model$X), tolerance = 0, check.attributes = FALSE)))
    stop("The fixed-effect design differs from the fitted data.")
  invisible(TRUE)
}

.gpvam_simulation_boundary <- function(model) {
  blocks <- model$teach.cov
  if (model$student.side == "R") blocks <- c(blocks, list(model$R_i)) else
    blocks <- c(blocks, list(model$stu.cov))
  any(vapply(blocks, function(b) {
    v <- eigen(as.matrix(b), symmetric = TRUE, only.values = TRUE)$values
    min(v) <= 0 || min(v) < 1e-8 * max(v)
  }, logical(1)))
}

.gpvam_simulation_residual_factor <- function(model) {
  chol(forceSymmetric(model$R.inv), pivot = FALSE)
}

#' Print a GPvam response-refit simulation
#'
#' @param x An object returned by \code{\link{simulate_bias}}.
#' @param ... Additional arguments passed to \code{print}.
#' @return The object, invisibly.
#' @method print GPvam_simulation
#' @export
print.GPvam_simulation <- function(x, ...) {
  cat("GPvam conditional response-refit simulation\n")
  cat(x$model_spec$persistence, "persistence;", x$model_spec$student.side,
      "student side;", if (x$model_spec$REML) "REML" else "ML", "\n")
  print(x$counts, row.names = FALSE, ...)
  cat("Component means and Monte Carlo SEs: $summary; individual draws: $replicates.\n")
  cat("This conditional experiment does not identify unconditional population bias.\n")
  invisible(x)
}

Try the GPvam package in your browser

Any scripts or data that you put into this service are public.

GPvam documentation built on Oct. 23, 2026, 1:06 a.m.