R/GPvam.R

Defines functions GPvam

Documented in GPvam

#' Fit a Generalized or Variable Persistence Model
#'
#' Fit a multiple-membership linear mixed model by an EM algorithm. Teacher
#' effects may persist into future years, and dependence among a student's
#' observations is represented by a residual covariance matrix or a random
#' student effect. See Karl, Yang and Lohr (2013) and Mariano et al. (2010).
#'
#' @param vam_data A data frame containing \code{y} (numeric student score),
#'   \code{student} (student identifier), \code{teacher} (teacher identifier),
#'   and \code{year} (numeric year or other ordered time period), together with
#'   any fixed-effect covariates. There must be at most one row per student
#'   and year. Student identifiers and years must not be missing. Scores and
#'   teacher identifiers may be \code{NA}; fixed-effect covariates must be
#'   complete on rows with observed scores. Unused columns are ignored.
#'   See \sQuote{Missing data} below.
#' @param fixed_effects A one-sided formula specifying fixed effects. The
#'   default estimates a separate mean for each year, or an intercept if there
#'   is only one year. Use factors for
#'   categorical covariates; for example,
#'   \code{~ as.factor(year) + cont_var + 0} also includes a continuous covariate.
#'   Offsets and the reserved variable names \code{mis}, \code{pat}, and
#'   \code{r} are not supported in this formula.
#' @param student.side Character string, either \code{"R"} for an unstructured
#'   within-student residual covariance or \code{"G"} for a random student
#'   intercept and year-specific residual variances. A G-side model requires
#'   repeated observed scores for at least one student.
#' @param persistence Character string selecting \code{"GP"}, \code{"rGP"},
#'   \code{"VP"}, \code{"CP"}, or \code{"ZP"}. Only \code{"GP"} is supported
#'   with \code{student.side = "G"}. See \sQuote{Persistence models} below.
#' @param max.iter.EM Positive integer giving the maximum number of EM
#'   iterations. A fit that reaches this limit without convergence is returned
#'   with a warning and \code{converged = FALSE}.
#' @param tol1 Positive finite convergence tolerance for the relative change
#'   in log likelihood. The default is \code{1e-7}.
#' @param hessian Logical; calculate the observed information for covariance
#'   parameters and, for \code{persistence = "VP"}, persistence parameters.
#'   This supplies their standard errors. Fixed-effect standard errors and
#'   teacher prediction-error standard deviations are computed regardless.
#'   Covariance/persistence standard errors are \code{NA} when the information
#'   matrix cannot be computed or is not positive definite.
#' @param hes.method Numerical differentiation method for the information
#'   matrix: \code{"simple"} or \code{"richardson"}.
#' @param REML Logical; use restricted maximum likelihood (the default) rather
#'   than maximum likelihood. Supported for every available persistence/student-side
#'   combination.
#' @param verbose Logical; print model information and iteration progress.
#'
#' @details
#' The model is fitted by an EM algorithm based on Karl, Yang and Lohr
#' (2013). Covariance updates use conditional random-effect moments for ML
#' and prediction-error covariance adjustments for REML; see Karl (2026).
#' With incomplete observation patterns, the unstructured residual covariance
#' update uses a Cholesky parameterization to retain positive definiteness.
#' Setting \code{hessian = TRUE} adds numerical differentiation after fitting
#' and can substantially increase runtime. Historical timings are available
#' in \code{\link{GPvam.benchmark}}.
#'
#' Fixed-effect standard errors and teacher prediction-error standard
#' deviations use the inverse mixed-model coefficient matrix with estimated
#' covariance parameters treated as known. They do not include an additional
#' adjustment for uncertainty in those covariance estimates.
#'
#' @section Persistence models:
#' The generalized persistence model (\code{"GP"}) assigns each teacher
#' correlated current-year and future-year effects. The reduced model
#' (\code{"rGP"}) combines a teacher's future-year effects into one shared
#' effect. The variable persistence model (\code{"VP"}) makes a teacher's
#' future-year effects multiples of the current-year effect and estimates
#' those multipliers. Complete (\code{"CP"}) and zero (\code{"ZP"}) persistence
#' fix the multipliers at one and zero, respectively.
#'
#' With \code{student.side = "R"}, an unstructured covariance matrix models
#' within-student dependence. With \code{student.side = "G"}, a random student
#' intercept represents that dependence and the remaining errors are
#' independent, with a separate variance for each year. The latter
#' formulation assumes that scores across years use a common scale and can
#' have substantial memory requirements for large numbers of random effects.
#'
#' @section Missing data:
#' A missing score is omitted from the response used for fitting. A known
#' teacher assignment on that row is retained in the student's teaching
#' history and can contribute persistent effects to later observed scores.
#' A missing teacher identifier represents an unknown assignment, not a
#' teacher named \code{"NA"}; no teacher effect is attached to that assignment.
#'
#' @note
#' Each teacher-year is a separate unit. If the same teacher identifier
#' appears in different years, the corresponding effects are modeled
#' independently. Internally, a year suffix is added to each known teacher
#' identifier, and the original year labels are retained in the output.
#' Before evaluating \code{fixed_effects}, years are recoded to consecutive
#' numeric indices in increasing order. Thus \code{~ year} specifies a linear
#' trend across those indices. To use the original numerical spacing, copy
#' the original years into a separate covariate and use that covariate in
#' the formula.
#'
#' The fixed-effects design must have full column rank. Use
#' \code{~ as.factor(year) * cont_var + 0} to include year-specific slopes;
#' see \code{\link[stats]{formula}} for formula syntax.
#'
#' @return An object of class \code{GPvam}, a list that includes:
#' \item{loglik}{Log likelihood (restricted log likelihood for REML) at the
#'   returned fit. Meeting the relative-change convergence criterion does not
#'   guarantee a global maximum.}
#' \item{converged}{Whether the convergence criterion was met.}
#' \item{iter, loglik_history}{Number of evaluated iterations and the
#'   corresponding sequence of log likelihoods.}
#' \item{teach.effects}{Data frame containing teacher identifiers, originating
#'   and effect years, predicted effects (\code{EBLUP}), and prediction-error
#'   standard deviations (\code{std_error}).}
#' \item{parameters}{Matrix of fixed-effect, covariance, and applicable
#'   persistence estimates with their standard errors. Covariance and
#'   persistence standard errors are \code{NA} unless \code{hessian = TRUE}.}
#' \item{Hessian}{Observed information matrix for covariance and persistence
#'   parameters when requested, otherwise \code{NA}.}
#' \item{R_i}{Within-student residual covariance for R-side models;
#'   \code{NA} for G-side models.}
#' \item{teach.cov}{List of teacher covariance blocks, one per originating year.}
#' \item{stu.cov}{Student random-effect variance for a G-side model;
#'   \code{NA} for an R-side model.}
#' \item{mresid, cresid, sresid}{Marginal residuals, conditional residuals, and
#'   conditional residuals scaled using the residual precision matrix.}
#' \item{y, yhat, yhat.m, yhat.s}{Observed responses, conditional fitted values,
#'   marginal fitted values, and scaled conditional fitted values, in the
#'   internal fitting order.}
#' \item{X, Z, G, R.inv, vinv}{Fixed- and random-effects design matrices,
#'   random-effects covariance, residual precision, and marginal precision.}
#' \item{ybetas, eta.hat}{Unrounded fixed-effect estimates and random-effect
#'   predictions.}
#' \item{num.obs, num.student, num.year, num.teach}{Observation, student, year,
#'   and per-year teacher counts.}
#' \item{persistence, student.side, REML}{The fitted model and estimation options.}
#' \item{key}{Mapping between original year values and internal year indices.}
#' \item{observation}{Input-row indices in the internal fitting order.}
#' \item{simulation_spec}{Original student/year/teacher row keys, observed-score
#'   pattern, collation locale, and resolved numerical controls used by \code{\link{simulate_bias}}.
#'   These keys are private input metadata, not an aggregate summary.}
#' \item{call, fixed_effects}{Original model call and fixed-effects formula.}
#'
#' Use the standard \code{\link{GPvam-methods}} to extract full-precision
#' fixed-effect estimates and covariances, fitted values, residuals, likelihood,
#' and observation count. The fitted-value and residual methods return
#' observations in input order with missing responses omitted.
#'
#' @inherit GPvam-package references
#' @author Andrew Karl \email{akarl@asu.edu}, Yan Yang and Sharon Lohr.
#' @seealso \code{\link{plot.GPvam}}, \code{\link{summary.GPvam}},
#'   \code{\link{bias.test.custom}}, \code{\link{vam_data}}
#' @examples
#' data(vam_data)
#' \donttest{
#' result <- GPvam(vam_data, persistence = "CP",
#'                 fixed_effects = ~ as.factor(year) + cont_var + 0,
#'                 verbose = FALSE)
#' summary(result)
#' plot(result, ask = FALSE)
#' }
#' @keywords regression
#' @export
GPvam <- function(vam_data, fixed_effects = formula(~as.factor(year) + 0),
                  student.side = "R", persistence = "GP", max.iter.EM = 1000,
                  tol1 = 1e-07, hessian = FALSE, hes.method = "simple",
                  REML = TRUE, verbose = TRUE) {
    cl <- match.call()
    default_fixed <- missing(fixed_effects)
    if (!is.data.frame(vam_data) || nrow(vam_data) == 0L)
        stop("vam_data must be a nonempty data frame.", call. = FALSE)
    if (anyDuplicated(names(vam_data)))
        stop("vam_data must have unique column names.", call. = FALSE)
    required <- c("y", "student", "teacher", "year")
    absent <- setdiff(required, names(vam_data))
    if (length(absent))
        stop("vam_data is missing required columns: ", paste(absent, collapse = ", "), call. = FALSE)
    if (!is.numeric(vam_data$y) || any(!is.finite(vam_data$y) & !is.na(vam_data$y)))
        stop("y must be numeric, with finite observed values (NA is allowed).", call. = FALSE)
    if (!is.numeric(vam_data$year) || any(!is.finite(vam_data$year)))
        stop("year must contain finite numeric values with no missing values.", call. = FALSE)
    valid_id <- function(x) is.factor(x) || is.character(x) || is.numeric(x)
    if (!valid_id(vam_data$student) || anyNA(vam_data$student) ||
        (is.numeric(vam_data$student) && any(!is.finite(vam_data$student))))
        stop("student must contain nonmissing finite numeric, character, or factor IDs.", call. = FALSE)
    if (!valid_id(vam_data$teacher) ||
        (is.numeric(vam_data$teacher) && any(!is.finite(vam_data$teacher) & !is.na(vam_data$teacher))))
        stop("teacher must contain numeric, character, or factor IDs (NA is allowed).", call. = FALSE)
    if (anyDuplicated(vam_data[c("student", "year")]))
        stop("Each student must have at most one row per year.", call. = FALSE)
    choice <- function(x, choices, name) {
        if (!is.character(x) || length(x) != 1L || is.na(x) || !x %in% choices)
            stop(name, " must be one of: ", paste(choices, collapse = ", "), ".", call. = FALSE)
        x
    }
    student.side <- choice(student.side, c("R", "G"), "student.side")
    persistence <- choice(persistence, c("GP", "rGP", "VP", "CP", "ZP"), "persistence")
    hes.method <- choice(hes.method, c("simple", "richardson"), "hes.method")
    for (nm in c("hessian", "REML", "verbose")) {
        value <- get(nm)
        if (!is.logical(value) || length(value) != 1L || is.na(value))
            stop(nm, " must be TRUE or FALSE.", call. = FALSE)
    }
    if (!is.numeric(max.iter.EM) || length(max.iter.EM) != 1L ||
        !is.finite(max.iter.EM) || max.iter.EM < 1 || max.iter.EM != floor(max.iter.EM) ||
        max.iter.EM > .Machine$integer.max)
        stop("max.iter.EM must be a positive integer.", call. = FALSE)
    if (!is.numeric(tol1) || length(tol1) != 1L || !is.finite(tol1) || tol1 <= 0)
        stop("tol1 must be a positive finite number.", call. = FALSE)
    if (!inherits(fixed_effects, "formula") || length(fixed_effects) != 2L)
        stop("fixed_effects must be a one-sided formula.", call. = FALSE)
    if (length(attr(stats::terms(fixed_effects), "offset")))
        stop("Offsets in fixed_effects are not supported.", call. = FALSE)
    if (length(intersect(all.vars(fixed_effects), c("mis", "pat", "r"))))
        stop("The names mis, pat, and r are reserved for internal model bookkeeping.", call. = FALSE)
    if (student.side == "G" && persistence != "GP")
        stop('student.side = "G" currently requires persistence = "GP".', call. = FALSE)
    Z_mat <- as.data.frame(vam_data)
    observed <- !is.na(Z_mat$y)
    if (!any(observed)) stop("There are no observed scores.", call. = FALSE)
    if (student.side == "G" && all(table(Z_mat$student[observed]) <= 1L))
        stop("G-side student variance requires repeated observed scores for at least one student.",
             call. = FALSE)
    years <- sort(unique(Z_mat$year))
    if (default_fixed && length(years) == 1L) fixed_effects <- ~1
    year_group <- match(Z_mat$year, years)
    counts <- tabulate(year_group[observed], nbins = length(years))
    if (any(counts < 2L))
        stop("Each year must have at least two observed scores.", call. = FALSE)
    spreads <- vapply(split(Z_mat$y[observed], year_group[observed]), stats::var, numeric(1))
    if (any(!is.finite(spreads) | spreads <= 0))
        stop("Each year must have positive observed score variance.", call. = FALSE)
    retained <- Z_mat$student %in% Z_mat$student[observed]
    known_years <- tabulate(year_group[retained & !is.na(Z_mat$teacher)],
                            nbins = length(years))
    if (any(known_years == 0L))
        stop("Each year must have a known teacher for a student with observed scores.", call. = FALSE)
    # Keep actual NA teacher IDs; do not turn them into a teacher named 'NA'.
    Z_mat$teacher <- as.character(Z_mat$teacher)
    known <- !is.na(Z_mat$teacher)
    Z_mat$teacher[known] <- paste0(Z_mat$teacher[known], "(year", year_group[known], ")")
    Z_mat$year <- year_group
    key <- cbind(years, seq_along(years))
    # Evaluate only rows with observed responses. Irrelevant extra columns and
    # covariates on missing-score rows cannot affect the likelihood.
    mf <- stats::model.frame(fixed_effects, data = Z_mat[observed, , drop = FALSE],
                             na.action = stats::na.fail, drop.unused.levels = TRUE)
    if (nrow(mf) != sum(observed))
        stop("fixed_effects must describe the observed rows of vam_data.", call. = FALSE)
    design <- stats::model.matrix(fixed_effects, data = mf)
    if (!ncol(design) || any(!is.finite(design)))
        stop("The fixed-effects model matrix must have finite entries and at least one column.", call. = FALSE)
    if (qr(design)$rank < ncol(design))
        stop("The fixed-effects design matrix must have full column rank.", call. = FALSE)
    if (nrow(design) <= ncol(design))
        stop("The number of observations must exceed the number of fixed effects.", call. = FALSE)
    control <- list(max.iter.EM = as.integer(max.iter.EM), tol1 = tol1,
                    hessian = hessian, hes.method = hes.method, verbose = verbose,
                    persistence = persistence, REML = REML, key = key)
    if (student.side == "G") {
        res <- GP.csh(Z_mat, fixed_effects, control)
    } else if (persistence == "GP") {
        res <- GP.un(Z_mat, fixed_effects, control)
    } else if (persistence == "rGP") {
        res <- rGP.un(Z_mat, fixed_effects, control)
    } else {
        res <- VP.CP.ZP.un(Z_mat, fixed_effects, control)
    }
    res$key <- key
    res$teach.effects$teacher_year <- years[match(res$teach.effects$teacher_year, seq_along(years))]
    res$teach.effects$effect_year <- years[match(res$teach.effects$effect_year, seq_along(years))]
    res$REML <- REML
    res$student.side <- student.side
    res$persistence <- persistence
    res$call <- cl
    res$fixed_effects <- fixed_effects
    res$observation <- match(rownames(res$X), rownames(vam_data))
    if (length(res$observation) != res$num.obs || anyNA(res$observation))
        stop("Internal observation mapping failed.", call. = FALSE)
    # Retain only input keys and resolved controls needed to validate a refit.
    # Scores and covariates are already represented by y and X; do not duplicate
    # the full input data or evaluate unevaluated expressions from match.call().
    res$simulation_spec <- list(
        data_keys = vam_data[, c("student", "teacher", "year"), drop = FALSE],
        observed = !is.na(vam_data$y), row_names = rownames(vam_data),
        max.iter.EM = as.integer(max.iter.EM), tol1 = tol1,
        collation = Sys.getlocale("LC_COLLATE"))
    class(res) <- "GPvam"
    if (identical(res$converged, FALSE))
        warning("The EM algorithm did not converge within max.iter.EM iterations.", call. = FALSE)
    res
}

Try the GPvam package in your browser

Any scripts or data that you put into this service are public.

GPvam documentation built on Oct. 23, 2026, 1:06 a.m.