Nothing
#' Fit a Generalized or Variable Persistence Model
#'
#' Fit a multiple-membership linear mixed model by an EM algorithm. Teacher
#' effects may persist into future years, and dependence among a student's
#' observations is represented by a residual covariance matrix or a random
#' student effect. See Karl, Yang and Lohr (2013) and Mariano et al. (2010).
#'
#' @param vam_data A data frame containing \code{y} (numeric student score),
#' \code{student} (student identifier), \code{teacher} (teacher identifier),
#' and \code{year} (numeric year or other ordered time period), together with
#' any fixed-effect covariates. There must be at most one row per student
#' and year. Student identifiers and years must not be missing. Scores and
#' teacher identifiers may be \code{NA}; fixed-effect covariates must be
#' complete on rows with observed scores. Unused columns are ignored.
#' See \sQuote{Missing data} below.
#' @param fixed_effects A one-sided formula specifying fixed effects. The
#' default estimates a separate mean for each year, or an intercept if there
#' is only one year. Use factors for
#' categorical covariates; for example,
#' \code{~ as.factor(year) + cont_var + 0} also includes a continuous covariate.
#' Offsets and the reserved variable names \code{mis}, \code{pat}, and
#' \code{r} are not supported in this formula.
#' @param student.side Character string, either \code{"R"} for an unstructured
#' within-student residual covariance or \code{"G"} for a random student
#' intercept and year-specific residual variances. A G-side model requires
#' repeated observed scores for at least one student.
#' @param persistence Character string selecting \code{"GP"}, \code{"rGP"},
#' \code{"VP"}, \code{"CP"}, or \code{"ZP"}. Only \code{"GP"} is supported
#' with \code{student.side = "G"}. See \sQuote{Persistence models} below.
#' @param max.iter.EM Positive integer giving the maximum number of EM
#' iterations. A fit that reaches this limit without convergence is returned
#' with a warning and \code{converged = FALSE}.
#' @param tol1 Positive finite convergence tolerance for the relative change
#' in log likelihood. The default is \code{1e-7}.
#' @param hessian Logical; calculate the observed information for covariance
#' parameters and, for \code{persistence = "VP"}, persistence parameters.
#' This supplies their standard errors. Fixed-effect standard errors and
#' teacher prediction-error standard deviations are computed regardless.
#' Covariance/persistence standard errors are \code{NA} when the information
#' matrix cannot be computed or is not positive definite.
#' @param hes.method Numerical differentiation method for the information
#' matrix: \code{"simple"} or \code{"richardson"}.
#' @param REML Logical; use restricted maximum likelihood (the default) rather
#' than maximum likelihood. Supported for every available persistence/student-side
#' combination.
#' @param verbose Logical; print model information and iteration progress.
#'
#' @details
#' The model is fitted by an EM algorithm based on Karl, Yang and Lohr
#' (2013). Covariance updates use conditional random-effect moments for ML
#' and prediction-error covariance adjustments for REML; see Karl (2026).
#' With incomplete observation patterns, the unstructured residual covariance
#' update uses a Cholesky parameterization to retain positive definiteness.
#' Setting \code{hessian = TRUE} adds numerical differentiation after fitting
#' and can substantially increase runtime. Historical timings are available
#' in \code{\link{GPvam.benchmark}}.
#'
#' Fixed-effect standard errors and teacher prediction-error standard
#' deviations use the inverse mixed-model coefficient matrix with estimated
#' covariance parameters treated as known. They do not include an additional
#' adjustment for uncertainty in those covariance estimates.
#'
#' @section Persistence models:
#' The generalized persistence model (\code{"GP"}) assigns each teacher
#' correlated current-year and future-year effects. The reduced model
#' (\code{"rGP"}) combines a teacher's future-year effects into one shared
#' effect. The variable persistence model (\code{"VP"}) makes a teacher's
#' future-year effects multiples of the current-year effect and estimates
#' those multipliers. Complete (\code{"CP"}) and zero (\code{"ZP"}) persistence
#' fix the multipliers at one and zero, respectively.
#'
#' With \code{student.side = "R"}, an unstructured covariance matrix models
#' within-student dependence. With \code{student.side = "G"}, a random student
#' intercept represents that dependence and the remaining errors are
#' independent, with a separate variance for each year. The latter
#' formulation assumes that scores across years use a common scale and can
#' have substantial memory requirements for large numbers of random effects.
#'
#' @section Missing data:
#' A missing score is omitted from the response used for fitting. A known
#' teacher assignment on that row is retained in the student's teaching
#' history and can contribute persistent effects to later observed scores.
#' A missing teacher identifier represents an unknown assignment, not a
#' teacher named \code{"NA"}; no teacher effect is attached to that assignment.
#'
#' @note
#' Each teacher-year is a separate unit. If the same teacher identifier
#' appears in different years, the corresponding effects are modeled
#' independently. Internally, a year suffix is added to each known teacher
#' identifier, and the original year labels are retained in the output.
#' Before evaluating \code{fixed_effects}, years are recoded to consecutive
#' numeric indices in increasing order. Thus \code{~ year} specifies a linear
#' trend across those indices. To use the original numerical spacing, copy
#' the original years into a separate covariate and use that covariate in
#' the formula.
#'
#' The fixed-effects design must have full column rank. Use
#' \code{~ as.factor(year) * cont_var + 0} to include year-specific slopes;
#' see \code{\link[stats]{formula}} for formula syntax.
#'
#' @return An object of class \code{GPvam}, a list that includes:
#' \item{loglik}{Log likelihood (restricted log likelihood for REML) at the
#' returned fit. Meeting the relative-change convergence criterion does not
#' guarantee a global maximum.}
#' \item{converged}{Whether the convergence criterion was met.}
#' \item{iter, loglik_history}{Number of evaluated iterations and the
#' corresponding sequence of log likelihoods.}
#' \item{teach.effects}{Data frame containing teacher identifiers, originating
#' and effect years, predicted effects (\code{EBLUP}), and prediction-error
#' standard deviations (\code{std_error}).}
#' \item{parameters}{Matrix of fixed-effect, covariance, and applicable
#' persistence estimates with their standard errors. Covariance and
#' persistence standard errors are \code{NA} unless \code{hessian = TRUE}.}
#' \item{Hessian}{Observed information matrix for covariance and persistence
#' parameters when requested, otherwise \code{NA}.}
#' \item{R_i}{Within-student residual covariance for R-side models;
#' \code{NA} for G-side models.}
#' \item{teach.cov}{List of teacher covariance blocks, one per originating year.}
#' \item{stu.cov}{Student random-effect variance for a G-side model;
#' \code{NA} for an R-side model.}
#' \item{mresid, cresid, sresid}{Marginal residuals, conditional residuals, and
#' conditional residuals scaled using the residual precision matrix.}
#' \item{y, yhat, yhat.m, yhat.s}{Observed responses, conditional fitted values,
#' marginal fitted values, and scaled conditional fitted values, in the
#' internal fitting order.}
#' \item{X, Z, G, R.inv, vinv}{Fixed- and random-effects design matrices,
#' random-effects covariance, residual precision, and marginal precision.}
#' \item{ybetas, eta.hat}{Unrounded fixed-effect estimates and random-effect
#' predictions.}
#' \item{num.obs, num.student, num.year, num.teach}{Observation, student, year,
#' and per-year teacher counts.}
#' \item{persistence, student.side, REML}{The fitted model and estimation options.}
#' \item{key}{Mapping between original year values and internal year indices.}
#' \item{observation}{Input-row indices in the internal fitting order.}
#' \item{simulation_spec}{Original student/year/teacher row keys, observed-score
#' pattern, collation locale, and resolved numerical controls used by \code{\link{simulate_bias}}.
#' These keys are private input metadata, not an aggregate summary.}
#' \item{call, fixed_effects}{Original model call and fixed-effects formula.}
#'
#' Use the standard \code{\link{GPvam-methods}} to extract full-precision
#' fixed-effect estimates and covariances, fitted values, residuals, likelihood,
#' and observation count. The fitted-value and residual methods return
#' observations in input order with missing responses omitted.
#'
#' @inherit GPvam-package references
#' @author Andrew Karl \email{akarl@asu.edu}, Yan Yang and Sharon Lohr.
#' @seealso \code{\link{plot.GPvam}}, \code{\link{summary.GPvam}},
#' \code{\link{bias.test.custom}}, \code{\link{vam_data}}
#' @examples
#' data(vam_data)
#' \donttest{
#' result <- GPvam(vam_data, persistence = "CP",
#' fixed_effects = ~ as.factor(year) + cont_var + 0,
#' verbose = FALSE)
#' summary(result)
#' plot(result, ask = FALSE)
#' }
#' @keywords regression
#' @export
GPvam <- function(vam_data, fixed_effects = formula(~as.factor(year) + 0),
student.side = "R", persistence = "GP", max.iter.EM = 1000,
tol1 = 1e-07, hessian = FALSE, hes.method = "simple",
REML = TRUE, verbose = TRUE) {
cl <- match.call()
default_fixed <- missing(fixed_effects)
if (!is.data.frame(vam_data) || nrow(vam_data) == 0L)
stop("vam_data must be a nonempty data frame.", call. = FALSE)
if (anyDuplicated(names(vam_data)))
stop("vam_data must have unique column names.", call. = FALSE)
required <- c("y", "student", "teacher", "year")
absent <- setdiff(required, names(vam_data))
if (length(absent))
stop("vam_data is missing required columns: ", paste(absent, collapse = ", "), call. = FALSE)
if (!is.numeric(vam_data$y) || any(!is.finite(vam_data$y) & !is.na(vam_data$y)))
stop("y must be numeric, with finite observed values (NA is allowed).", call. = FALSE)
if (!is.numeric(vam_data$year) || any(!is.finite(vam_data$year)))
stop("year must contain finite numeric values with no missing values.", call. = FALSE)
valid_id <- function(x) is.factor(x) || is.character(x) || is.numeric(x)
if (!valid_id(vam_data$student) || anyNA(vam_data$student) ||
(is.numeric(vam_data$student) && any(!is.finite(vam_data$student))))
stop("student must contain nonmissing finite numeric, character, or factor IDs.", call. = FALSE)
if (!valid_id(vam_data$teacher) ||
(is.numeric(vam_data$teacher) && any(!is.finite(vam_data$teacher) & !is.na(vam_data$teacher))))
stop("teacher must contain numeric, character, or factor IDs (NA is allowed).", call. = FALSE)
if (anyDuplicated(vam_data[c("student", "year")]))
stop("Each student must have at most one row per year.", call. = FALSE)
choice <- function(x, choices, name) {
if (!is.character(x) || length(x) != 1L || is.na(x) || !x %in% choices)
stop(name, " must be one of: ", paste(choices, collapse = ", "), ".", call. = FALSE)
x
}
student.side <- choice(student.side, c("R", "G"), "student.side")
persistence <- choice(persistence, c("GP", "rGP", "VP", "CP", "ZP"), "persistence")
hes.method <- choice(hes.method, c("simple", "richardson"), "hes.method")
for (nm in c("hessian", "REML", "verbose")) {
value <- get(nm)
if (!is.logical(value) || length(value) != 1L || is.na(value))
stop(nm, " must be TRUE or FALSE.", call. = FALSE)
}
if (!is.numeric(max.iter.EM) || length(max.iter.EM) != 1L ||
!is.finite(max.iter.EM) || max.iter.EM < 1 || max.iter.EM != floor(max.iter.EM) ||
max.iter.EM > .Machine$integer.max)
stop("max.iter.EM must be a positive integer.", call. = FALSE)
if (!is.numeric(tol1) || length(tol1) != 1L || !is.finite(tol1) || tol1 <= 0)
stop("tol1 must be a positive finite number.", call. = FALSE)
if (!inherits(fixed_effects, "formula") || length(fixed_effects) != 2L)
stop("fixed_effects must be a one-sided formula.", call. = FALSE)
if (length(attr(stats::terms(fixed_effects), "offset")))
stop("Offsets in fixed_effects are not supported.", call. = FALSE)
if (length(intersect(all.vars(fixed_effects), c("mis", "pat", "r"))))
stop("The names mis, pat, and r are reserved for internal model bookkeeping.", call. = FALSE)
if (student.side == "G" && persistence != "GP")
stop('student.side = "G" currently requires persistence = "GP".', call. = FALSE)
Z_mat <- as.data.frame(vam_data)
observed <- !is.na(Z_mat$y)
if (!any(observed)) stop("There are no observed scores.", call. = FALSE)
if (student.side == "G" && all(table(Z_mat$student[observed]) <= 1L))
stop("G-side student variance requires repeated observed scores for at least one student.",
call. = FALSE)
years <- sort(unique(Z_mat$year))
if (default_fixed && length(years) == 1L) fixed_effects <- ~1
year_group <- match(Z_mat$year, years)
counts <- tabulate(year_group[observed], nbins = length(years))
if (any(counts < 2L))
stop("Each year must have at least two observed scores.", call. = FALSE)
spreads <- vapply(split(Z_mat$y[observed], year_group[observed]), stats::var, numeric(1))
if (any(!is.finite(spreads) | spreads <= 0))
stop("Each year must have positive observed score variance.", call. = FALSE)
retained <- Z_mat$student %in% Z_mat$student[observed]
known_years <- tabulate(year_group[retained & !is.na(Z_mat$teacher)],
nbins = length(years))
if (any(known_years == 0L))
stop("Each year must have a known teacher for a student with observed scores.", call. = FALSE)
# Keep actual NA teacher IDs; do not turn them into a teacher named 'NA'.
Z_mat$teacher <- as.character(Z_mat$teacher)
known <- !is.na(Z_mat$teacher)
Z_mat$teacher[known] <- paste0(Z_mat$teacher[known], "(year", year_group[known], ")")
Z_mat$year <- year_group
key <- cbind(years, seq_along(years))
# Evaluate only rows with observed responses. Irrelevant extra columns and
# covariates on missing-score rows cannot affect the likelihood.
mf <- stats::model.frame(fixed_effects, data = Z_mat[observed, , drop = FALSE],
na.action = stats::na.fail, drop.unused.levels = TRUE)
if (nrow(mf) != sum(observed))
stop("fixed_effects must describe the observed rows of vam_data.", call. = FALSE)
design <- stats::model.matrix(fixed_effects, data = mf)
if (!ncol(design) || any(!is.finite(design)))
stop("The fixed-effects model matrix must have finite entries and at least one column.", call. = FALSE)
if (qr(design)$rank < ncol(design))
stop("The fixed-effects design matrix must have full column rank.", call. = FALSE)
if (nrow(design) <= ncol(design))
stop("The number of observations must exceed the number of fixed effects.", call. = FALSE)
control <- list(max.iter.EM = as.integer(max.iter.EM), tol1 = tol1,
hessian = hessian, hes.method = hes.method, verbose = verbose,
persistence = persistence, REML = REML, key = key)
if (student.side == "G") {
res <- GP.csh(Z_mat, fixed_effects, control)
} else if (persistence == "GP") {
res <- GP.un(Z_mat, fixed_effects, control)
} else if (persistence == "rGP") {
res <- rGP.un(Z_mat, fixed_effects, control)
} else {
res <- VP.CP.ZP.un(Z_mat, fixed_effects, control)
}
res$key <- key
res$teach.effects$teacher_year <- years[match(res$teach.effects$teacher_year, seq_along(years))]
res$teach.effects$effect_year <- years[match(res$teach.effects$effect_year, seq_along(years))]
res$REML <- REML
res$student.side <- student.side
res$persistence <- persistence
res$call <- cl
res$fixed_effects <- fixed_effects
res$observation <- match(rownames(res$X), rownames(vam_data))
if (length(res$observation) != res$num.obs || anyNA(res$observation))
stop("Internal observation mapping failed.", call. = FALSE)
# Retain only input keys and resolved controls needed to validate a refit.
# Scores and covariates are already represented by y and X; do not duplicate
# the full input data or evaluate unevaluated expressions from match.call().
res$simulation_spec <- list(
data_keys = vam_data[, c("student", "teacher", "year"), drop = FALSE],
observed = !is.na(vam_data$y), row_names = rownames(vam_data),
max.iter.EM = as.integer(max.iter.EM), tol1 = tol1,
collation = Sys.getlocale("LC_COLLATE"))
class(res) <- "GPvam"
if (identical(res$converged, FALSE))
warning("The EM algorithm did not converge within max.iter.EM iterations.", call. = FALSE)
res
}
Any scripts or data that you put into this service are public.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.