data-raw/AR_Modeling.md

Okay, this is excellent. The responses clarify all previous ambiguities and the proposed patches/adjustments are sensible and maintain the efficiency goals.

Here's a draft proposal followed by a ticketed sprint plan:

Proposal: High-Performance AR(p) Prewhitening for fmri_lm

Preamble:

The fmrireg package currently offers a highly optimized "fast path" for fitting fMRI linear models assuming i.i.d. errors. This proposal outlines an extension to incorporate AR(p) serial correlation modeling via prewhitening, maintaining the core speed advantages of the existing pipeline. The method involves estimating run-level AR coefficients from OLS residuals, applying a causal filter to the data (Y) and design matrix (X) for that run, and then re-fitting the GLM on the whitened data. This approach offers a substantial portion of the statistical efficiency of full REML/GLS at a fraction of the computational cost.

1. High-Level Design:

The process for a single run (or globally, if specified) will be:

  1. (If iter == 1): Perform an initial OLS fit on the original (unwhitened) Y_run and X_run.
    • Obtain OLS residuals: e_OLS = Y_run_orig - X_run_orig %*% Beta_OLS.
    • Estimate AR(p) coefficients (phi_hat) from the run-mean e_OLS using Yule-Walker.
  2. Whitening: Apply an AR(p) causal filter to Y_run_orig and X_run_orig using phi_hat to produce Y_run_whitened and X_run_whitened. This is done in-place via an efficient C++ routine.
  3. GLS Fit: Perform a GLM fit on Y_run_whitened and X_run_whitened using the existing .fast_lm_matrix machinery. The resulting Betas and sigma^2 are the GLS estimates.
  4. (Optional, if cor_iter > 1): Re-estimate phi_hat from the residuals of the current GLS fit (Y_run_whitened - X_run_whitened %*% Beta_GLS) and repeat steps 2-3.

2. Key Mechanisms & API Changes:

3. Performance:

4. Benefits:

Ticketed Sprint to Completion:

Epic: Implement AR(p) Prewhitening for fmri_lm

Sprint Goal: Deliver a functional and tested AR(p) prewhitening capability within fmri_lm, significantly improving model accuracy for serially correlated fMRI data while maintaining high performance.

Ticket 1: API Definition & Argument Threading (R) Task: Modify fmri_lm() signature to include cor_struct, cor_iter, cor_global, ar_p, ar1_exact_first. Task: Thread these new arguments down through fmri_lm_fit() to runwise_lm() and chunkwise_lm(). Task: Add basic input validation for new arguments (e.g., ar_p must be positive integer if cor_struct == "arp"). Definition of Done: New arguments are present in function signatures and passed correctly. Unit tests for argument passing. Documentation stubs for new arguments. * Estimate: 0.5 days

Ticket 2: AR Coefficient Estimation (R) Task: Implement R helper function .estimate_ar(residuals_vec, p_order) using stats::acf and Yule-Walker equations (solve(toeplitz(...))). Task: Unit test .estimate_ar with known AR(1) and AR(2) series to verify coefficient recovery. Definition of Done: .estimate_ar function implemented and unit tested. Estimate: 0.5 days

Ticket 3: C++ Whitening Filter Implementation (C++/RcppArmadillo) Task: Implement ar_whiten_inplace(arma::mat& Y, arma::mat& X, const arma::vec& phi_coeffs, bool exact_first_ar1) in C++. * Handle general AR(p) based on phi_coeffs.n_elem. * Implement prev_k rolling buffer for past values. * Include OpenMP pragmas for parallelization over voxels (Y) and predictors (X). * Implement optional exact_first_ar1 scaling for AR(1) case. Task: Expose this function to R via Rcpp. Task: Unit test the C++ filter directly with simple matrices and known phi values, checking output against manual calculation for AR(1) and AR(2). Test exact_first_ar1. Definition of Done: C++ filter implemented, Rcpp-exposed, and unit tested for correctness on small examples. * Estimate: 1.5 days

Ticket 4: Modify .fast_lm_matrix (R) Task: Amend signature: .fast_lm_matrix(X, Y, proj, return_fitted = FALSE). Task: Implement conditional calculation: if return_fitted, compute Fitted = X %*% Betas and rss = colSums((Y - Fitted)^2). Task: Ensure list(..., fitted = if(return_fitted) Fitted else NULL, ...) is returned. Definition of Done: .fast_lm_matrix updated and existing unit tests (if any specifically for it) still pass or are adapted. * Estimate: 0.5 days

Ticket 5: Integrate Whitening into runwise_lm (R) Task: Implement the iterative loop logic within runwise_lm: * If cor_struct != "iid": * Initial OLS pass (iter == 1 or if phi_hat is NULL): * Call .fast_lm_matrix(X_run_orig, Y_run_orig, proj_run_orig, return_fitted = TRUE). * Compute Resid_OLS = Y_run_orig - ols$fitted. * Estimate phi_hat_run = .estimate_ar(rowMeans(Resid_OLS), p = ar_order). * (If cor_global = TRUE, this phi_hat_run contributes to a global estimate; actual global estimation logic is part of Ticket 7). * Make copies X_run_iter = X_run_orig, Y_run_iter = Y_run_orig. * Call ar_whiten_inplace(Y_run_iter, X_run_iter, phi_hat_run, ar1_exact_first). * proj_run_iter = .fast_preproject(X_run_iter). * gls_results = .fast_lm_matrix(X_run_iter, Y_run_iter, proj_run_iter). * Update Betas, sigma, etc. from gls_results for this iteration's output. * If cor_iter > iter: * Compute Resid_GLS = Y_run_iter - X_run_iter %*% gls_results$betas. * Re-estimate phi_hat_run = .estimate_ar(rowMeans(Resid_GLS), p = ar_order). Task: Ensure correct X_run_orig, Y_run_orig (unwhitened versions) are used for the first OLS residual calculation and as input to ar_whiten_inplace in each iteration. Task: Ensure proj_run_orig (from unwhitened X_run_orig) is used for the initial OLS, and proj_run_iter (from whitened X_run_iter) is used for the GLS step. Definition of Done: runwise_lm correctly performs OLS (if iter==1), estimates phi_hat_run, whitens, and performs GLS. Results are passed to existing pooling logic. * Estimate: 2 days

Ticket 6: Integrate Whitening into chunkwise_lm (R) Task: Adapt chunkwise_lm to incorporate run-level AR estimation and whitening: * Before chunking loop for a given run: * Perform initial OLS on the full run's data (Y_run_orig, X_run_orig) to get Resid_OLS_run. * Estimate phi_hat_run using .estimate_ar(rowMeans(Resid_OLS_run), p = ar_order). * (If cor_global = TRUE, this contributes to global estimate, see Ticket 7). * Create whitened versions of the full run's data: Y_run_w = Y_run_orig, X_run_w = X_run_orig. Call ar_whiten_inplace(Y_run_w, X_run_w, phi_hat_run, ar1_exact_first). * The existing chunking loop in chunkwise_lm then operates on slices of Y_run_w. * The design matrix X_run_w is pre-projected once for the whitened run: proj_run_w = .fast_preproject(X_run_w). * Inside the chunk loop, .fast_lm_matrix is called with Y_chunk_w (from Y_run_w) and proj_run_w. Task: Iteration (cor_iter > 1) for chunkwise_lm would be more complex as phi_hat would need to be re-estimated from full-run GLS residuals. Initially, chunkwise_lm might support cor_iter = 1 only, or this ticket needs careful planning for iteration. (Clarification: for iter > 1, re-estimate phi_hat_run from residuals reconstructed from all chunks of that run, then re-whiten the full run data and re-chunk). Definition of Done: chunkwise_lm correctly uses run-level phi_hat to whiten full run data before chunking and GLM fitting. Handles at least cor_iter = 1. Estimate: 2.5 days (extra 0.5 for managing iteration if included)

Ticket 7: Implement cor_global = TRUE Logic (R) Task: Modify fmri_lm_fit (or a new orchestrator function called by it). * If cor_global = TRUE and cor_struct != "iid": * First pass: Loop through all runs (using runwise_lm or chunkwise_lm logic for OLS part only up to Resid_OLS_run). * Collect/Aggregate Resid_OLS_run from all runs (e.g., concatenate or average autocovariances). * Estimate a single phi_hat_global using .estimate_ar. * Second pass: Loop through all runs/chunks again. This time, all runs/chunks use phi_hat_global for whitening. The GLS estimation proceeds as in Tickets 5 & 6, but phi_hat is fixed to phi_hat_global and not re-estimated per run/iteration. cor_iter for global would typically be 1. Definition of Done: cor_global = TRUE correctly estimates and applies a global phi_hat. * Estimate: 1 day

Ticket 8: Unit & Integration Tests (R/testthat) Task: Create test cases: * Simulated data with no serial correlation (cor_struct = "iid" should match cor_struct = "ar1" with phi_hat near 0). * Simulated data with known AR(1) (e.g., phi = 0.4). Verify phi_hat recovery and that GLS standard errors are more accurate than OLS. * Simulated data with known AR(2). * Test cor_global = TRUE vs. cor_global = FALSE. * Test ar1_exact_first = TRUE vs. FALSE. * Test with cor_iter > 1 (if implemented for chunkwise_lm). Task: Compare results (betas, SEs, t-stats) against a known package (e.g., nlme::gls or a manual GLS calculation on a small dataset) if feasible for validation. Definition of Done: Comprehensive test suite covering different cor_struct options, iterations, and global/run-specific estimation. Estimate: 2 days

Ticket 9: Documentation & Vignette Update (R/Rmd) Task: Document new fmri_lm arguments (cor_struct, cor_iter, etc.). Task: Explain the AR(p) prewhitening method, its benefits, and usage. Task: Add an example to a vignette demonstrating AR(1) correction. Task: Clarify "arp" means AR-only in docs. Definition of Done: User-facing documentation and vignette updated. Estimate: 1 day

Total Estimated Time: 9 days (adjusting for chunkwise_lm iteration complexity)

Okay, this is a very interesting and practical extension to the AR(p) prewhitening proposal. It addresses a common concern with global or run-level AR models: their potential inadequacy in brain regions with heterogeneous noise characteristics. The tiered approach is sensible.

Here's the draft appendix, incorporating my spin:

Appendix: Locally Adaptive Serial Correlation Modeling

A.1 Motivation: Beyond Global AR Models

While run-level AR(p) prewhitening (as outlined in the main proposal) significantly improves upon i.i.d. assumptions with minimal computational overhead, fMRI noise characteristics can vary spatially. Different tissue types (grey matter, white matter, CSF) and even distinct grey matter regions can exhibit different temporal autocorrelation structures. A single AR(p) coefficient vector (phi_hat_run) applied to an entire run might be suboptimal for some voxels.

This appendix proposes a tiered strategy to introduce locally adaptive AR(p) coefficients, allowing the whitening process to be more sensitive to these spatial variations, while still aiming to preserve the "lightning-fast" nature of the GLM pipeline.

A.2 Tiered Strategy for AR Coefficient Estimation & Application

The core idea is to refine the phi_hat used for whitening based on local information, ranging from a single run-level estimate to voxel-specific, regularized estimates.

A.3 Implementation Details & API Considerations

A.4 Performance:

A.5 User Guidance & Defaults:

A.6 Ticketed Sprint Additions (for Locally Adaptive AR):

This would follow the initial AR(p) implementation.

This tiered approach offers a flexible and performant way to handle spatially varying serial correlations, building incrementally on the foundational run-level AR prewhitening.



bbuchsbaum/fmrireg documentation built on June 10, 2025, 8:18 p.m.