sim.train.test: Simulate in-sample and out-of-sample model performance

In rmcelreath/rethinking: Statistical Rethinking book package

Simulate in-sample and out-of-sample model performance

Description

Simulates in-sample and out-of-sample model performance for simple quap models, returning lppd in and out of sample, WAIC, LOOIC, and LOOCV.

Usage

sim_train_test(N = 20, k = 3, rho = c(0.15, -0.4), b_sigma = 100, 
    WAIC = FALSE, LOOCV=FALSE , LOOIC=FALSE , cv.cores=1 , 
    return_model=FALSE )

# old function from 1st edition
sim.train.test( N=20 , k=3 , rho=c(0.15,-0.4) , b_sigma=100 , 
    DIC=FALSE , WAIC=FALSE, devbar=FALSE , devbarout=FALSE )

Arguments

N	Number of cases in simulated data
k	Number of parameters in model to fit to data
rho	Vector of correlations between predictors and outcome, in simulated data
b_sigma	Standard deviation of beta-coefficient priors
DIC	If TRUE, returns DIC
WAIC	If TRUE, returns WAIC
LOOIC	If TRUE, returns LOOIC as produced by LOO
LOOCV	If TRUE, returns LOOCV as produced by cv_quap
devbar	If TRUE, returns the average deviance in-sample
devbarout	If TRUE, returns average deviance out-of-sample
cv.cores	Number or cores to use for cross-validation
return_model	If TRUE, includes fit model in result

Details

This function simulates Gaussian data and then fits linear regression models to it, returning the lppd of the fit as produced by lppd (training, in-sample) and the deviance on a new sample, computed using the posterior from the training sample.

Author(s)

Richard McElreath

See Also

lppd, WAIC, LOO, cv_quap

rmcelreath/rethinking documentation built on Sept. 18, 2023, 2:01 p.m.