graper: Fit a regression model with graper

In graper: Adaptive penalization in high-dimensional regression and classification with external covariates using variational Bayes

Description

Fit a regression model with graper given a matrix of predictors (X), a response vector (y) and a vector of group memberships for each predictor in X (annot). For each group a different strength of penalization is determined adaptively.

Usage

graper(X, y, annot, factoriseQ = TRUE, spikeslab = TRUE,
  intercept = TRUE, family = "gaussian", standardize = TRUE,
  n_rep = 1, max_iter = 3000, th = 0.01, d_tau = 0.001,
  r_tau = 0.001, d_gamma = 0.001, r_gamma = 0.001, r_pi = 1,
  d_pi = 1, calcELB = TRUE, verbose = TRUE, freqELB = 1,
  nogamma = FALSE, init_psi = 1)

Arguments

X	design matrix of size n (samples) x p (features)
y	response vector of size n
annot	factor of length p indicating group membership of each feature (column) in X
factoriseQ	if set to TRUE, the variational distribution is assumed to fully factorize across features (faster, default). If FALSE, a multivariate variational distribution is used.
spikeslab	if set to TRUE, a spike and slab prior on the coefficients (default).
intercept	whether to include an intercept into the model
family	Likelihood model for the response, either "gaussian" for linear regression or "binomial" for logistic regression
standardize	whether to standardize the predictors to unit variance
n_rep	number of repetitions with different random initializations to be fit
max_iter	maximum number of iterations
th	convergence threshold for the evidence lower bound (ELB)
d_tau	hyper-parameters for prior of tau (noise precision)
r_tau	hyper-parameters for prior of tau (noise precision)
d_gamma	hyper-parameters for prior of gamma (coefficients' prior precision)
r_gamma	hyper-parameters for prior of gamma (coefficients' prior precision)
r_pi	hyper-parameters for Beta prior of the mixture probabilities in the spike and slab prior
d_pi	hyper-parameters for Beta prior of the mixture probabilities in the spike and slab prior
calcELB	whether to calculate the evidence lower bound (ELB)
verbose	whether to print out intermediate messages during fitting
freqELB	frequency at which the evidence lower bound (ELB) is to be calculated, i.e. each freqELB-th iteration
nogamma	if TRUE, the normal prior will have same variance for all groups (only relevant for spikeslab = TRUE)
init_psi	initial value for the spike variables

Details

The function trains the graper model given a matrix of predictors (X), a response vector (y) and a vector of group memberships for each predictor in X (annot). For each feature group as specified in annot a penalty factor and sparsity level is learnt.

By default it uses a Spike-and-Slab prior on the coefficients and uses a fully factorized variational distribution in the inference. This provides a fast way to train the model. Using spikeslab=FALSE a ridge regression like model can be fitted using a normal instead of the spike and slab prior. Setting factoriseQ = FALSE gives a more exact inference scheme based on a multivariate variational distribution, but can be much slower.

As the optimization is non-convex is can be helpful to use multiple random initializations by setting n_rep to a value larger 1. The returned model is then chosen as the optimal fit with respect to the evidence lower bound (ELB).

Depending on the response vector a linear regression model (family = "gaussian") or a logistic regression model (family = "binomial") is fitted. Note, that the implementation of logistic regression is still experimental.

Value

A graper object containing

EW_beta

estimated model coefficients in liner/logistic regression

EW_s

estimated posterior-inclusion probabilities for each feature

intercept

estimated intercept term

annot

annotation vector of features to the groups as specified when calling graper

EW_gamma

estimated penalty factor per group

EW_pi

estimated sparsity level per group (from 1 (dense) to 0 (sparse))

EW_tau

estimated noise precision

sigma2_tildebeta_s1, EW_tildebeta_s1, alpha_gamma, alpha_tau, beta_tau, Sigma_beta, alpha_pi, beta_pi

parameters of the variational distributions of beta, gamma, tau and pi

ELB

final value of the evidence lower bound

ELB_trace

values of the evidence lower bound for all iterations

Options

other options used when calling graper

Examples

# create data
dat <- makeExampleData()

# fit a sparse model with spike and slab prior
fit <- graper(dat$X, dat$y, dat$annot)
fit # print fitted object
beta <- coef(fit, include_intercept=FALSE) # model coeffients
pips <- getPIPs(fit) # posterior inclusion probabilities
pf <- fit$EW_gamma # penalty factors per group
sparsities <- fit$EW_pi # sparsity levels per group

# fit a dense model without spike and slab prior
fit <- graper(dat$X, dat$y, dat$annot, spikeslab=FALSE)

# fit a dense model using a multivariate variational distribution
fit <- graper(dat$X, dat$y, dat$annot, factoriseQ=TRUE,
            spikeslab=FALSE)

Example output

Fitting a model with 4 groups, 100 samples and 200 features.
Fitting with random init 1
iteration 1
Updating parameters of variational distributions...
Calculating ELB..
ELB=-822.083
ELB improved by inf

iteration 2
Updating parameters of variational distributions...
Calculating ELB..
ELB=-557.275
ELB improved by 264.809

iteration 3
Updating parameters of variational distributions...
Calculating ELB..
ELB=-492.352
ELB improved by 64.9229

iteration 4
Updating parameters of variational distributions...
Calculating ELB..
ELB=-454.013
ELB improved by 38.3385

iteration 5
Updating parameters of variational distributions...
Calculating ELB..
ELB=-437.505
ELB improved by 16.5077

iteration 6
Updating parameters of variational distributions...
Calculating ELB..
ELB=-430.189
ELB improved by 7.31653

iteration 7
Updating parameters of variational distributions...
Calculating ELB..
ELB=-425.753
ELB improved by 4.43587

iteration 8
Updating parameters of variational distributions...
Calculating ELB..
ELB=-422.099
ELB improved by 3.65432

iteration 9
Updating parameters of variational distributions...
Calculating ELB..
ELB=-418.641
ELB improved by 3.45764

iteration 10
Updating parameters of variational distributions...
Calculating ELB..
ELB=-415.2
ELB improved by 3.44072

iteration 11
Updating parameters of variational distributions...
Calculating ELB..
ELB=-411.697
ELB improved by 3.50329

iteration 12
Updating parameters of variational distributions...
Calculating ELB..
ELB=-408.102
ELB improved by 3.59486

iteration 13
Updating parameters of variational distributions...
Calculating ELB..
ELB=-404.465
ELB improved by 3.63737

iteration 14
Updating parameters of variational distributions...
Calculating ELB..
ELB=-400.968
ELB improved by 3.49687

iteration 15
Updating parameters of variational distributions...
Calculating ELB..
ELB=-397.921
ELB improved by 3.04671

iteration 16
Updating parameters of variational distributions...
Calculating ELB..
ELB=-395.576
ELB improved by 2.34505

iteration 17
Updating parameters of variational distributions...
Calculating ELB..
ELB=-393.941
ELB improved by 1.63469

iteration 18
Updating parameters of variational distributions...
Calculating ELB..
ELB=-392.84
ELB improved by 1.10143

iteration 19
Updating parameters of variational distributions...
Calculating ELB..
ELB=-392.074
ELB improved by 0.765761

iteration 20
Updating parameters of variational distributions...
Calculating ELB..
ELB=-391.507
ELB improved by 0.567768

iteration 21
Updating parameters of variational distributions...
Calculating ELB..
ELB=-391.057
ELB improved by 0.449511

iteration 22
Updating parameters of variational distributions...
Calculating ELB..
ELB=-390.682
ELB improved by 0.37535

iteration 23
Updating parameters of variational distributions...
Calculating ELB..
ELB=-390.355
ELB improved by 0.326375

iteration 24
Updating parameters of variational distributions...
Calculating ELB..
ELB=-390.062
ELB improved by 0.292903

iteration 25
Updating parameters of variational distributions...
Calculating ELB..
ELB=-389.792
ELB improved by 0.269909

iteration 26
Updating parameters of variational distributions...
Calculating ELB..
ELB=-389.538
ELB improved by 0.254717

iteration 27
Updating parameters of variational distributions...
Calculating ELB..
ELB=-389.292
ELB improved by 0.245896

iteration 28
Updating parameters of variational distributions...
Calculating ELB..
ELB=-389.049
ELB improved by 0.242752

iteration 29
Updating parameters of variational distributions...
Calculating ELB..
ELB=-388.804
ELB improved by 0.245102

iteration 30
Updating parameters of variational distributions...
Calculating ELB..
ELB=-388.551
ELB improved by 0.253227

iteration 31
Updating parameters of variational distributions...
Calculating ELB..
ELB=-388.283
ELB improved by 0.267864

iteration 32
Updating parameters of variational distributions...
Calculating ELB..
ELB=-387.993
ELB improved by 0.289983

iteration 33
Updating parameters of variational distributions...
Calculating ELB..
ELB=-387.673
ELB improved by 0.319934

iteration 34
Updating parameters of variational distributions...
Calculating ELB..
ELB=-387.317
ELB improved by 0.356117

iteration 35
Updating parameters of variational distributions...
Calculating ELB..
ELB=-386.922
ELB improved by 0.395198

iteration 36
Updating parameters of variational distributions...
Calculating ELB..
ELB=-386.486
ELB improved by 0.435555

iteration 37
Updating parameters of variational distributions...
Calculating ELB..
ELB=-386.006
ELB improved by 0.479902

iteration 38
Updating parameters of variational distributions...
Calculating ELB..
ELB=-385.474
ELB improved by 0.532656

iteration 39
Updating parameters of variational distributions...
Calculating ELB..
ELB=-384.878
ELB improved by 0.595729

iteration 40
Updating parameters of variational distributions...
Calculating ELB..
ELB=-384.211
ELB improved by 0.666537

iteration 41
Updating parameters of variational distributions...
Calculating ELB..
ELB=-383.474
ELB improved by 0.737409

iteration 42
Updating parameters of variational distributions...
Calculating ELB..
ELB=-382.678
ELB improved by 0.795859

iteration 43
Updating parameters of variational distributions...
Calculating ELB..
ELB=-381.851
ELB improved by 0.82677

iteration 44
Updating parameters of variational distributions...
Calculating ELB..
ELB=-381.034
ELB improved by 0.817151

iteration 45
Updating parameters of variational distributions...
Calculating ELB..
ELB=-380.272
ELB improved by 0.762097

iteration 46
Updating parameters of variational distributions...
Calculating ELB..
ELB=-379.604
ELB improved by 0.668506

iteration 47
Updating parameters of variational distributions...
Calculating ELB..
ELB=-379.05
ELB improved by 0.553243

iteration 48
Updating parameters of variational distributions...
Calculating ELB..
ELB=-378.614
ELB improved by 0.436068

iteration 49
Updating parameters of variational distributions...
Calculating ELB..
ELB=-378.282
ELB improved by 0.331992

iteration 50
Updating parameters of variational distributions...
Calculating ELB..
ELB=-378.034
ELB improved by 0.247893

iteration 51
Updating parameters of variational distributions...
Calculating ELB..
ELB=-377.85
ELB improved by 0.183939

iteration 52
Updating parameters of variational distributions...
Calculating ELB..
ELB=-377.713
ELB improved by 0.136908

iteration 53
Updating parameters of variational distributions...
Calculating ELB..
ELB=-377.611
ELB improved by 0.102808

iteration 54
Updating parameters of variational distributions...
Calculating ELB..
ELB=-377.533
ELB improved by 0.078134

iteration 55
Updating parameters of variational distributions...
Calculating ELB..
ELB=-377.472
ELB improved by 0.060197

iteration 56
Updating parameters of variational distributions...
Calculating ELB..
ELB=-377.425
ELB improved by 0.0470508

iteration 57
Updating parameters of variational distributions...
Calculating ELB..
ELB=-377.388
ELB improved by 0.0373209

iteration 58
Updating parameters of variational distributions...
Calculating ELB..
ELB=-377.358
ELB improved by 0.0300429

iteration 59
Updating parameters of variational distributions...
Calculating ELB..
ELB=-377.333
ELB improved by 0.024539

iteration 60
Updating parameters of variational distributions...
Calculating ELB..
ELB=-377.313
ELB improved by 0.0203304

iteration 61
Updating parameters of variational distributions...
Calculating ELB..
ELB=-377.296
ELB improved by 0.0170759

iteration 62
Updating parameters of variational distributions...
Calculating ELB..
ELB=-377.281
ELB improved by 0.0145306

iteration 63
Updating parameters of variational distributions...
Calculating ELB..
ELB=-377.269
ELB improved by 0.0125175

iteration 64
Updating parameters of variational distributions...
Calculating ELB..
ELB=-377.258
ELB improved by 0.0109071

iteration 65
Updating parameters of variational distributions...
Calculating ELB..
ELB=-377.248
ELB improved by 0.0096043

ELB converged
Sparse graper object for a linear regression model with 200 predictors in 4 groups.
 Group-wise shrinkage:
 1	2	3	4 
 0.15	0.93	124.72	128.14 
Group-wise sparsity (1 = dense, 0 = sparse):
1	2	3	4 
0.41	0.22	0.28	0.29Fitting a model with 4 groups, 100 samples and 200 features.
Fitting with random init 1
iteration 1
ELB=-785.165 diff=inf
iteration 2
ELB=-550.081 diff=235.084
iteration 3
ELB=-493.043 diff=57.0378
iteration 4
ELB=-451.073 diff=41.9705
iteration 5
ELB=-429.918 diff=21.1542
iteration 6
ELB=-422.02 diff=7.89852
iteration 7
ELB=-418.407 diff=3.61263
iteration 8
ELB=-415.732 diff=2.67528
iteration 9
ELB=-413.388 diff=2.34425
iteration 10
ELB=-411.251 diff=2.13663
iteration 11
ELB=-409.29 diff=1.9613
iteration 12
ELB=-407.495 diff=1.79447
iteration 13
ELB=-405.865 diff=1.63
iteration 14
ELB=-404.398 diff=1.46779
iteration 15
ELB=-403.088 diff=1.30978
iteration 16
ELB=-401.929 diff=1.15843
iteration 17
ELB=-400.913 diff=1.01612
iteration 18
ELB=-400.028 diff=0.884762
iteration 19
ELB=-399.263 diff=0.765691
iteration 20
ELB=-398.603 diff=0.659568
iteration 21
ELB=-398.037 diff=0.566419
iteration 22
ELB=-397.551 diff=0.485733
iteration 23
ELB=-397.134 diff=0.416599
iteration 24
ELB=-396.777 diff=0.357859
iteration 25
ELB=-396.468 diff=0.308244
iteration 26
ELB=-396.202 diff=0.266485
iteration 27
ELB=-395.97 diff=0.231385
iteration 28
ELB=-395.769 diff=0.201866
iteration 29
ELB=-395.592 diff=0.17699
iteration 30
ELB=-395.436 diff=0.155956
iteration 31
ELB=-395.298 diff=0.138096
iteration 32
ELB=-395.175 diff=0.122858
iteration 33
ELB=-395.065 diff=0.109791
iteration 34
ELB=-394.966 diff=0.098525
iteration 35
ELB=-394.878 diff=0.0887612
iteration 36
ELB=-394.797 diff=0.0802553
iteration 37
ELB=-394.725 diff=0.0728081
iteration 38
ELB=-394.658 diff=0.0662568
iteration 39
ELB=-394.598 diff=0.0604677
iteration 40
ELB=-394.542 diff=0.0553302
iteration 41
ELB=-394.492 diff=0.0507531
iteration 42
ELB=-394.445 diff=0.0466598
iteration 43
ELB=-394.402 diff=0.0429865
iteration 44
ELB=-394.362 diff=0.0396793
iteration 45
ELB=-394.326 diff=0.0366927
iteration 46
ELB=-394.292 diff=0.0339878
iteration 47
ELB=-394.26 diff=0.0315316
iteration 48
ELB=-394.231 diff=0.0292956
iteration 49
ELB=-394.204 diff=0.0272552
iteration 50
ELB=-394.178 diff=0.0253892
iteration 51
ELB=-394.155 diff=0.0236792
iteration 52
ELB=-394.132 diff=0.022109
iteration 53
ELB=-394.112 diff=0.0206646
iteration 54
ELB=-394.092 diff=0.0193336
iteration 55
ELB=-394.074 diff=0.0181051
iteration 56
ELB=-394.057 diff=0.0169694
iteration 57
ELB=-394.041 diff=0.015918
iteration 58
ELB=-394.027 diff=0.0149432
iteration 59
ELB=-394.012 diff=0.0140384
iteration 60
ELB=-393.999 diff=0.0131974
iteration 61
ELB=-393.987 diff=0.0124149
iteration 62
ELB=-393.975 diff=0.0116859
iteration 63
ELB=-393.964 diff=0.0110062
iteration 64
ELB=-393.954 diff=0.0103716
iteration 65
ELB=-393.944 diff=0.00977875
ELB converged
Fitting a model with 4 groups, 100 samples and 200 features.
Fitting with random init 1
iteration 1
ELB=-840.81 diff=inf
iteration 2
ELB=-569.729 diff=271.081
iteration 3
ELB=-510.196 diff=59.5327
iteration 4
ELB=-463.528 diff=46.6679
iteration 5
ELB=-438.147 diff=25.3809
iteration 6
ELB=-428.045 diff=10.1026
iteration 7
ELB=-423.303 diff=4.74183
iteration 8
ELB=-420.078 diff=3.22518
iteration 9
ELB=-417.413 diff=2.66511
iteration 10
ELB=-415.014 diff=2.39905
iteration 11
ELB=-412.793 diff=2.22074
iteration 12
ELB=-410.73 diff=2.06337
iteration 13
ELB=-408.822 diff=1.90715
iteration 14
ELB=-407.074 diff=1.74819
iteration 15
ELB=-405.487 diff=1.58765
iteration 16
ELB=-404.058 diff=1.42814
iteration 17
ELB=-402.786 diff=1.27258
iteration 18
ELB=-401.662 diff=1.12378
iteration 19
ELB=-400.678 diff=0.984185
iteration 20
ELB=-399.822 diff=0.855712
iteration 21
ELB=-399.083 diff=0.73961
iteration 22
ELB=-398.446 diff=0.636443
iteration 23
ELB=-397.9 diff=0.546145
iteration 24
ELB=-397.432 diff=0.468125
iteration 25
ELB=-397.03 diff=0.40142
iteration 26
ELB=-396.686 diff=0.344844
iteration 27
ELB=-396.388 diff=0.29712
iteration 28
ELB=-396.131 diff=0.256989
iteration 29
ELB=-395.908 diff=0.223276
iteration 30
ELB=-395.713 diff=0.19493
iteration 31
ELB=-395.542 diff=0.171038
iteration 32
ELB=-395.391 diff=0.150831
iteration 33
ELB=-395.258 diff=0.133664
iteration 34
ELB=-395.139 diff=0.119008
iteration 35
ELB=-395.032 diff=0.10643
iteration 36
ELB=-394.937 diff=0.0955778
iteration 37
ELB=-394.851 diff=0.0861647
iteration 38
ELB=-394.773 diff=0.0779574
iteration 39
ELB=-394.702 diff=0.0707659
iteration 40
ELB=-394.637 diff=0.0644344
iteration 41
ELB=-394.579 diff=0.0588351
iteration 42
ELB=-394.525 diff=0.0538626
iteration 43
ELB=-394.475 diff=0.0494291
iteration 44
ELB=-394.43 diff=0.0454617
iteration 45
ELB=-394.388 diff=0.0418991
iteration 46
ELB=-394.349 diff=0.0386896
iteration 47
ELB=-394.313 diff=0.0357896
iteration 48
ELB=-394.28 diff=0.0331618
iteration 49
ELB=-394.249 diff=0.0307743
iteration 50
ELB=-394.221 diff=0.0285998
iteration 51
ELB=-394.194 diff=0.0266146
iteration 52
ELB=-394.169 diff=0.0247984
iteration 53
ELB=-394.146 diff=0.0231333
iteration 54
ELB=-394.125 diff=0.0216038
iteration 55
ELB=-394.105 diff=0.0201963
iteration 56
ELB=-394.086 diff=0.0188989
iteration 57
ELB=-394.068 diff=0.017701
iteration 58
ELB=-394.051 diff=0.0165933
iteration 59
ELB=-394.036 diff=0.0155675
iteration 60
ELB=-394.021 diff=0.0146162
iteration 61
ELB=-394.007 diff=0.013733
iteration 62
ELB=-393.995 diff=0.0129119
iteration 63
ELB=-393.982 diff=0.0121477
iteration 64
ELB=-393.971 diff=0.0114356
iteration 65
ELB=-393.96 diff=0.0107715
iteration 66
ELB=-393.95 diff=0.0101515
iteration 67
ELB=-393.94 diff=0.00957202
ELB converged

graper documentation built on Nov. 8, 2020, 5:45 p.m.