SlrArd: Sparse Logistic Regression

In sprsmdl: Sparse modeling toolkit

Description

Sparse logistic regression (SLR) with automatic relevance determination (ARD).

Usage

SlrArd(T, X, bias = TRUE, method = c("VB", "VBMacKay", "PX-VB"), control = list(), check.lb=TRUE, a0 = 0, b0 = 0, mu0 = 0, xi0 = 1)

Arguments

T	A vector of binary categorical dependent variable.
X	A matrix, each row of which is a vector of independent variables.
bias	A logical value specifying whether to include intercept or not.
method	The method to be used. See ‘Details’.
control	A list of control parameters. See ‘Details’.
check.lb	A logical value whether to check the validity of parameter updates. Defaults to TRUE.

Hyper parameters:

a0	Shape of Gamma distribution.
b0	Rate of Gamma distribution.

Initial values:

mu0	Coefficients.
xi0	Variational parameter.

Details

Some independent variables can be pruned in each VB-EM iteration and never be used for successive iterations when irrelevance of the variables exceed a threshold determined by control$pruning. Explicitly set control$pruning = Inf to prevent the pruning.

Method ‘"VB"’ is a variational Bayesian method that is robust. See Bishop (2006).

Method ‘"VBMacKay"’ is basically same as ‘"VB"’ but some parameters are updated based on method of MacKay (1992). Global convergence has not been proven but this algorithm will be faster.

Method ‘"PX-VB"’ based on the Parameter eXpanded VB method proposed in Qi and Jaakkola (2007). Global convergence is proven and could be faster than ‘"VB"’.

The control argument is a list that can supply any of the following components:

• pruning: threshold of independent variables pruning. No variables are pruned if ‘Inf’. Defaults to ‘1e+8’.

See ?optim for meanings of the following control parameters.

• maxit: Defaults to ‘10^5’.

• reltoj: Defaults to ‘sqrt(.Machine$double.eps)’.

• trace: Defaults to ‘TRUE’.

• REPORT: Defaults to ‘floor(control$maxit / 20)’.

Value

coefficients	A named vector of coefficients.
irrelevance	A named vector of irrelevance.
iterations	A number of iterations.
converged	A logical value giving whether the iterations converged or not.
lower.bound	The lower bound of the marginal log-likelihood minus a constant term.
method	The method used.
lb.diff	The difference of the lower bounds in each update. This exists when check.lb = TRUE.
fitted.values	The fitted values.
residuals	The residuals, that is T minus fitted values.

Author(s)

Hiroshi Saito [e-mail]

References

Bishop, C. M. (2006) Pattern recognition and machine learning. Springer.

MacKay, D. J. C. (1992) Bayesian interpolation. _Neural Computation_, *4*(3), 415-447.

Qi, Y. and Jaakkola, T. S. (2007) Parameter expanded variational Bayesian methods. _Advances in Neural Information Processing Systems_, *19*, 1097.

Examples

data(iris)

tmp <- iris[iris$Species != 'versicolor',]
T <- tmp$Species == 'setosa'
X <- as.matrix(tmp[,1:4])

res <- SlrArd(T, X, bias=TRUE, method="VB", control = list(maxit=500))
print(coefficients(res))

res <- SlrArd(T, X, bias=TRUE, method="VBMacKay")  ## faster
print(coefficients(res))

res <- SlrArd(T, X, bias=FALSE, method="VBMacKay")  ## without bias
print(coefficients(res))

Example output

[1] "Iter#25: 5 parameters remain, lower bound = 8.508883 + const."
[1] "Iter#50: 5 parameters remain, lower bound = 10.807418 + const."
[1] "Iter#75: 5 parameters remain, lower bound = 11.968423 + const."
[1] "Iter#100: 5 parameters remain, lower bound = 12.761647 + const."
[1] "Iter#125: 5 parameters remain, lower bound = 13.371518 + const."
[1] "Iter#150: 5 parameters remain, lower bound = 13.870521 + const."
[1] "Iter#175: 5 parameters remain, lower bound = 14.294654 + const."
[1] "Iter#200: 5 parameters remain, lower bound = 14.664487 + const."
[1] "Iter#225: 5 parameters remain, lower bound = 14.992928 + const."
[1] "Iter#250: 5 parameters remain, lower bound = 15.288643 + const."
[1] "Iter#275: 5 parameters remain, lower bound = 15.557755 + const."
[1] "Iter#300: 5 parameters remain, lower bound = 15.804763 + const."
[1] "Iter#325: 5 parameters remain, lower bound = 16.033084 + const."
[1] "Iter#350: 5 parameters remain, lower bound = 16.245377 + const."
[1] "Iter#375: 5 parameters remain, lower bound = 16.443759 + const."
[1] "Iter#400: 5 parameters remain, lower bound = 16.629945 + const."
[1] "Iter#425: 5 parameters remain, lower bound = 16.805346 + const."
[1] "Iter#450: 5 parameters remain, lower bound = 16.971141 + const."
[1] "Iter#475: 5 parameters remain, lower bound = 17.128320 + const."
[1] "Iter#500: 5 parameters remain, lower bound = 17.277732 + const."
         Bias  Sepal.Length   Sepal.Width  Petal.Length   Petal.Width 
 0.0023440502  0.0004596858  3.2740933316 -3.1478529806 -0.0011116445 
[1] "Iter#18: pruned 2"
[1] "Iter#21: pruned 1"
[1] "Iter#22: pruned 5"
Warning message:
In SlrArd(T, X, bias = TRUE, method = "VBMacKay") :
  Sometimes the lower bound decreases. Check lb.diff.
        Bias Sepal.Length  Sepal.Width Petal.Length  Petal.Width 
    0.000000     0.000000     3.274797    -3.147309     0.000000 
[1] "Iter#19: pruned 1"
[1] "Iter#22: pruned 4"
Warning message:
In SlrArd(T, X, bias = FALSE, method = "VBMacKay") :
  Sometimes the lower bound decreases. Check lb.diff.
Sepal.Length  Sepal.Width Petal.Length  Petal.Width 
    0.000000     3.274797    -3.147309     0.000000 

sprsmdl documentation built on May 1, 2019, 11:31 p.m.