In nixgank-wang/bis557: What the Package Does (One Line, Title Case)

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)

library(bis557)

Problem 1

We mentioned that the Hessian matrix in Equation 5.19 can be more ill- conditioned than the matrix XtX itself. Generate a matrix X and propabilities p such that the linear Hessian (XtX) is well-conditioned but the logistic variation is not. Build up a linear Hessian that is well-conditioned, we make sure the condition number is small. When the probabilities are close to 0 or 1, the logistic variation is ill-conditoned.

X<- matrix(rnorm(100),20)
solve(t(X)%*%X)
#set vector p to be close to 0
p<- c(1e-3,1e-4,1e-5,1e-6,1e-7)
solve(X%*%diag(p*(1-p))%*%t(X))

Problem 2

Describe and implement a first-order solution for the GLM maximum likelihood problem using only gradient information, avoiding the Hessian matrix. Include both a constant step size along with an adaptive one. You may use a standard adaptive update Momentum, Nesterov, AdaGrad, Adam, or your own.

library(tidyverse)
#create demo dataset
pd <- tibble(offers=c(rep(0,50),rep(1,30),rep(2,10),rep(3,7),rep(4,3)),
division=sample(c("A","B","C"),100,replace = TRUE),
exam=c(runif(50,60,80),runif(30,65,95),runif(20,75,95)))
y<- matrix(pd$offers,ncol = 1)
X<- model.matrix(offers~division+exam,data=pd)
#implement first-order solution for GLM with constant step size
constant<-gdConstantGLM(X,y,mu_fun=function(eta) exp(eta),lambda = 1e-3)$beta
#implement first-order solution for GLM with adaptive update by momentum gradient descent algorithm
momentum<-gdMomentum_GLM(X,y,mu_fun=function(eta) exp(eta),lambda = 1e-3)$beta
(compare<-cbind(constant,momentum))

Problem 3

Describe and implement a classification model generalizing logistic regression to accommodate more than two classes.

library(palmerpenguins)
data(penguins)
form = is_gentoo ~ body_mass_g +bill_length_mm
# create dataset
my_penguins<- penguins %>% mutate(is_gentoo = as.numeric(species == "Gentoo")) %>% model.frame(form,.)
#apply multinom_logit function to see the performance of classification
multinom_logit(form=form,data=my_penguins)