In afuergut/lib7af: Reference Class for Linear and Ridge Regression

library(caret)
library(mlbench)
library(leaps)
library(lib7af)
#library(MASS)
seed <- 111
set.seed(seed)
tracon <- caret::trainControl("cv")
data("BostonHousing")

Introduction

In lib7af, we created a package with a function called ridgereg. This function is able to take a formula object, a dataset and a parameter lambda $\lambda$. The situation when a ridge regression is intended to be used, is when p>n or when multikollinearity is present. Different scalings of the covariates in X will affect results. That is why we finetune the the model to best fit the lambda parameters. Before we do so, we must normalize all covariates so the analysis can be done.

QR decomposition is used to calculate the regression coefficients.

Create training and test data

Create are partition of the dataset. 70% for the training data and 30% for the test data.

tr_data_index<-createDataPartition(BostonHousing$medv, p=0.7, times= 1, list= FALSE)
tr_data<-BostonHousing[tr_data_index,]
te_data<-BostonHousing[-tr_data_index,]

Fitting and evaluating the model

The model is fitted - one time with linear regression and one using forward selection.

l_n_m<-train(medv~., tr_data, method="lm", trControl = tracon)
l_n_m
l_n_M<-train(medv~., tr_data, method="leapForward", trControl = tracon) 
l_n_M

For the forward selection, the RMSE was used to select the optimal model. Comparing the linear model and the model selected with forward regression shows that the linear model has a lower RMSE. A lower RMSE value suggests a better fit and $R^2$ indicates how much data is represented by the model.

l_n_m$results$RMSE
l_n_M$results$RMSE

Finding the best $\lambda$

# Acknowledgement to Eric Herwin and Albin Vasterlund
ridge_example<- list(type = "Regression",
              library ="lib7af",
              loop= NULL,
              prob=NULL)

ridge_example$parameters<-data.frame(parameter="lambda",
                            class="numeric",
                            label="Ridge Regression")

ridge_example$grid<-function(x,y,len=NULL, search="grid"){
data.frame(lambda=lambda)
}
ridge_example$fit <- function (x, y, wts, param, lev, last, classProbs, ...) {
    d_rid <- if (is.data.frame(x))
    x
    else as.data.frame(x)
    d_rid$.outcome<-y
    results<-ridgereg(.outcome ~ ., data=d_rid ,lambda = param$lambda, ...)

    return(results)
  }
  ridge_example$predict<-function (modelFit, newdata, submodels = NULL) {
    if (!is.data.frame(newdata))
      newdata <- as.data.frame(newdata)
    newdata <- scale(newdata)
    return(modelFit$predict(newdata))
  }

set.seed(seed)
result<-c()
for(lambda in seq(0, 20, by = 1)) {
  zwes <- train(medv~., data = tr_data, ridge_example)
  result[lambda] <- zwes$results$RMSE
}
lambda <- which.min(result) / 10
RMSE <- result[which.min(result)]
lambda
RMSE

We can see, that the best $\lambda$ is 1.1 and the RMSE is 23.1903.

Finding the best hyperparameter value for $\lambda$ using 10-folf cross-validation on the training set.

set.seed(seed)
count<-10
l_val<-seq(0.1,by=.1)
fit_control<-trainControl(method="repeatedcv",
                                  number=count,
                                  repeats= count)
ridge_example<-train(medv~.,
                    data=te_data,
                    method=ridge_example,
                    trControl= fit_control)
ridge_example

Evaluation

lin_mod <- predict(l_n_m, te_data)
postResample(lin_mod,te_data$medv)

lin_mod_with <- predict(l_n_M, te_data)
postResample(lin_mod_with,te_data$medv)

ridge_ex <- predict(ridge_example, te_data)
postResample(ridge_ex,te_data$medv)

Even though the model fit by ridge regression has the highest $R^2$, the RMSE is really high. Therefore the linear models seem to be more suitable here.

afuergut/lib7af documentation built on May 28, 2019, 4:42 p.m.