Source: https://medium.com/dsnet/linear-regression-with-pytorch-3dde91d60b50
Original title: Linear Regression and Gradient Descent from scratch in PyTorch
knitr::opts_chunk$set( collapse = TRUE, comment = "#>" )
library(rTorch)
torch$manual_seed(0) device = torch$device('cpu')
Let's re-implement the same model using some built-in functions and classes from PyTorch.
nn <- torch$nn
# Input (temp, rainfall, humidity) inputs = np$array(list( list(73, 67, 43), list(91, 88, 64), list(87, 134, 58), list(102, 43, 37), list(69, 96, 70), list(73, 67, 43), list(91, 88, 64), list(87, 134, 58), list(102, 43, 37), list(69, 96, 70), list(73, 67, 43), list(91, 88, 64), list(87, 134, 58), list(102, 43, 37), list(69, 96, 70) ), dtype='float32') # Targets (apples, oranges) targets = np$array(list( list(56, 70), list(81, 101), list(119, 133), list(22, 37), list(103, 119), list(56, 70), list(81, 101), list(119, 133), list(22, 37), list(103, 119), list(56, 70), list(81, 101), list(119, 133), list(22, 37), list(103, 119) ), dtype='float32')
torch$set_default_dtype(torch$double)
# Convert inputs and targets to tensors inputs <- torch$from_numpy(inputs) targets <- torch$from_numpy(targets)
We'll create a TensorDataset
, which allows access to rows from inputs and targets as tuples. We'll also create a DataLoader, to split the data into batches while training. It also provides other utilities like shuffling and sampling.
TensorDataset <- torch$utils$data$TensorDataset DataLoader <- torch$utils$data$DataLoader
# Define dataset train_ds = TensorDataset(inputs, targets) train_ds$tensors[1:2]
# Define data loader batch_size = 5L train_dl = DataLoader(train_ds, batch_size, shuffle = TRUE) iter_next(import_builtins()$iter(train_dl))
nn.Linear
Instead of initializing the weights and biases manually, we can define the model using nn.Linear
.
# Define model model = nn$Linear(3L, 2L) print(model$weight) print(model$bias)
Instead of manually manipulating the weights & biases using gradients, we can use the optimizer optim.SGD
.
# Define optimizer opt = torch$optim$SGD(model$parameters(), lr=1e-5)
Instead of defining a loss function manually, we can use the built-in loss function mse_loss
.
# Import nn.functional # in Python: import torch.nn.functional as F F <- torch$nn$functional
# Define loss function loss_fn = F$mse_loss
loss = loss_fn(model(inputs), targets) print(loss)
We are ready to train the model now. We can define a utility function fit which trains the model for a given number of epochs.
fit <- function(num_epochs, model, loss_fn, opt) { for (epoch in 1:num_epochs) { for (xy in iterate(train_dl)) { # Generate predictions xb <- xy[[1]]; yb <- xy[[2]] # print(yb) pred <- model(xb) loss <- loss_fn(pred, yb) # Perform gradient descent loss$backward() opt$step() opt$zero_grad() } } cat('Training loss: ') print(loss_fn(model(inputs), targets)) }
```{python, eval=FALSE}
def fit(num_epochs, model, loss_fn, opt): for epoch in range(num_epochs): for xb,yb in train_dl: # Generate predictions pred = model(xb) loss = loss_fn(pred, yb) # Perform gradient descent loss.backward() opt.step() opt.zero_grad() print('Training loss: ', loss_fn(model(inputs), targets))
```r # Train the model for 100 epochs fit(100, model, loss_fn, opt)
# Generate predictions preds = model(inputs) preds
# Compare with targets
targets
The weights and biases can also be represented as matrices, initialized with random values. The first row of $w$ and the first element of $b$ are used to predict the first target variable, i.e. yield for apples, and, similarly, the second for oranges.
# random numbers for weights and biases. Then convert to double() torch$set_default_dtype(torch$double) w = torch$randn(2L, 3L, requires_grad=TRUE) #$double() b = torch$randn(2L, requires_grad=TRUE) #$double() print(w) print(b)
The model is simply a function that performs a matrix multiplication of the input $x$ and the weights $w$ (transposed), and adds the bias $b$ (replicated for each observation).
model <- function(x) { wt <- w$t() return(torch$add(torch$mm(x, wt), b)) }
The matrix obtained by passing the input data to the model is a set of predictions for the target variables.
# Generate predictions preds = model(inputs) print(preds)
# Compare with targets print(targets)
Because we've started with random weights and biases, the model does not a very good job of predicting the target variables.
We can compare the predictions with the actual targets, using the following method:
The result is a single number, known as the mean squared error (MSE).
# MSE loss mse = function(t1, t2) { diff <- torch$sub(t1, t2) mul <- torch$sum(torch$mul(diff, diff)) return(torch$div(mul, diff$numel())) }
# Compute loss loss = mse(preds, targets) print(loss) # 46194 # 33060.8070
The resulting number is called the loss, because it indicates how bad the model is at predicting the target variables. Lower the loss, better the model.
With PyTorch, we can automatically compute the gradient or derivative of the loss w.r.t. to the weights and biases, because they have requires_grad
set to True.
# Compute gradients loss$backward()
The gradients are stored in the .grad property of the respective tensors.
# Gradients for weights print(w) print(w$grad)
# Gradients for bias print(b) print(b$grad)
A key insight from calculus is that the gradient indicates the rate of change of the loss, or the slope of the loss function w.r.t. the weights and biases.
decreasing the element's value slightly will decrease the loss.
If a gradient element is negative,
The increase or decrease is proportional to the value of the gradient.
Finally, we'll reset the gradients to zero before moving forward, because PyTorch accumulates gradients.
# Reset the gradients w$grad$zero_() b$grad$zero_() print(w$grad) print(b$grad)
We'll reduce the loss and improve our model using the gradient descent algorithm, which has the following steps:
# Generate predictions preds = model(inputs) print(preds)
# Calculate the loss loss = mse(preds, targets) print(loss)
# Compute gradients loss$backward() print(w$grad) print(b$grad)
# Adjust weights and reset gradients with(torch$no_grad(), { print(w); print(b) # requires_grad attribute remains w$data <- torch$sub(w$data, torch$mul(w$grad$data, torch$scalar_tensor(1e-5))) b$data <- torch$sub(b$data, torch$mul(b$grad$data, torch$scalar_tensor(1e-5))) print(w$grad$data$zero_()) print(b$grad$data$zero_()) }) print(w) print(b)
With the new weights and biases, the model should have a lower loss.
# Calculate loss preds = model(inputs) loss = mse(preds, targets) print(loss)
To reduce the loss further, we repeat the process of adjusting the weights and biases using the gradients multiple times. Each iteration is called an epoch.
# Running all together # Adjust weights and reset gradients for (i in 1:100) { preds = model(inputs) loss = mse(preds, targets) loss$backward() with(torch$no_grad(), { w$data <- torch$sub(w$data, torch$mul(w$grad, torch$scalar_tensor(1e-5))) b$data <- torch$sub(b$data, torch$mul(b$grad, torch$scalar_tensor(1e-5))) w$grad$zero_() b$grad$zero_() }) } # Calculate loss preds = model(inputs) loss = mse(preds, targets) print(loss) # predictions preds # Targets targets
Any scripts or data that you put into this service are public.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.