In albhasan/sits.prodes: What the Package Does (One Line, Title Case)

knitr::opts_chunk$set(
collapse = TRUE,
comment = "#>"
)

library(dplyr)
library(ggplot2)
library(kableExtra)
library(knitr)
library(purrr)
library(tidyr)
library(sits.prodes)
library(xtable)

Training

Experiment summary

data("training_logs", package = "sits.prodes")
na_remover <- function(x){return(trimws(gsub("NA", "", x[!is.na(x)])))}
training_logs %>% tidyr::unnest(setup) %>%
    dplyr::mutate(bands = na_remover(paste(Bands_experiment, Bands__experiment))) %>%
    dplyr::mutate(labels = na_remover(paste(Labels, Labels_experiment))) %>%
    dplyr::select(experiment, Clasification_type, labels, bands, Scenes_experiment) %>%
    knitr::kable() %>%
    kableExtra::kable_styling(full_width = F)

Training results

BLUE is accuracy
BLACK is validation accuracy

plot_tb <- training_logs %>% tidyr::unnest(trains) %>% 
    dplyr::select(experiment, model_name, acc, loss, val_acc, val_loss) %>%
    dplyr::mutate(acc  = purrr::map(acc,  function(x) eval(parse(text = x))),
                  loss = purrr::map(loss, function(x) eval(parse(text = x))),
                  val_acc  = purrr::map(val_acc, function(x) eval(parse(text = x))),
                  val_loss = purrr::map(val_loss, function(x) eval(parse(text = x))), 
                  val_acc_mean = purrr::map_dbl(val_acc, mean)) 

#plot_tb %>% dplyr::select(experiment, model_name, val_acc_mean) %>%
#    knitr::kable() %>% kableExtra::kable_styling(full_width = F)


# plot_ls <- lapply(unique(plot_tb$experiment), function(exp){
#   plot_tb %>% dplyr::filter(experiment %in% exp) %>% tidyr::unnest() %>%
#     dplyr::group_by(model_name) %>% dplyr::mutate(epoch = row_number()) %>% 
#     ggplot2::ggplot() +
#     ggplot2::geom_path(aes(y = acc, x = epoch), colour = "blue") +
#     ggplot2::geom_path(ggplot2::aes(y = val_acc, x = epoch)) +
#     ggplot2::ggtitle(exp) +
#     ggplot2::ylim(0.95, 1.0) +
#     #ggplot2::theme(plot.title = element_text(size = 8)) +
#     ggplot2::labs(x = "Epochs", y = "Accuracy") + 
#     ggplot2::facet_wrap(~ model_name, nrow = 5) %>%
#     return()
# })


plot_ls <- lapply(unique(plot_tb$experiment), function(exp){
    plot_tb %>% 
        dplyr::filter(experiment %in% exp) %>% 
        tidyr::unnest() %>%
        dplyr::group_by(model_name) %>% 
        dplyr::mutate(epoch = row_number()) %>%
        ggplot2::ggplot() +
        ggplot2::geom_path(aes(y = loss, x = epoch), colour = "blue") +
        ggplot2::geom_path(ggplot2::aes(y = val_loss, x = epoch)) +
        ggplot2::ggtitle(exp) +
        ggplot2::ylim(0.0, 0.25) +
        #ggplot2::theme(plot.title = element_text(size = 8)) +
        ggplot2::labs(x = "Epochs", y = "Loss") +
        ggplot2::facet_wrap(~ model_name, nrow = 5) %>%
        return()
})



for(p in plot_ls)
    suppressWarnings(print(p))

selected trainings

train_20_model_1
train_21_model_12
train_30_model_18
train_31_model_5
train_32_model_10
train_33_model_19
train_34_model_6
train_35_model_8
train_40_model_17
train_41_model_2
train_42_model_2
train_50_model_2
train_51_model_14
train_52_model_13

Bias and variance

One way to understand overfitting is by decomposing generalization error into bias and variance. Bias is a learner’s tendency to consistently learn the same wrong thing. Variance is the tendency to learn random things irrespective of irrespective of the real signal - [@Domingos:2012]

High bias: High crossvalidation error using a low degree polynomial +There is a high error but training and validation errors are similar.
High variance: High crossvalidation error using a high degree polynomial
High variance: There is a big difference between training error and crossvalidation error
Classification with a HIGH BIAS problem (underfit)
Training set error is high
Crossvalidation error is close to the trainng error
If a leraning algorithm is suffering from high bias, getting more training data will not (by itself) help much
Classification with a HIGH VARAINCE problem (overfit)
Training set error is low
Crossvalidation error is much bigger than the training set error
If a leraning algorithm is suffering from high variance getting more training data is likely to help
Fix HIGH BIAS (underfit)
Try getting additional features (curretn hyppothesis is too simple)
Try adding polynomial features
Try decreasing regularization (cost function)
Fix HIGH VARIANCE (overfit)
Get more training examples
Try smaller set of features (the attributes of each sample)
Try increasing regularization (keep the features but reduce the magnitud/values of the parameters. Works well if there are a lot of features)

Source: Machine Learning — Andrew Ng, Stanford University [FULL COURSE]

The loss function takes the predictions of the network and the true target (what you wanted the network to output) and computes a distance score, capturing how well the network has done on this specific example. - [@Chollet:2018]

Two quantities are displayed during training: the loss of the network over the training data, and the accuracy of the network over the training data. - [@Chollet:2018]

[The] gap between training accuracy and test accuracy is an example of overfitting - [@Chollet:2018]