In gbonte/gbcode: Code from the handbook "Statistical foundations of machine learning"

Shiny dashboard "Statistical foundations of machine learning"

Parametric identification and validation

Goal: show the relation between parametric identification and optimization

Common left panel:

• Number of samples: number of samples of observed dataset
• Conditional variance $\sigma_w^2$
• Number of bootstrap repetitions: number of resampling repetitions with replacement
• 3D theta: angle defining the azimuthal direction in the 3D visualization
• 3D phi: angle defining the colatitude in the 3D visualization

Linear least-squares

Data generating process: ${\bf y}=\beta_0+\beta_1 x+{\bf w}$ where ${\bf w} \sim {\mathcal N} (0,\sigma_w^2)$.

Top sliders:

• $\beta_0$: intercept parameter
• $\beta_1$: slope parameter
• $\hat{\beta}_0$: estimation intercept parameter
• $\hat{\beta}_1$: estimation slope parameter
• $\eta$: gradient learning rate

Bottom left panel: training data set, regression function (in green) and estimated regression function $h(x)=\hat{\beta}_0+\hat{\beta}_1 x$ (in red)

Bottom right panel: convex empirical risk function $J(\hat{\beta}_0,\hat{\beta}_1)$. Green dot denotes the pair of real parameters $\beta_0,\beta_1$. Red dot denotes the pair of estimations $\hat{\beta}_0,\hat{\beta}_1$

Suggested manipulations:

• by clicking on the button "Gradient step" an iteration of gradient-based minimization is performed: as a consequence the estimated regression function is updated and the red dot changes its location
• change the learning rate $\eta$ and see the impact on the gradient-based iteration
• by reducing the number of samples and/or increasing the conditional variance, see the gap between the real regression line (in green) and the estimated least-squares line (in red)

NNET least-squares

Data generating process: ${\bf y}=\sin (\pi x) + {\bf w}$ where ${\bf w} \sim {\mathcal N} (0,\sigma_w^2)$.

Hypothesis function: $y=w_7 {\mathcal s}(w_2 x+ w_1)+ w_8 {\mathcal s}(w_4 x+ w_3)+w_9 {\mathcal s}(w_6 x+ w_5) +w_{10}$ where ${\mathcal s}(z)=\frac{1.0}{1.0+\exp^{-z}}$ stands for the sigmoid function.

Top sliders:

• # steps: number of gradient iterations made for each click of the "Gradient step NNet" button
• $\eta$: gradient learning rate

Suggested manipulations:

• by clicking on the button "Gradient step" an iteration of gradient-based minimization is performed: as a consequence the estimated regression function is updated
• change the learning rate $\eta$ and see the impact on the gradient-based iteration
• reinitialize the weights of the NNET and see the impact on the gradient-based iteration

KNN cross-validation

Data generating process: ${\bf y}=\sin (\pi x) + {\bf w}$ where ${\bf w} \sim {\mathcal N} (0,\sigma_w^2)$.

Hypothesis function: K nearest-neighbors

• # folds: number of cross-validation folds
• # neghbours: number of neighbors in the KNN algorithm

Bottom left panel: training data set. At each click of the button "CV step" the points that belong to the test fold are put in green and the corresponding prediction is in red

Bottom red panel: it shows for each input sample the associated CV error: the figure is updated as far as we proceed with the cross-validation folds

Suggested manipulations:

• change the number of neighbors and see the impact on resulting CV error
• change the number of folds and see the impact on resulting CV error

gbonte/gbcode documentation built on Feb. 27, 2024, 7:38 a.m.