LearnerRegrKerasDistill: Keras Feed Forward Neural Network for Knowledge Distillation

In pfistfl/distillery: Compress arbitrary mlr3 models into keras models

Description

Internally, samples new observations from the task data using a nonparametric sampling technique (here: Munge (c.f. Caruana et al., 2011 Model Compression). Those observations are then labeled using the supplied 'teacher'; a fitted 'mlr3' learner. In addition to all hyperparameters from 'kerasff', this learner has the following hyperparameters which allow for control over the distillation process: * 'n' :: number of artificial rows to sample * ‘switch_prob' :: probability to switch out each entry with it’s nearest neighbour * 'var' :: For numeric features, a new value is sampled from N(x_nn, |x -x_nn| / var) where x is a feature value, x_nn the nearest neighbour's corresponding feature value. * 'alpha' :: Specifies a convex-combination of labels predicted by the learner (alpha = 0) and labels of the observation before swapping.

Feed Forward Neural Network using Keras and Tensorflow. This learner builds and compiles the keras model from the hyperparameters in 'param_set', and does not require a supplied and compiled model.

Calls [keras::fit] from package keras. Layers are set up as follows: * The inputs are connected to a 'layer_dropout', applying the 'input_dropout'. Afterwards, each 'layer_dense()' is followed by a 'layer_activation', and depending on hyperparameters by a 'layer_batch_normalization' and or a 'layer_dropout' depending on the architecture hyperparameters. This is repeated 'length(layer_units)' times, i.e. one 'dense->activation->batchnorm->dropout' block is appended for each 'layer_unit'. The last layer is either 'softmax' or 'sigmoid' for classification or 'linear' or 'sigmoid' for regression.

Parameters:
Most of the parameters can be obtained from the 'keras' documentation. Some exceptions are documented here. * 'use_embedding': A logical flag, should embeddings be used? Either uses 'make_embedding' (if TRUE) or if set to FALSE 'model.matrix(~. - 1, data)' to convert factor, logical and ordered factors into numeric features. * 'layer_units': An integer vector storing the number of units in each consecutive layer. 'layer_units = c(32L, 32L, 32L)' results in a 3 layer network with 32 neurons in each layer. Can be 'integer(0)', in which case we fit a (multinomial) logistic regression model.

* 'initializer': Weight and bias initializer. “' "glorot_uniform" : initializer_glorot_uniform(seed) "glorot_normal" : initializer_glorot_normal(seed) "he_uniform" : initializer_he_uniform(seed) "..." : see '??keras::initializer' “'

* 'optimizer': Some optimizers and their arguments can be found below.
Inherits from 'tensorflow.python.keras.optimizer_v2'. “' "sgd" : optimizer_sgd(lr, momentum, decay = decay), "rmsprop" : optimizer_rmsprop(lr, rho, decay = decay), "adagrad" : optimizer_adagrad(lr, decay = decay), "adam" : optimizer_adam(lr, beta_1, beta_2, decay = decay), "nadam" : optimizer_nadam(lr, beta_1, beta_2, schedule_decay = decay) “'

* 'regularizer': Regularizer for keras layers: “' "l1" : regularizer_l1(l = 0.01) "l2" : regularizer_l2(l = 0.01) "l1_l2" : regularizer_l1_l2(l1 = 0.01, l2 = 0.01) “'

* 'class_weights': needs to be a named list of class-weights for the different classes numbered from 0 to c-1 (for c classes). “' Example: wts = c(0.5, 1) setNames(as.list(wts), seq_len(length(wts)) - 1) “' * 'callbacks': A list of keras callbacks. See '?callbacks'.

Format

[R6::R6Class()] inheriting from [mlr3keras::LearnerClassifKerasFF].

Construction

“' LearnerregrKerasDistill$new() mlr3::mlr_learners$get("regr.kerasdistill") mlr3::lrn("regr.kerasdistill") “'

Learner Methods

Keras Learners offer several methods for easy access to the stored models.

* '.$plot()'
Plots the history, i.e. the train-validation loss during training. * '.$save(file_path)'
Dumps the model to a provided file_path in 'h5' format. * '.$load_model_from_file(file_path)'
Loads a model saved using 'saved' back into the learner. The model needs to be saved separately when the learner is serialized. In this case, the learner can be restored from this function. Currently not implemented for 'TabNet'.

Super classes

mlr3::Learner -> mlr3::LearnerRegr -> mlr3keras::LearnerRegrKeras -> mlr3keras::LearnerRegrKerasFF -> LearnerRegrKerasDistill

Public fields

teacher

['Learner']
Trained method to distill into a neural network

Methods

Public methods

• LearnerRegrKerasDistill$new()

• LearnerRegrKerasDistill$clone()

Inherited methods

• mlr3::Learner$format()

• mlr3::Learner$help()

• mlr3::Learner$predict()

• mlr3::Learner$predict_newdata()

• mlr3::Learner$print()

• mlr3::Learner$reset()

• mlr3::Learner$train()

• mlr3keras::LearnerRegrKeras$load_model_from_file()

• mlr3keras::LearnerRegrKeras$lr_find()

• mlr3keras::LearnerRegrKeras$plot()

• mlr3keras::LearnerRegrKeras$save()

---

Method new()

Creates a new instance of this [R6][R6::R6Class] class.

Usage

LearnerRegrKerasDistill$new(teacher)

Arguments

teacher

['Learner']
Trained learner to compress into a neural network.

---

Method clone()

The objects of this class are cloneable with this method.

Usage

LearnerRegrKerasDistill$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

See Also

[Dictionary][mlr3misc::Dictionary] of [Learners][mlr3::Learner]: [mlr3::mlr_learners]

Examples

learner = mlr3::lrn("regr.kerasdistill")
print(learner)

# available parameters:
learner$param_set$ids()

pfistfl/distillery documentation built on April 17, 2021, 10 p.m.