iucnn_modeltest: Model-Testing IUCNN Models using Cross-Validation...

In azizka/IUCNN: Neural Networks to Approximate IUCN Red List Conservation Assessments

Model-Testing IUCNN Models using Cross-Validation (Hyperparameter-Tuning)

Description

Takes as input features produced with iucnn_prepare_features and labels produced with iucnn_prepare_labels, as well as a path to a log-file where results for each tested model will be stored. All available options are identical to the iucnn_train_model function and can be provided as vectors, e.g. dropout_rate = c(0.0,0.1,0.3) and n_layers = c('30','40_20','50_30_10'). iucnn_modeltest will then iterate through all possible permutations of the provided hyperparameter settings, train a separate model for each hyperparameter combination, and store the results in the provided log-file.

Usage

iucnn_modeltest(
  x,
  lab,
  logfile = tempfile(),
  model_outpath = paste0(tempdir(), "/a"),
  mode = "nn-class",
  cv_fold = 1,
  test_fraction = 0.2,
  n_layers = c("50_30_10", "30"),
  dropout_rate = c(0, 0.1, 0.3),
  use_bias = TRUE,
  balance_classes = FALSE,
  seed = 1234,
  label_stretch_factor = 1,
  label_noise_factor = 0,
  act_f = "relu",
  act_f_out = "auto",
  max_epochs = 5000,
  patience = 200,
  mc_dropout = TRUE,
  mc_dropout_reps = 100,
  randomize_instances = TRUE,
  rescale_features = FALSE,
  init_logfile = TRUE,
  recycle_settings = FALSE
)

Arguments

x	a data.set, containing a column "species" with the species names, and subsequent columns with different features.
lab	an object of the class iucnn_labels, as generated by iucnn_prepare_labels containing the labels for all species.
logfile	character string. Define the filepath/name for the output log-file.
model_outpath	the path where to save the results on disk
mode	character string. Choose between the IUCNN models "nn-class" (default, tensorflow neural network classifier), "nn-reg" (tensorflow neural network regression), or "bnn-class" (Bayesian neural network classifier)
cv_fold	integer (default=1). When setting cv_fold > 1, iucnn_train_model will perform k-fold cross-validation. In this case, the provided setting for test_fraction will be ignored, as the test size of each CV-fold is determined by the specified number provided here.
test_fraction	numeric. The fraction of the input data used as test set.
n_layers	character string. Define number node per layer by providing a character string where the number of nodes for each layer are separated by underscores. E.g. '50_30_10' (default) will train a model with 3 hidden layers with 50, 30, and 10 nodes respectively. Note that the number of nodes in the output layer is automatically determined based on the number of unique labels in the training set.
dropout_rate	numeric. This will randomly turn off the specified fraction of nodes of the neural network during each epoch of training making the NN more stable and less reliant on individual nodes/weights, which can prevent over-fitting (only available for modes nn-class and nn-reg). See mc_dropout setting explained below if dropout shall also be applied to the predictions.
use_bias	logical (default=TRUE). Specifies if a bias node is used in the first hidden layer.
balance_classes	logical (default=FALSE). If set to TRUE, iucnn_train_model will perform supersampling of the training instances to account for uneven class distribution in the training data. In case of training an bnn-class model, choosing this option will add the estimation of class weights instead, to account for class imbalances.
seed	integer. Set a starting seed for reproducibility.
label_stretch_factor	numeric (only for mode nn-reg). The provided value will be applied as a factor to stretch or compress the labels before training a regression model. A factor smaller < 1.0 will compress the range of labels, while a factor > 1 will stretch the range.
label_noise_factor	numeric (only for mode nn-reg). Add specified amount of random noise to the input labels to give the categorical labels a more continuous spread before training the regression model. E.g. a value of 0.2 will redraw a label of a species categorized as Vulnerable (class=2) randomly between 1.8 and 2.2, based on a uniform probability distribution.
act_f	character string. Specifies the activation function should be used in the hidden layers. Available options are: "relu", "tanh", "sigmoid", or "swish" (latter only for bnn-class). If set to 'auto' (default), iucnn_train_model will pick a reasonable default ('relu' for nn-class or nn-reg, and 'swish' for bnn-class).
act_f_out	character string. Similar to act_f, this specifies the activation function for the output layer. Available options are "softmax" (nn-class, bnn-class), "tanh" (nn-reg), "sigmoid" (nn-reg), or no activation function "" (nn-reg). When set to "auto" (default), a suitable output activation function will be chosen based on the chosen mode ('softmax' for nn-class or bnn-class, 'tanh' for nn-reg).
max_epochs	integer. The maximum number of epochs.
patience	integer. Number of epochs with no improvement after which training will be stopped.
mc_dropout	logical. If set to TRUE, the predictions (including the validation accuracy) based on a model trained with a dropout fraction > 0 will reflect the stochasticity introduced by the dropout method (MC dropout predictions). This is e.g. required when wanting to predict with a specified accuracy threshold (see target_acc option in iucnn_predict_status). This option is activated by default when chosing a dropout_rate > 0, unless it is manually set to FALSE here.
mc_dropout_reps	integer. The number of MC iterations to run when predicting validation accuracy and calculating the accuracy-threshold table required for making predictions with an accuracy threshold. The default of 100 is usually sufficient, larger values will lead to longer computation times, particularly during model testing with cross-validation.
randomize_instances	logical. When set to TRUE (default) the instances will be shuffled before training (recommended).
rescale_features	logical. Set to TRUE if all feature values shall be rescaled to values between 0 and 1 prior to training (default=FALSE).
init_logfile	logical (default=TRUE). If set to TRUE, modeltest_iucnn will attempt to initiate a new log-file under the provided path, possibly overwriting already existing model-testing results stored in the same location. Set to FALSE if instead you want to append to an already existing log-file.
recycle_settings	logical (default=FALSE). If set to TRUE, iucnn_modeltest will read the log-file stored at the path specified under the "logfile" argument and run model-testing for the input features and labels using the same models stored in that file. This setting can be useful when e.g. wanting to test the same models for different sets of input data.

Value

outputs a data.frame object containing stats and settings of all tested models.

Note

See vignette("Approximate_IUCN_Red_List_assessments_with_IUCNN") for a tutorial on how to run IUCNN.

Examples

## Not run: 
data("training_occ") #geographic occurrences of species with IUCN assessment
data("training_labels")# the corresponding IUCN assessments

# 1. Feature and label preparation
features <- iucnn_prepare_features(training_occ, type = "geographic") # Training features
labels_train <- iucnn_prepare_labels(training_labels, features) # Training labels


# Model-testing
mod_test <- iucnn_modeltest(x = features,
                            lab = labels_train,
                            mode = "nn-class",
                            dropout_rate = c(0.0, 0.1, 0.3),
                            n_layers = c("30", "40_20", "50_30_10"),
                            cv_fold = 2,
                            init_logfile = TRUE)

## End(Not run)

azizka/IUCNN documentation built on March 29, 2024, 9:38 a.m.