R/document_control.R

In multiRL: Reinforcement Learning Tools for Multi-Armed Bandit

#' @title Controls of Estimation Methods
#' @name control
#' @description
#'
#'  The \code{control} argument is a mandatory list used to customize and manage
#'    various aspects of the iterative process, covering everything from
#'    optimization settings to model configuration.
#'
#' @section Class:
#' \code{control [List]}
#'
#' @section Note:
#'  Different estimation methods require different slots. However, there is no
#'    need to worry if you set unnecessary slots, as this will not affect the
#'    execution.
#'
#' @section 0. General:
#' \itemize{
#'
#'    \item \code{seed [int]}
#'
#'          The random seed controls the reproducibility of each iteration.
#'          Specifically, it determines how the algorithm package generates
#'          "random" input parameters when searching for the optimal parameters.
#'          Fixing the seed ensures that the optimal parameters found are the
#'          same in every run. The default value is \code{123}.
#'
#'    \item \code{core [int]}
#'
#'          Since the parameter fitting process for individual subjects is
#'          independent, this procedure can be accelerated using CPU
#'          parallelism. This argument specifies the number of subjects to
#'          be fitted simultaneously (the number of parallel threads),
#'          with the default set to \code{1}. If the user wishes to speed up the
#'          fitting, they can increase the number of cores appropriately based
#'          on their system specifications.
#'
#'    \item \code{sample [int]}
#'
#'          This parameter denotes the quantity of simulated data generated
#'          during the parameter recovery process.
#'
#'    \item \code{dash [Numeric]}
#'
#'          To prevent the optimal parameter estimates from converging to
#'          boundary values when the number of iterations is insufficient, a
#'          small value is added to the lower bound and subtracted from the
#'          upper bound.
#'
#'          For instance, if the input parameter bounds are \code{(0, 1)},
#'          the actual bounds used for fitting will be \code{[0.00001, 0.99999]}.
#'          This design prevents the occurrence of Infinite values.
#'
#' }
#'
#' @section 1. Likelihood Based Inference (LBI):
#' \itemize{
#'
#'    \item \code{algorithm [Character]}
#'
#'          The package supports the following eight optimization packages for
#'          finding the optimal values of the model's free parameters.
#'          \enumerate{
#'             \item L-BFGS-B (from \code{stats::optim})
#'             \item Simulated Annealing (\code{GenSA::GenSA})
#'             \item Genetic Algorithm (\code{GA::ga})
#'             \item Differential Evolution (\code{DEoptim::DEoptim})
#'             \item Bayesian Optimization (\code{mlrMBO::mbo})
#'             \item Particle Swarm Optimization (\code{pso::psoptim})
#'             \item Covariance Matrix Adapting Evolutionary Strategy (\code{cmaes::cma_es})
#'             \item Nonlinear Optimization (\code{nloptr::nloptr})
#'          }
#'
#'    \item \code{pars [NumericVector]}
#'
#'          Some algorithms require the specification of initial iteration
#'          values. If this value is left as the default NA, the iteration will
#'          commence with an initial value set to the lower bound of the
#'          estimate plus \code{0.01}.
#'
#'    \item \code{size [int]}
#'
#'          Some algorithms, such as Genetic Algorithms (\code{GA}), require the
#'          specification of initial population values. For the definition of
#'          the population, users may refer to the relevant documentation on
#'          evolutionary algorithms. The default value is consistent with the
#'          standard default in \code{GA}, which is \code{50}.
#'
#' }
#'
#' \subsection{1.1 Maximum Likelihood Estimation (MLE)}{
#'
#' \itemize{
#'
#'    \item \code{iter [int]}
#'
#'          This parameter defines the maximum number of iterations. The
#'          iterative process will stop when this value is reached. The default
#'          value is \code{10}. It is recommended that you set this value to at
#'          least \code{100} for formal fitting procedures.
#'
#' }
#' }
#'
#' \subsection{1.2 Maximum A Posteriori (MAP)}{
#'
#' \itemize{
#'
#'    \item \code{iter [int]}
#'
#'          You can input a numeric vector of length 2. The first element
#'          specifies the number of iterations per algorithm call. The second
#'          element determines the total number of EM-MAP executions across all
#'          participants. In other words, if the first element matches your
#'          MLE settings, the second element represents the computational
#'          fold-change of MAP relative to MLE.
#'
#'    \item \code{diff [double]}
#'
#'          In the Expectation-Maximization with Maximum A Posteriori algorithm
#'          (EM-MAP), after estimating the optimal parameters for all subjects
#'          in each iteration, the posterior distribution of each free
#'          parameter is calculated, followed by continuous refinement of the
#'          prior distribution. The process stops when the change in the
#'          log-posterior value is less than the \code{diff}, which defaults
#'          to \code{0.001}.
#'
#'    \item \code{patience [int]}
#'
#'          Given that the Expectation-Maximization with Maximum A Posteriori
#'          (EM-MAP) process can be time-consuming and often encounters
#'          non-convergence issues-for instance, when the log-posterior
#'          oscillates around a certain value-the patience parameter is used to
#'          manage early termination.Specifically, patience is incremented by
#'          1 when the current result is better than the best previous result,
#'          and decremented by 1 when it is worse. The iteration is prematurely
#'          terminated when the patience count reaches zero.
#'
#' }
#' }
#'
#' @section 2. Simulation Based Inference (SBI):
#' \itemize{
#'
#'    \item \code{train [int]}
#'
#'          This parameter is used to specify the quantity of simulated data
#'          utilized when training the Approximate Bayesian Computation (ABC) or
#'          Recurrent Neural Network (RNN) models.
#'
#'    \item \code{scope [Character]}
#'
#'          This parameter can be defined as \code{individual} or \code{shared}.
#'          The former indicates that a separate Approximate Bayesian
#'          Computation (ABC) or Recurrent Neural Network (RNN) model is
#'          trained for each dataset, while the latter means that only one
#'          Approximate Bayesian Computation (ABC) or Recurrent Neural Network
#'          (RNN) model is trained and shared across all datasets. In the
#'          context of the \code{rcv_d} function, the default setting is
#'          \code{"shared"}, whereas in \code{fit_p}, the default is
#'          \code{"individual"}.
#'
#' }
#'
#' \subsection{2.1 Approximate Bayesian Computation (ABC)}{
#'
#' \itemize{
#'
#'    \item \code{tol [double]}
#'
#'          This parameter, aka tolerance, controls how strict the Approximate
#'          Bayesian Computation (ABC) algorithm is when selecting good
#'          simulated data. It sets the acceptance rate. For example, setting
#'          \code{tol = 0.1} (the default) means only the 10 percent of
#'          simulated data that is closest to your actual data is used.
#'
#'    \item \code{reduction [Character]}
#'
#'          Specifies the dimension reduction method for summary statistics.
#'          In ABC, high-dimensional summary statistics often lead to the
#'          "curse of dimensionality," where the algorithm struggles to find a
#'          solution or suffers from extremely slow convergence. Reducing the
#'          dimensions (compressing the data) helps retain the "fingerprint"
#'          of the original data while removing noise, ensuring the program
#'          can efficiently identify the underlying parameters.
#'
#'          \itemize{
#'            \item \code{NULL}: No compression is applied. This is suitable for
#'            smaller datasets where the total number of features (e.g., 
#'            blocks * responses) is relatively low (typically less than 200).
#'            
#'            \item \code{"PLS"} (Partial Least Squares): A supervised reduction 
#'            method that compresses the summary statistics into a space with 
#'            dimensions equal to the number users set (as default, it is equal 
#'            to the number of blocks).
#'            
#'            \item \code{"PCA"} (Principal Component Analysis): An unsupervised 
#'            reduction method that compresses the information into a space with 
#'            dimensions equal to users set (as default, it is equal to the 
#'            number of blocks).
#' }
#'
#'    \item \code{ncomp [int]}
#'
#'          The number of components represents the quantity of information
#'          after compression. By default, this value is equal to the number
#'          of blocks. Since the summary statistics consist of the selection
#'          ratios for each action within each block, an excessive number of
#'          blocks or available actions can lead to high-dimensional
#'          information, making it difficult for the ABC to converge on a
#'          solution. In such cases, PLS or PCA can be selected for
#'          dimensionality reduction.
#'
#'    \item \code{metric [Character]}
#'
#'          Specifies the statistical metric used to determine the best
#'          estimated parameter from the posterior distribution. By default,
#'          this is set to \code{"mode"}, which uses the mode of the accepted
#'          simulated parameters as the best estimate. Users can also change
#'          this to \code{"mean"} or \code{"median"} to use the average or
#'          the median value, respectively.
#' }
#' }
#'
#' \subsection{2.2 Recurrent Neural Network (RNN)}{
#'
#' \itemize{
#'
#'    \item \code{layer [Character]}
#'
#'          Recurrent Neural Networks (RNNs) are neural networks where the
#'          sequence order is meaningful. Currently, the package supports the
#'          following types of recurrent layers:
#'          \itemize{
#'              \item \code{"RNN"} (Simple Recurrent Neural Network):
#'              \itemize{
#'                  \item \code{"RNN"} (Simple Recurrent Neural Network):
#'                  \item \code{"BiRNN"} (Bidirectional SimpleRNN):
#'              }
#'              \item Gated Recurrent Unit (GRU) and Bidirectional GRU (BiGRU):
#'              \itemize{
#'                  \item \code{"GRU"} (Gated Recurrent Unit):
#'                  \item \code{"BiGRU"} (Bidirectional GRU):
#'              }
#'              \item Long Short-Term Memory (LSTM) and Bidirectional LSTM (BiLSTM):
#'              \itemize{
#'                  \item \code{"LSTM"} (Long Short-Term Memory):
#'                  \item \code{"BiLSTM"} (Bidirectional LSTM):
#'              }
#'          }
#'
#'    \item \code{loss [Character]}
#'
#'          Specifies the loss function used to train the Recurrent Neural
#'          Network (RNN). The choice of loss function depends on the nature of
#'          the prediction task and the desired properties of the estimated
#'          parameters.
#'
#'          \itemize{
#'              \item \code{"MSE"} (Mean Squared Error):
#'                  A common loss function that measures the average squared
#'                  difference between predicted and actual values. It is
#'                  sensitive to outliers.
#'              \item \code{"MAE"} (Mean Absolute Error):
#'                  Measures the average absolute difference between predicted
#'                  and actual values. It is more robust to outliers than MSE.
#'              \item \code{"HBR"} (Huber Loss):
#'                  A combination of MSE and MAE, acting as MSE for small
#'                  errors and MAE for large errors. It is less sensitive to
#'                  outliers than MSE while being smoother than MAE.
#'              \item \code{"NLL"} (Negative Log-Likelihood):
#'                  Used for probabilistic predictions where the model outputs
#'                  both the mean and variance of a Gaussian distribution,
#'                  aiming to maximize the likelihood of the observed data.
#'              \item \code{"QRL"} (Quantile Regression Loss):
#'                  Allows the model to predict specific quantiles
#'                  (e.g., 0.05, 0.50, 0.95) of the target distribution,
#'                  rather than just the mean.
#'              \item \code{"MDN"} (Mixture Density Network):
#'                  Enables the model to predict a mixture of probability
#'                  distributions, useful for capturing complex, multimodal,
#'                  or uncertain posterior distributions of parameters.
#'          }
#'
#'    \item \code{info [CharacterVector]}
#'
#'          The Recurrent Neural Network (RNN) needs to find the mapping
#'          relationship between the dataset and the free parameters. To
#'          minimize the time required for this process, we should only include
#'          useful information in the input dataset. The \code{info} parameter
#'          accepts a character vector which represents the amount of
#'          information (i.e., the specific columns) you deem necessary for
#'          training the Recurrent Neural Network (RNN) model. By default, only
#'          the \code{colnames$object} and \code{colnames$action} columns are
#'          included as input.
#'
#'    \item \code{units [int]}
#'
#'          The number of neurons (or units) in the Recurrent Layer
#'          (RNN, GRU or LSTM). Conceptually, this parameter represents the 
#'          memory capacity and complexity of the network; it dictates how much
#'          information about the sequential trials the model can store and
#'          process.
#'          
#'    \item \code{dropout [double]}
#'    
#'          Dropout is a powerful regularization technique used to prevent 
#'          overfitting in RNNs. During each training iteration, a predefined 
#'          percentage of neurons (units) are randomly "dropped" or deactivated 
#'          by setting their activations to zero.
#' 
#'   \item \code{L [Character]}
#'
#'         This parameter determines the type of regularization applied to the
#'         log-likelihood to penalize model complexity, which helps prevent
#'         overfitting. The default is \code{NA_character_}, meaning no
#'         regularization is applied. Supported values include:
#'         \itemize{
#'           \item \code{L = 1}: L1 regularization (Lasso), which adds a
#'                 penalty proportional to the sum of the absolute values of
#'                 the free parameters.
#'           \item \code{L = 2}: L2 regularization (Ridge), which adds a
#'                 penalty proportional to the sum of the squared values of
#'                 the free parameters.
#'           \item \code{L = 12}: Elastic Net regularization, which applies
#'                 both L1 and L2 penalties simultaneously.
#'         }
#'
#'    \item \code{penalty [double]}
#'
#'          This parameter specifies the strength of the regularization, acting
#'          as a multiplier for the penalty term defined by \code{L}. A larger
#'          value imposes a stronger penalty on the free parameters. The
#'          default value is \code{1e-5}.
#'
#'    \item \code{batch_size [int]}
#'
#'          The number of samples processed before the model's parameters are
#'          updated. Think of this as the size of a study group; the model
#'          reviews this batch of data before adjusting its internal weights.
#'          A larger batch size speeds up calculation but may lead to less
#'          optimal convergence.
#'
#'    \item \code{epochs [int]}
#'
#'          The number of times the learning algorithm will work through the
#'          entire training dataset. This is equivalent to running through the
#'          "textbook" multiple times. Each epoch means the model has seen
#'          every training sample once. More epochs allow for more training but
#'          increase the risk of overfitting.
#'          
#'     \item \code{keras3 [logical]}: 
#'     
#'          The version of Keras to be used for model construction. Currently 
#'          supports \code{keras} and \code{keras3} . Note that 
#'          \code{keras3 = TRUE} enables multi-backend support via the 
#'          \code{backend} parameter.
#'          
#'     \item \code{backend [Character]}: 
#'     
#'          The deep learning framework to serve as the computation engine 
#'          when \code{keras3 = TRUE}. Options include \code{"tensorflow"}, 
#'          \code{"jax"}, and \code{"torch"}. This parameter is ignored 
#'          if \code{keras3 = FALSE}, as it defaults to the \code{"tensorflow"} 
#'          backend.
#'
#'    \item \code{check [logical]}
#'
#'          A logical value indicating whether to perform environment
#'          verification. The default is \code{TRUE}. If set to \code{FALSE},
#'          the function will skip the interactive check regarding whether the
#'          user has properly loaded the \code{tensorflow} environment.
#'
#' }
#' }
#'
#' @section Example:
#' \preformatted{ # default values
#'  control = list(
#'    # General
#'    seed = 123,
#'    core = 1,
#'    sample = 100,
#'    dash = 1e-5,
#'    # LBI
#'    algorithm = "NLOPT_GN_MLSL",
#'    pars = NA,
#'    size = 50,
#'    # MLE
#'    iter = 10,
#'    # MAP
#'    diff = 0.001,
#'    patience = 10,
#'    # SBI
#'    sample = 100,
#'    train = 1000,
#'    scope = "individual",
#'    # ABC
#'    tol = 0.1,
#'    reduction = "PCA",
#'    ncomp = NULL,
#'    metric = "mode",
#'    # RNN
#'    layer = "GRU",
#'    loss = "MSE",
#'    info = c(colnames$object, colnames$action),
#'    units = 128,
#'    dropout = 0,
#'    L = NA_character_,
#'    penalty = 1e-5,
#'    batch_size = 10,
#'    epochs = 100,
#'    keras3 = FALSE,
#'    backend = "tensorflow",
#'    check = TRUE
#'  )
#' }
NULL