func_epsilon: Function: Epsilon Related

In binaryRL: Reinforcement Learning Tools for Two-Alternative Forced Choice Tasks

Function: Epsilon Related

Description

The exploration strategy parameters are threshold, epsilon, and lambda.

• Epsilon-first: Used when only threshold is set. Subjects choose randomly for trials less than threshold and by value for trials greater than 'threshold.

• Epsilon-greedy: Used if threshold is default (1) and epsilon is set. Subjects explore with probability epsilon throughout the experiment.

• Epsilon-decreasing: Used if threshold is default (1), and lambda is set. In this strategy, the probability of random choice (exploration) decreases as trials increase. The parameter lambda controls the rate at which this probability declines with each trial.

Usage

func_epsilon(
  i,
  L_freq,
  R_freq,
  L_pick,
  R_pick,
  L_value,
  R_value,
  var1 = NA,
  var2 = NA,
  threshold = 1,
  epsilon = NA,
  lambda = NA,
  alpha,
  beta
)

Arguments

i	The current row number.
L_freq	The frequency of left option appearance
R_freq	The frequency of right option appearance
L_pick	The number of times left option was picked
R_pick	The number of times left option was picked
L_value	The value of the left option
R_value	The value of the right option
var1	[character] Column name of extra variable 1. If your model uses more than just reward and expected value, and you need other information, such as whether the choice frame is Gain or Loss, then you can input the 'Frame' column as var1 into the model. default: var1 = "Extra_Var1"
var2	[character] Column name of extra variable 2. If one additional variable, var1, does not meet your needs, you can add another additional variable, var2, into your model. default: var2 = "Extra_Var2"
threshold	[integer] Controls the initial exploration phase in the epsilon-first strategy. This is the number of early trials where the subject makes purely random choices, as they haven't yet learned the options' values. For example, threshold = 20 means random choices for the first 20 trials. For epsilon-greedy or epsilon-decreasing strategies, threshold should be kept at its default value. P(x) = \begin{cases} \text{trial} \le \text{threshold}, & x=1 \text{ (random choosing)} \\ \text{trial} > \text{threshold}, & x=0 \text{ (value-based choosing)} \end{cases} default: threshold = 1 epsilon-first: threshold = 20, epsilon = NA, lambda = NA
epsilon	[numeric] A parameter used in the epsilon-greedy exploration strategy. It defines the probability of making a completely random choice, as opposed to choosing based on the relative values of the left and right options. For example, if epsilon = 0.1, the subject has a 10 choice and a 90 relevant when threshold is at its default value (1) and lambda is not set. P(x) = \begin{cases} \epsilon, & x=1 \text{ (random choosing)} \\ 1-\epsilon, & x=0 \text{ (value-based choosing)} \end{cases} epsilon-greedy: threshold = 1, epsilon = 0.1, lambda = NA
lambda	[vector] A numeric value that controls the decay rate of exploration probability in the epsilon-decreasing strategy. A higher lambda value means the probability of random choice will decrease more rapidly as the number of trials increases. P(x) = \begin{cases} \frac{1}{1+\lambda \cdot trial}, & x=1 \text{ (random choosing)} \\ \frac{\lambda \cdot trial}{1+\lambda \cdot trial}, & x=0 \text{ (value-based choosing)} \end{cases} epsilon-decreasing threshold = 1, epsilon = NA, lambda = 0.5
alpha	[vector] Extra parameters that may be used in functions.
beta	[vector] Extra parameters that may be used in functions.

Value

A numeric value, either 0 or 1. 0 indicates no exploration (choice based on value), and 1 indicates exploration (random choice) for that trial.

Note

When customizing these functions, please ensure that you do not modify the arguments. Instead, only modify the if-else statements or the internal logic to adapt the function to your needs.

Examples

## Not run: 
func_epsilon <- function(
  # Trial number
  i,
  # Number of times this option has appeared
  L_freq,
  R_freq,
  # Number of times this option has been chosen
  L_pick,
  R_pick,
  # Current value of this option
  L_value,
  R_value,
  # Extra variables
  var1 = NA,
  var2 = NA,
  
  # Free Parameters
  threshold = 1,
  epsilon = NA,
  lambda = NA,
  # Extra parameters
  alpha,
  beta
){
  set.seed(i)
  # Epsilon-First: random choosing before a certain trial number
  if (i <= threshold) {
    try <- 1
  } else if (i > threshold & is.na(epsilon) & is.na(lambda)) {
    try <- 0
  # Epsilon-Greedy: random choosing throughout the experiment with probability epsilon
  } else if (i > threshold & !(is.na(epsilon)) & is.na(lambda)){
    try <- sample(
      c(1, 0),
      prob = c(epsilon, 1 - epsilon),
      size = 1
    )
  # Epsilon-Decreasing: probability of random choosing decreases as trials increase
  } else if (i > threshold & is.na(epsilon) & !(is.na(lambda))) {
    try <- sample(
      c(1, 0),
      prob = c(
        1 / (1 + lambda * i),
        lambda * i / (1 + lambda * i)
      ),
      size = 1
    )
  }
  else {
    try <- "ERROR"
  }

  return(try)
}

## End(Not run)

binaryRL documentation built on Aug. 21, 2025, 6:01 p.m.