truncation_selection_policy: Define a truncation selection policy for early termination of...

Description Usage Arguments Value Details Examples

View source: R/hyperdrive.R

Description

Truncation selection cancels a given percentage of lowest performing runs at each evaluation interval. Runs are compared based on their performance on the primary metric and the lowest X% are terminated.

Usage

1
2
3
4
5
truncation_selection_policy(
  truncation_percentage,
  evaluation_interval = 1L,
  delay_evaluation = 0L
)

Arguments

truncation_percentage

An integer of the percentage of lowest performing runs to terminate at each interval.

evaluation_interval

An integer of the frequency for applying policy.

delay_evaluation

An integer of the number of intervals for which to delay the first evaluation.

Value

The TruncationSelectionPolicy object.

Details

This policy periodically cancels the given percentage of runs that rank the lowest for their performance on the primary metric. The policy strives for fairness in ranking the runs by accounting for improving model performance with training time. When ranking a relatively young run, the policy uses the corresponding (and earlier) performance of older runs for comparison. Therefore, runs aren't terminated for having a lower performance because they have run for less time than other runs.

The truncation selection policy takes the following configuration parameters:

For example, when evaluating a run at a interval N, its performance is only compared with the performance of other runs up to interval N even if they reported metrics for intervals greater than N.

Examples

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# In this example, the early termination policy is applied at every interval
# starting at evaluation interval 5. A run will be terminated at interval 5
# if its performance at interval 5 is in the lowest 20% of performance of all
# runs at interval 5
## Not run: 
early_termination_policy = truncation_selection_policy(
                                                 truncation_percentage = 20L,
                                                 evaluation_interval = 1L,
                                                 delay_evaluation = 5L)

## End(Not run)

azuremlsdk documentation built on Oct. 23, 2020, 8:22 p.m.