median_stopping_policy: Define a median stopping policy for early termination of...

Description Usage Arguments Value Details Examples

View source: R/hyperdrive.R

Description

Median stopping is an early termination policy based on running averages of primary metrics reported by the runs. This policy computes running averages across all training runs and terminates runs whose performance is worse than the median of the running averages. Specifically, a run will be canceled at interval N if its best primary metric reported up to interval N is worse than the median of the running averages for intervals 1:N across all runs.

Usage

1
median_stopping_policy(evaluation_interval = 1L, delay_evaluation = 0L)

Arguments

evaluation_interval

An integer of the frequency for applying policy.

delay_evaluation

An integer of the number of intervals for which to delay the first evaluation.

Value

The MedianStoppingPolicy object.

Details

The median stopping policy takes the following optional configuration parameters:

This policy is inspired from the research publication Google Vizier: A Service for Black-Box Optimization.

If you are looking for a conservative policy that provides savings without terminating promising jobs, you can use a MedianStoppingPolicy with evaluation_interval = 1 and delay_evaluation = 5. These are conservative settings that can provide approximately 25%-35% savings with no loss on the primary metric (based on our evaluation data).

Examples

1
2
3
4
5
6
7
8
9
# In this example, the early termination policy is applied at every
# interval starting at evaluation interval 5. A run will be terminated at
# interval 5 if its best primary metric is worse than the median of the
# running averages over intervals 1:5 across all training runs
## Not run: 
early_termination_policy = median_stopping_policy(evaluation_interval = 1L,
                                                  delay_evaluation = 5L)

## End(Not run)

azuremlsdk documentation built on Oct. 23, 2020, 8:22 p.m.