IOBottleneck: Debugger IOBottleneck class

In DyfanJones/sagemaker-r-debugger: R6sagemaker debugger for sagemaker operations

Description

This rule helps to detect if GPU is underutilized due to IO bottlenecks. Rule returns True if number of IO bottlenecks exceeds a predefined threshold.

Super class

sagemaker.debugger::ProfilerRuleBase -> IOBottleneck

Methods

Public methods

• IOBottleneck$new()

• IOBottleneck$clone()

Inherited methods

• sagemaker.debugger::ProfilerRuleBase$format()

---

Method new()

Initialize IOBottleneck class

Usage

IOBottleneck$new(
  threshold = 50,
  gpu_threshold = 10,
  io_threshold = 50,
  patience = 1000,
  scan_interval_us = 60 * 1000 * 1000
)

Arguments

threshold

: defines the threshold when Rule should return True. Default is 50 percent. So if there is a bottleneck more than 50% of the time during the training Rule will return True.

gpu_threshold

: threshold that defines when GPU is considered being under-utilized. Default is 70%

io_threshold

: threshold that defines high IO wait time. Default is above 50%

patience

: How many values to record before checking for IO bottlenecks. During training initilization, GPU is likely at 0 percent, so Rule should not check for underutilization immediatly. Default 1000.

scan_interval_us

: interval with which timeline files are scanned. Default is 60000000 us.

---

Method clone()

The objects of this class are cloneable with this method.

Usage

IOBottleneck$clone(deep = FALSE)

Arguments

deep

Whether to make a deep clone.

DyfanJones/sagemaker-r-debugger documentation built on Jan. 20, 2022, 5:49 p.m.