dataset_shard: Creates a dataset that includes only 1 / num_shards of this...

View source: R/dataset_methods.R

dataset_shardR Documentation

Creates a dataset that includes only 1 / num_shards of this dataset.

Description

This dataset operator is very useful when running distributed training, as it allows each worker to read a unique subset.

Usage

dataset_shard(dataset, num_shards, index)

Arguments

dataset

A dataset

num_shards

A integer representing the number of shards operating in parallel.

index

A integer, representing the worker index.

Value

A dataset


tfdatasets documentation built on June 30, 2022, 1:04 a.m.