dataset_shard: Creates a dataset that includes only 1 / num_shards of this...

View source: R/dataset_methods.R

dataset_shardR Documentation

Creates a dataset that includes only 1 / num_shards of this dataset.

Description

This dataset operator is very useful when running distributed training, as it allows each worker to read a unique subset.

Usage

dataset_shard(dataset, num_shards, index)

Arguments

dataset

A dataset

num_shards

A integer representing the number of shards operating in parallel.

index

A integer, representing the worker index.

Value

A dataset


tfdatasets documentation built on Sept. 11, 2024, 8:53 p.m.