In rstudio/tfds: Interface to 'TensorFlow' Collection of Datasets

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)

library(tfds)

All DatasetBuilders expose various data subsets defined as splits (eg: train, test). When constructing a tf.data.Dataset instance using either tfds_load() or tfds$DatasetBuilder$as_dataset(), one can specify which split(s) to retrieve. It is also possible to retrieve slice(s) of split(s) as well as combinations of those.

Slicing API

Slicing instructions are specified in tfds_load or tfds$DatasetBuilder$as_dataset.

Instructions can be provided as either strings or ReadInstructions. Strings are more compact and readable for simple cases, while ReadInstructions provide more options and might be easier to use with variable slicing parameters.

Examples

# The full `train` split.
train_ds <- tfds_load('mnist:3.*.*', split='train')

# The full `train` split and the full `test` split as two distinct datasets.
train_and_test_ds <-  tfds_load('mnist:3.*.*', split=list('train', 'test'))

# The full `train` and `test` splits, concatenated together.
train_test_ds <- tfds_load('mnist:3.*.*', split='train+test')

# From record 10 (included) to record 20 (excluded) of `train` split.
train_10_20_ds <- tfds_load('mnist:3.*.*', split='train[10:20]')

# The first 10% of train split.
train_10pct_ds <- tfds_load('mnist:3.*.*', split='train[:10%]')

# The first 10% of train + the last 80% of train.
train_10_80pct_ds <- tfds_load('mnist:3.*.*', split='train[:10%]+train[-80%:]')

# 10-fold cross-validation (see also next section on rounding behavior):
# The validation datasets are each going to be 10%:
# [0%:10%], [10%:20%], ..., [90%:100%].
# And the training datasets are each going to be the complementary 90%:
# [10%:100%] (for a corresponding validation set of [0%:10%]),
# [0%:10%] + [20%:100%] (for a validation set of [10%:20%]), ...,
# [0%:90%] (for a validation set of [90%:100%]).
vals_ds <- tfds_load('mnist:3.*.*', glue::glue("train[{10*0:9}%:{10*1:10}%]"))
trains_ds <- tfds_load('mnist:3.*.*', glue::glue("train[:{10*0:9}%]+train[:{10*1:10}%]"))

Percentage slicing

If a slice of a split is requested using the percent (%) unit, and the requested slice boundaries do not divide evenly by 100, then the default behaviour it to round boundaries to the nearest integer (closest). This means that some slices may contain more examples than others. For example:

# Assuming "train" split contains 101 records.
# 100 records, from 0 to 100.
tfds_load("mnist:3.*.*", split="test[:99%]")
# 2 records, from 49 to 51.
tfds_load("mnist:3.*.*", split="test[49%:50%]")

Reproducibility

The sub-split API guarantees that any given split slice (or ReadInstruction) will always produce the same set of records on a given dataset, as long as the major version of the dataset is constant.

For example, tfds_load("mnist:3.0.0", split="train[10:20]") and tfds_load("mnist:3.2.0", split="train[10:20]") will always contain the same elements - regardless of platform, architecture, etc. - even though some of the records might have different values (eg: image encoding, label, ...).

rstudio/tfds documentation built on Nov. 25, 2021, 6:20 p.m.