generator_rds: Rds data generator

In GenomeNet/deepG: Deep Learning for Genome Sequence Data

Rds data generator

Description

Creates training batches from rds files. Rds files must contain a list of length 2 (input/target) or of length 1 (for language model). If target_len is not NULL will take the last target_len entries of the first list element as targets and the rest as input.

Usage

generator_rds(
  rds_folder,
  batch_size,
  path_file_log = NULL,
  max_samples = NULL,
  proportion_per_seq = NULL,
  target_len = NULL,
  seed = NULL,
  delete_used_files = FALSE,
  reverse_complement = FALSE,
  sample_by_file_size = FALSE,
  n_gram = NULL,
  n_gram_stride = 1,
  reverse_complement_encoding = FALSE,
  add_noise = NULL,
  reshape_xy = NULL
)

Arguments

rds_folder	Path to input files.
batch_size	Number of samples in one batch.
path_file_log	Write name of files to csv file if path is specified.
max_samples	Maximum number of samples to use from one file. If not NULL and file has more than max_samples samples, will randomly choose a subset of max_samples samples.
proportion_per_seq	Numerical value between 0 and 1. Proportion of sequence to take samples from (use random subsequence).
target_len	Number of target nucleotides for language model.
seed	Sets seed for set.seed function for reproducible results.
delete_used_files	Whether to delete file once used. Only applies for rds files.
reverse_complement	Boolean, for every new file decide randomly to use original data or its reverse complement.
sample_by_file_size	Sample new file weighted by file size (bigger files more likely).
n_gram	Integer, encode target not nucleotide wise but combine n nucleotides at once. For example for ⁠n=2, "AA" -> (1, 0,..., 0),⁠ ⁠"AC" -> (0, 1, 0,..., 0), "TT" -> (0,..., 0, 1)⁠, where the one-hot vectors have length length(vocabulary)^n.
n_gram_stride	Step size for n-gram encoding. For AACCGGTT with n_gram = 4 and n_gram_stride = 2, generator encodes ⁠(AACC), (CCGG), (GGTT)⁠; for n_gram_stride = 4 generator encodes ⁠(AACC), (GGTT)⁠.
reverse_complement_encoding	Whether to use both original sequence and reverse complement as two input sequences.
add_noise	NULL or list of arguments. If not NULL, list must contain the following arguments: noise_type can be "normal" or "uniform"; optional arguments sd or mean if noise_type is "normal" (default is sd=1 and mean=0) or ⁠min, max⁠ if noise_type is "uniform" (default is ⁠min=0, max=1⁠).
reshape_xy	Can be a list of functions to apply to input and/or target. List elements (containing the reshape functions) must be called x for input or y for target and each have arguments called x and y. For example: reshape_xy = list(x = function(x, y) {return(x+1)}, y = function(x, y) {return(x+y)}) . For rds generator needs to have an additional argument called sw.

Value

A generator function.

Examples

# create 3 rds files
rds_folder <- tempfile()
dir.create(rds_folder)
batch_size <- 7
maxlen <- 11
voc_len <- 4
for (i in 1:3) {
  x <- sample(0:(voc_len-1), maxlen*batch_size, replace = TRUE)
  x <- keras::to_categorical(x, num_classes = voc_len)
  x <- array(x, dim = c(batch_size, maxlen, voc_len))
  y <- sample(0:2, batch_size ,replace = TRUE)
  y <- keras::to_categorical(y, num_classes = 3)
  xy_list <- list(x, y)
  file_name <- paste0(rds_folder, "/file_", i, ".rds")
  saveRDS(xy_list, file_name) 
}

# create generator
gen <- generator_rds(rds_folder, batch_size = 2)
z <- gen()
x <- z[[1]]
y <- z[[2]]
x[1, , ]
y[1, ]

GenomeNet/deepG documentation built on Dec. 24, 2024, 12:11 p.m.