get_generator: Wrapper for generator functions

In GenomeNet/deepG: Deep Learning for Genome Sequence Data

Wrapper for generator functions

Description

For a detailed description see the data generator tutorial. Will choose one of the generators from generator_fasta_lm, generator_fasta_label_folder, generator_fasta_label_header_csv, generator_rds, generator_random, generator_dummy or generator_fasta_lm according to the train_type and random_sampling arguments.

Usage

get_generator(
  path = NULL,
  train_type,
  batch_size,
  maxlen,
  step = NULL,
  shuffle_file_order = FALSE,
  vocabulary = c("A", "C", "G", "T"),
  seed = 1,
  proportion_entries = NULL,
  shuffle_input = FALSE,
  format = "fasta",
  path_file_log = NULL,
  reverse_complement = FALSE,
  n_gram = NULL,
  n_gram_stride = NULL,
  output_format = "target_right",
  ambiguous_nuc = "zero",
  proportion_per_seq = NULL,
  skip_amb_nuc = NULL,
  use_quality_score = FALSE,
  padding = FALSE,
  added_label_path = NULL,
  target_from_csv = NULL,
  add_input_as_seq = NULL,
  max_samples = NULL,
  concat_seq = NULL,
  target_len = 1,
  file_filter = NULL,
  use_coverage = NULL,
  sample_by_file_size = FALSE,
  add_noise = NULL,
  random_sampling = FALSE,
  set_learning = NULL,
  file_limit = NULL,
  reverse_complement_encoding = FALSE,
  read_data = FALSE,
  target_split = NULL,
  path_file_logVal = NULL,
  model = NULL,
  vocabulary_label = NULL,
  masked_lm = NULL,
  val = FALSE,
  return_int = FALSE,
  verbose = TRUE,
  delete_used_files = FALSE,
  reshape_xy = NULL
)

Arguments

path	Path to training data. If train_type is label_folder, should be a vector or list where each entry corresponds to a class (list elements can be directories and/or individual files). If train_type is not label_folder, can be a single directory or file or a list of directories and/or files.
train_type	Either "lm", "lm_rds", "masked_lm" for language model; "label_header", "label_folder", "label_csv", "label_rds" for classification or "dummy_gen". Language model is trained to predict character(s) in a sequence. "label_header"/"label_folder"/"label_csv" are trained to predict a corresponding class given a sequence as input. If "label_header", class will be read from fasta headers. If "label_folder", class will be read from folder, i.e. all files in one folder must belong to the same class. If "label_csv", targets are read from a csv file. This file should have one column named "file". The targets then correspond to entries in that row (except "file" column). Example: if we are currently working with a file called "a.fasta" and corresponding label is "label_1", there should be a row in our csv file file label_1 label_2 "a.fasta" 1 0 If "label_rds", generator will iterate over set of .rds files containing each a list of input and target tensors. Not implemented for model with multiple inputs. If "lm_rds", generator will iterate over set of .rds files and will split tensor according to target_len argument (targets are last target_len nucleotides of each sequence). If "dummy_gen", generator creates random data once and repeatedly feeds these to model. If "masked_lm", generator maskes some parts of the input. See masked_lm argument for details.
batch_size	Number of samples used for one network update.
maxlen	Length of predictor sequence.
step	Frequency of sampling steps.
shuffle_file_order	Boolean, whether to go through files sequentially or shuffle beforehand.
vocabulary	Vector of allowed characters. Characters outside vocabulary get encoded as specified in ambiguous_nuc.
seed	Sets seed for reproducible results.
proportion_entries	Proportion of fasta entries to keep. For example, if fasta file has 50 entries and proportion_entries = 0.1, will randomly select 5 entries.
shuffle_input	Whether to shuffle entries in file.
format	File format, "fasta", "fastq", "rds" or "fasta.tar.gz", "fastq.tar.gz" for tar.gz files.
path_file_log	Write name of files used for training to csv file if path is specified.
reverse_complement	Boolean, for every new file decide randomly to use original data or its reverse complement.
n_gram	Integer, encode target not nucleotide wise but combine n nucleotides at once. For example for ⁠n=2, "AA" -> (1, 0,..., 0),⁠ ⁠"AC" -> (0, 1, 0,..., 0), "TT" -> (0,..., 0, 1)⁠, where the one-hot vectors have length length(vocabulary)^n.
n_gram_stride	Step size for n-gram encoding. For AACCGGTT with n_gram = 4 and n_gram_stride = 2, generator encodes ⁠(AACC), (CCGG), (GGTT)⁠; for n_gram_stride = 4 generator encodes ⁠(AACC), (GGTT)⁠.
output_format	Determines shape of output tensor for language model. Either "target_right", "target_middle_lstm", "target_middle_cnn" or "wavenet". Assume a sequence "AACCGTA". Output correspond as follows ⁠"target_right": X = "AACCGT", Y = "A"⁠ ⁠"target_middle_lstm": X = (X_1 = "AAC", X_2 = "ATG"), Y = "C"⁠ (note reversed order of X_2) ⁠"target_middle_cnn": X = "AACGTA", Y = "C"⁠ ⁠"wavenet": X = "AACCGT", Y = "ACCGTA"⁠
ambiguous_nuc	How to handle nucleotides outside vocabulary, either "zero", "discard", "empirical" or "equal". If "zero", input gets encoded as zero vector. If "equal", input is repetition of 1/length(vocabulary). If "discard", samples containing nucleotides outside vocabulary get discarded. If "empirical", use nucleotide distribution of current file.
proportion_per_seq	Numerical value between 0 and 1. Proportion of sequence to take samples from (use random subsequence).
skip_amb_nuc	Threshold of ambiguous nucleotides to accept in fasta entry. Complete entry will get discarded otherwise.
use_quality_score	Whether to use fastq quality scores. If TRUE input is not one-hot-encoding but corresponds to probabilities. For example (0.97, 0.01, 0.01, 0.01) instead of (1, 0, 0, 0).
padding	Whether to pad sequences too short for one sample with zeros.
added_label_path	Path to file with additional input labels. Should be a csv file with one column named "file". Other columns should correspond to labels.
target_from_csv	Path to csv file with target mapping. One column should be called "file" and other entries in row are the targets.
add_input_as_seq	Boolean vector specifying for each entry in added_label_path if rows from csv should be encoded as a sequence or used directly. If a row in your csv file is a sequence this should be TRUE. For example you may want to add another sequence, say ACCGT. Then this would correspond to 1,2,2,3,4 in csv file (if vocabulary = c("A", "C", "G", "T")). If add_input_as_seq is TRUE, 12234 gets one-hot encoded, so added input is a 3D tensor. If add_input_as_seq is FALSE this will feed network just raw data (a 2D tensor).
max_samples	Maximum number of samples to use from one file. If not NULL and file has more than max_samples samples, will randomly choose a subset of max_samples samples.
concat_seq	Character string or NULL. If not NULL all entries from file get concatenated to one sequence with concat_seq string between them. Example: If 1.entry AACC, 2. entry TTTG and concat_seq = "ZZZ" this becomes AACCZZZTTTG.
target_len	Number of nucleotides to predict at once for language model.
file_filter	Vector of file names to use from path_corpus.
use_coverage	Integer or NULL. If not NULL, use coverage as encoding rather than one-hot encoding and normalize. Coverage information must be contained in fasta header: there must be a string "cov_n" in the header, where n is some integer.
sample_by_file_size	Sample new file weighted by file size (bigger files more likely).
add_noise	NULL or list of arguments. If not NULL, list must contain the following arguments: noise_type can be "normal" or "uniform"; optional arguments sd or mean if noise_type is "normal" (default is sd=1 and mean=0) or ⁠min, max⁠ if noise_type is "uniform" (default is ⁠min=0, max=1⁠).
random_sampling	Whether samples should be taken from random positions when using max_samples argument. If FALSE random samples are taken from a consecutive subsequence.
set_learning	When you want to assign one label to set of samples. Only implemented for train_type = "label_folder". Input is a list with the following parameters samples_per_target: how many samples to use for one target. maxlen: length of one sample. reshape_mode: ⁠"time_dist", "multi_input"⁠ or "concat". If reshape_mode is "multi_input", generator will produce samples_per_target separate inputs, each of length maxlen (model should have samples_per_target input layers). If reshape_mode is "time_dist", generator will produce a 4D input array. The dimensions correspond to ⁠(batch_size, samples_per_target, maxlen, length(vocabulary))⁠. If reshape_mode is "concat", generator will concatenate samples_per_target sequences of length maxlen to one long sequence. If reshape_mode is "concat", there is an additional buffer_len argument. If buffer_len is an integer, the subsequences are interspaced with buffer_len rows. The input length is (maxlen * samples_per_target) + buffer_len * (samples_per_target - 1).
file_limit	Integer or NULL. If integer, use only specified number of randomly sampled files for training. Ignored if greater than number of files in path.
reverse_complement_encoding	Whether to use both original sequence and reverse complement as two input sequences.
read_data	If TRUE the first element of output is a list of length 2, each containing one part of paired read. Maxlen should be 2*length of one read.
target_split	If target gets read from csv file, list of names to divide target tensor into list of tensors. Example: if csv file has header names ⁠"file", "label_1", "label_2", "label_3"⁠ and target_split = list(c("label_1", "label_2"), "label_3"), this will divide target matrix to list of length 2, where the first element contains columns named "label_1" and "label_2" and the second entry contains the column named "label_3".
path_file_logVal	Path to csv file logging used validation files.
model	A keras model.
vocabulary_label	Character vector of possible targets. Targets outside vocabulary_label will get discarded if train_type = "label_header".
masked_lm	If not NULL, input and target are equal except some parts of the input are masked or random. Must be list with the following arguments: mask_rate: Rate of input to mask (rate of input to replace with mask token). random_rate: Rate of input to set to random token. identity_rate: Rate of input where sample weights are applied but input and output are identical. include_sw: Whether to include sample weights. block_len (optional): Masked/random/identity regions appear in blocks of size block_len.
val	Logical, call initialized generator "genY" or "genValY" where Y is an integer between 1 and length of directories.
return_int	Whether to return integer encoding or one-hot encoding.
verbose	Whether to show messages.
delete_used_files	Whether to delete file once used. Only applies for rds files.
reshape_xy	Can be a list of functions to apply to input and/or target. List elements (containing the reshape functions) must be called x for input or y for target and each have arguments called x and y. For example: reshape_xy = list(x = function(x, y) {return(x+1)}, y = function(x, y) {return(x+y)}) . For rds generator needs to have an additional argument called sw.

Value

A generator function.

Examples

# create dummy fasta files
fasta_path <- tempfile()
dir.create(fasta_path)
create_dummy_data(file_path = fasta_path,
                  num_files = 3,
                  seq_length = 10,
                  num_seq = 5,
                  vocabulary = c("a", "c", "g", "t"))

gen <- get_generator(path = fasta_path,
                     maxlen = 5, train_type = "lm",
                     output_format = "target_right",
                     step = 3, batch_size = 7)
z <- gen()
x <- z[[1]]
y <- z[[2]]
dim(x)
dim(y)

GenomeNet/deepG documentation built on Jan. 25, 2025, 12:05 a.m.