In Bioconductor/DelayedDataFrame: Delayed operation on DataFrame using standard DataFrame metaphor

with new updates in S4Vectors, 1. define extractCOLS for SQLDataFrame, and apply to [ method. -- done! 2. add replaceCOLS -- done! 3. add mergeROWS -- added! - [<-,DataFrame => mergeROWS => .subassign_columns 4. replaceROWS? 3. replaceROWS (lower-level), mergeROWS are newly added and they call .subassign_columns (newly added). mergeROWS calls .make_rownames which was re-written by adding nsbs argument. .make_rownames <- function(x, i, nsbs, value). .make_rownames calls replaceROWS. See R/subsetting-utils.R for recycleSingleBracketReplacementValue() function. See R/DataFrame-class.R for normalizeSingleBracketReplacementValue() function. See R/Vector-class.R for implementation examples. 3. debug, update [<- method and fix test errors. setReplaceMethod("[", "DataFrame") calls normalizeSingleBracketReplacementValue & replaceCOLS, recycleSingleBracketReplacementValue() & .add_missing_columns & mergeROWS. 4. R/subsetting-utils.R: - setMethod("complement", "NSBS", function(x) is newly added generic and method for NSBS. - setGeneric("normalizeSingleBracketReplacementValue", signature="x", function(value, x) - recycleSingleBracketReplacementValue<-function(value, x, i) newly added. - 5 internal generics to ease implementation of [ and [<- subsetting for Vector and DataFrame subclasses: extractROWS(), replaceROWS(), mergeROWS(), extractCOLS(), replaceCOLS(). - mergeROWS() is a composition of replaceROWS() and bindROWS() to support appending in [<-(). Vector subclasses never need to implement mergeROWS(), but a custom method may be useful for e.g. optimization.

setGeneric("mergeROWS", signature=c("x", "i"),
    function(x, i, value) standardGeneric("mergeROWS")
)
setGeneric("extractCOLS", signature=c("x", "i"),
    function(x, i) standardGeneric("extractCOLS")
)

setGeneric("replaceCOLS", signature=c("x", "i"),
    function(x, i, value) standardGeneric("replaceCOLS")
)
default_replaceROWS <- function(x, i, value)
{
    mergeROWS(x, i, value)
}
default_mergeROWS <- function(x, i, value)
setMethod("replaceROWS", c("ANY", "ANY"), default_replaceROWS)
setMethod("mergeROWS", c("ANY", "ANY"), default_mergeROWS)

git log commits before the refactoring of `[`, `[<-`, ...

before submission (09/07/2018)

do extra check:

setMethod("cbind", "DelayedDataFrame", function(..., deparse.level = 1)
 if (!any(isDDF))
     callNextMethod()
 if (!is(objects[[1]], "DelayedDataFrame"))
     stop(paste('there must be at least one "DelayedDataFrame"',
                'object to have "cbind,DelayedDataFrame" work.')

5/11/2018, review code with Herve & Daniel:

DelayedDataFrame code polish.

.fulllength(LazyIndex(...)), return 0 if listData == NULL. -- done! so .fulllength(LazyIndex()) returns 0 instead of integer(0).
lazyIndex<- replacement method. do not update show method for nrows, rownames(). -- done! (specified in @details)
DDF constructor, construct DDF for each entry of argument, and cbind(DDF). -- done!
update test_LazyIndex, for .fulllength(). -- done! return 0 for LazyIndex(). NULL for NULL (initial) lazyIndex. integer for non-NULL lazyIndex.
reorganize and update test_DDF-class. -- done!
reorganize and update test_DDF-methods, with constructor function with non DDF and DDF inputs.
stand-alone package? convenient. -- done!
check colnames for DDF construction.
extractCOLS... (forked from S4Vectors to Liubuntu, will test)
concatenateObject renamed as "bindROWS" in S4Vectors 0.19.5. Need to rename "concatenateObject,LazyIndex" (intended for cbind of DDFs, to bind the lazyList for all columns, so that could concatenate the original listData without realization) into "bindROWS,LazyIndex". -- done! -- S4Vectors::bindROWS. the new "bindROWS" function which replaces the "concatenateObjects" now works for no-dim objects (vectors, factors, lists), 1-dim arrays, 2-dim objects (DelayedArray, DataFrame, data.frame, matrix, arrays...) and >2-dim objects. After "bindROWS", it returns objects of the same class of input value. For 2D objects, it calls "base::rbind". -- todo: test DF with meaningful lazyIndex. and try rbind(). Basically just "callNextMethod()" would work. Inside "bindROWS,DataFrame", it realized the listData by calling "[[]]" when "bind_cols_along_their_ROWS". So we should discard the old "lazyIndex" and assign "NULL" for all columns. "LazyIndex(vector("list", 1), rep(1L, ncol(x)))" -- done! rbind construct a new DDF with initial NULL as lazyIndex. -- done! debug "test_DelayedDataFrame". 1D DelayedArray subsetting in last test. -- asked Herve in slack#questions. refer to "https://github.com/Bioconductor/DelayedArray/issues/6#issue" for discussion of drop default value.

-- todo: debug "test_DelayedDataFrame". constructor. "concatenateObjects,DDF"... "cbind,DataFrame" calls "DataFrame()" constructor. maybe we should set "cbind,DDF" to call "DelayedDataFrame()" constructor, where if all inputs are DDF, they will concatenate the existing lazyIndex's and concatenate the listData. -- done! debug "test_DelayedDataFrame", "[[<-".

-- meeting with Martin/Herve for the "cbind" performance in "DDF". Now "rbind" realizes the lazyIndex. how about "cbind". should we call "DDF()" constructor, where check inputs if all DDF, then "concatenateObject,LazyInde"? or call "cbind,DF" and reconstruct a DDF?
for "c" calls "do.call(cbind, list(objects))", calls "DataFrame()" constructor. Am doing DelayedDataFrame() constructor calls cbind (if inputs are all DDF) / DataFrame() constructor for general inputs. cbind, DDF combines the lazyIndex and do not incur realization. So it would be "c,DDF">>"cbind,DDF"(no realization). Now cbind,DDF works as intended, need to make the DDF(DDF inputs) work. Environment difference: https://devteam-bioc.slack.com/archives/G7KH1498U/p1528903667000847 (need to import cbind generic from BiocGenerics!! ) - If "cbind,DelayedDataFrame" realize and construct a new DDF, there would be no problem with the test functions for "constructor" and "[[<-" - cbind(DDF, DF) returns what? cbind(DDF) calls DelayedDataFrame() constructor. rename the current cbind,DDF as something else? cbind,DDF calls DDF() and cbind,DF calls DF(), returns warning message:

In methods:::.selectDotsMethod(classes, .MTable, .AllMTable) :
  multiple direct matches: "DelayedDataFrame", "DataFrame"; using the first of these

wait for Herve or Martin's reply on slack. If they agree with "cbind,DDF" do not realize, would set up a meeting with Herve for some bugs above.

summary of meeting:

if rbind,DDF callNextMethod(), then it will incur realization automatically. Is it something we should expect DDF to do?
cbind,DDF incur realization? use case: adding new columns (realize) / cbind(DDF,DDF).
cbind(DDF,DF) returns what? need definition separately? because cbind functions are defined separately on DDF and DF.
What methods to export? -- export all "setMethod()", "setReplaceMethod()". grep for "setAs()" to export. constructor to export. getters? NO. setters? NO. back-end function "lazyIndex<-" not to export to users. :::, str... explain the DDF working flow in manual, vignette, with examples.

as.list, names, extractROWS, coerce, [... OR as long as DDF is been depended by other package, there is no need to export methods? and they will be available automatically? What's the difference between export methods and export functions??

-- todo: 1. rename "c,LazyIndex" into "cbind,LazyIndex". "bindROWS,LazyIndex" -- done! 2. DataFrame: c -- cbind -- DataFrame() DDF: c -- cbind -- .cbind_DDF -- done! DDF() -- cbind -- done! 3. cbind, coercion all objects into DDF. Do not need to define coercion of DDF to DDF with no-op because as() does not do the correct thing. still need is(x,"DDF") to check if DDF. -- done! 4. Question: cbind(DDF, DF) would incur both "cbind,DDF" and "cbind,DF". How to distinguish? just to make sure the first argument to be of the class we desire the "cbind" method to work. call "DelayedDataFrame:::.cbind_DelayedDataFrame" inside "cbind,DataFrame" after checking if any argument is of DDF derivative. And "Depends/imports" DelayedDataFrame in DESCRIPTION.

checkings?? cbind to make sure all DDF objects to have same nrows. otherwise, iterate atomic ...

S3 & S4: S3: function: class could be arbitrary; method could be defined directly without generic, even when class "x" does not exist. func.X <- function(x, ...); you can use S3 method on S4 class. S4: setClass(), setGeneric("method"), setMethod("method", signature="X")

think of SimpleList as rectangular!

bugs in pkg: methods

package attributes stripped with [[, and as.list()

further implementation in S4Vectors

acbind/arbind

future new features to implement

Michaels comments: https://github.com/Bioconductor/VariantExperiment/issues/3#issuecomment-387192874
Herve: SNPlocks, Gpos? on-disk long table? SQL...
relational/algebra? DataFrame S4 replacement or ordinary data.frame.