TPR-DAG-holdout: TPR-DAG holdout experiments
In gecko515/HEMDAG: Hierarchical Ensemble Methods for Directed Acyclic Graphs

Description Usage Arguments Details Value See Also Examples

High level function to correct the computed scores in a hierarchy according to the chosen ensemble algorithm.

Do.TPR.DAG.holdout(threshold = seq(from = 0.1, to = 0.9, by = 0.1),
  weight = seq(from = 0.1, to = 1, by = 0.1), kk = 5, folds = 5,
  seed = 23, norm = TRUE, norm.type = NULL, positive = "children",
  bottomup = "threshold.free", topdown = "HTD", W = NULL,
  parallel = FALSE, ncores = 1, recall.levels = seq(from = 0.1, to =
  1, by = 0.1), n.round = 3, f.criterion = "F", metric = NULL,
  compute.performance = FALSE, flat.file = flat.file,
  ann.file = ann.file, dag.file = dag.file,
  ind.test.set = ind.test.set, ind.dir = ind.dir,
  flat.dir = flat.dir, ann.dir = ann.dir, dag.dir = dag.dir,
  hierScore.dir = hierScore.dir, perf.dir = perf.dir)

`threshold`	range of threshold values to be tested in order to find the best threshold (`def:` `from:0.1`, `to:0.9`, `by:0.1`). The denser the range is, the higher the probability to find the best threshold is, but obviously the execution time will be higher. Set this parameter only for the thresholded variants; for the threshold-free variants, the parameter `threshold` is automatically set to zero.
`weight`	range of weight values to be tested in order to find the best weight (`def:` `from:0.1`, `to:0.9`, `by:0.1`). The denser the range is, the higher the probability to find the best threshold is, but obviously the execution time will be higher. Set this parameter only for the weighted variants; for the weight-free variants, the parameter `weight` is automatically set to zero.
`kk`	number of folds of the cross validation (`def: kk=5`) on which tuning the parameters `threshold`, `weight` and `tau` of the parametric variants of the hierarchical ensemble algorithms. For the non-parametric variants (i.e. if `bottomup = threshold.free`), the parameter `kk` is automatically set to zero.
`folds`	number of folds of the cross validation on which computing the performance metrics averaged across folds (`def. 5`). If `folds=NULL`, the performance metrics are computed one-shot, otherwise the performance metrics are averaged across folds. If `compute.performance` is set to `FALSE`, `folds` is automatically set to `NULL`.
`seed`	initialization seed for the random generator to create folds (`def. 23`). If `NULL` folds are generated without seed initialization. The parameter `seed` controls both the parameter `kk` and the parameter `folds`. If `compute.performance` is set to `FALSE` and `bottomup` is set to `threshold.free`, then `seed` is automatically set to `NULL`.
`norm`	boolean value: should the flat scores matrix be normalized? `TRUE` (`def.`): the flat scores matrix has been already normalized in according to a normalization method; `FALSE`: the flat scores matrix has not been normalized yet. See the parameter `norm.type` to set the on the fly normalization method to apply among those possible;
`norm.type`	can be one of the following three values: `NULL` (`def.`): set `norm.type` to `NULL` if and only if the parameter `norm` is set to `TRUE`; `MaxNorm`: each score is divided for the maximum of each class; `Qnorm`: quantile normalization. preprocessCore package is used;
`positive`	choice of the positive nodes to be considered in the bottom-up strategy. Can be one of the following values: `children` (`def.`): for each node are considered its positive children; `descendants`: for each node are considered its positive descendants;
`bottomup`	strategy to enhance the flat predictions by propagating the positive predictions from leaves to root. It can be one of the following values: `threshold.free` (`def.`): positive nodes are selected on the basis of the `threshold.free` strategy (`def.`); `threshold`: positive nodes are selected on the basis of the `threshold` strategy; `weighted.threshold.free`: positive nodes are selected on the basis of the `weighted.threshold.free` strategy; `weighted.threshold`: positive nodes are selected on the basis of the `weighted.threshold` strategy; `tau`: positive nodes are selected on the basis of the `tau` strategy; NOTE: `tau` is only a `DESCENS` variants. If you use `tau` strategy you must set the parameter `positive=descendants`;
`topdown`	strategy to make the scores hierarchy-consistent. It can be one of the following values: `HTD` (`def.`): `HTD-DAG` strategy is applied (`HTD-DAG`); `GPAV`: `GPAV` strategy is applied (`GPAV`);
`W`	vector of weight relative to a single example. If the vector `W` is not specified (by `def.` `W=NULL`), `W` is a unitary vector of the same length of the columns' number of the flat scores matrix (root node included). Set `W` only if `topdown=GPAV`.
`parallel`	boolean value: `TRUE`: execute the parallel implementation of GPAV (`GPAV.parallel`); `FALSE` (`def.`): execute the sequential implementation of GPAV (`GPAV.over.examples`); Use `parallel` if and only if `topdown=GPAV`; otherwise set `parallel=FALSE`.
`ncores`	number of cores to use for parallel execution (`def. 8`). Set `ncores=1` if `parallel=FALSE`, otherwise set `ncores` to the desired number of cores. Use `ncores` if and only if `topdown=GPAV`; otherwise set `parallel=1`.
`recall.levels`	a vector with the desired recall levels (`def:` `from:0.1`, `to:0.9`, `by:0.1`) to compute the Precision at fixed Recall level (PXR). If `compute.performance=FALSE` then `recall.levels` is automatically set to `NULL`.
`n.round`	number of rounding digits to be applied to the hierarchical scores matrix (`def. 3`). It is used for choosing the best threshold on the basis of the best F-measure. If `compute.performance` is set to `FALSE` and `bottomup` is set to `threshold.free`, then `n.round` is automatically set to `NULL`.
`f.criterion`	character. Type of F-measure to be used to select the best F-measure. Two possibilities: `F` (def.): corresponds to the harmonic mean between the average precision and recall; `avF`: corresponds to the per-example `F-score` averaged across all the examples; If `compute.performance` is set to `FALSE` and `bottomup` is set to `threshold.free`, then `f.criterion` is automatically set to `NULL`.
`metric`	a string character specifying the performance metric on which to maximize the parametric ensemble variant. It can be one of the following values: `PRC`: the parametric ensemble variant is maximized on the basis of AUPRC (`AUPRC`); `FMAX`: the parametric ensemble variant is maximized on the basis of Fmax (`Multilabel.F.measure`); `NULL`: on the `threshold.free` variant none parameter optimization is needed, since the variant is non-parametric. So, if `bottomup=threshold.free` set `metric=NULL` (`def.`);
`compute.performance`	boolean value: should the flat and hierarchical performance (`AUPRC`, `AUROC`, `PXR`, `multilabel F-score`) be returned? `FALSE`: performance are not computed and just the hierarchical scores matrix is returned; `TRUE` (`def.`): both performance and hierarchical scores matrix are returned;
`flat.file`	name of the file containing the flat scores matrix to be normalized or already normalized (without rda extension).
`ann.file`	name of the file containing the label matrix of the examples (without rda extension).
`dag.file`	name of the file containing the graph that represents the hierarchy of the classes (without rda extension).
`ind.test.set`	name of the file containing a vector of integer numbers corresponding to the indices of the elements (rows) of scores matrix to be used in the test set.
`ind.dir`	relative path to folder where `ind.test.set` is stored.
`flat.dir`	relative path where flat scores matrix is stored.
`ann.dir`	relative path where annotation matrix is stored.
`dag.dir`	relative path where graph is stored.
`hierScore.dir`	relative path where the hierarchical scores matrix must be stored.
`perf.dir`	relative path where the performance measures must be stored. If `compute.performance=FALSE`, the parameter `perf.dir` is automatically set to `NULL`.

The parametric hierarchical ensemble variants are cross-validated by maximizing in according to the metric chosen in the parameter metric, that is F-measure (Multilabel.F.measure) or AUPRC (AUPRC).

The function checks if the number of classes between the flat scores matrix and the annotations matrix mismatched. If so, the number of terms of the annotations matrix is shrunk to the number of terms of the flat scores matrix and the corresponding subgraph is computed as well. N.B.: it is supposed that all the nodes of the subgraph are accessible from the root.

We excluded the predictions of the root node in computing all the performances, since it is a dummy node added to the ontology for practical reasons (e.g. some graph-based software may require a single root node to work). However, the root node scores are stored in the hierarchical scores matrix.

Two rda files stored in the respective output directories:

Hierarchical Scores Results: a matrix with examples on rows and classes on columns representing the computed hierarchical scores for each example and for each considered class. It is stored in the hierScore.dir directory;
Performance Measures: flat and hierarchical performace results:
1. AUPRC results computed though AUPRC.single.over.classes (AUPRC);
2. AUROC results computed through AUROC.single.over.classes (AUROC);
3. PXR results computed though precision.at.given.recall.levels.over.classes (PXR);
4. FMM results computed though compute.Fmeasure.multilabel (FMM);

It is stored in the perf.dir directory.

TPR-DAG-variants

data(graph);
data(scores);
data(labels);
data(test.index);
tmpdir <- paste0(tempdir(),"/");
save(g, file=paste0(tmpdir,"graph.rda"));
save(L, file=paste0(tmpdir,"labels.rda"));
save(S, file=paste0(tmpdir,"scores.rda"));
save(test.index, file=paste0(tmpdir,"test.index.rda"));
ind.dir <- dag.dir <- flat.dir <- ann.dir <- tmpdir;
hierScore.dir <- perf.dir <- tmpdir;
ind.test.set <- "test.index";
dag.file <- "graph";
flat.file <- "scores";
ann.file <- "labels";
threshold <- weight <- 0;
norm.type <- "MaxNorm";
positive <- "children";
bottomup <- "threshold.free";
topdown <- "HTD";
recall.levels <- seq(from=0.25, to=1, by=0.25);
Do.TPR.DAG.holdout(threshold=threshold, weight=weight, kk=NULL, folds=NULL, seed=NULL, norm=FALSE, 
norm.type=norm.type, positive=positive, bottomup=bottomup, topdown=topdown, W=NULL, 
compute.performance=TRUE, parallel=FALSE, ncores=1, recall.levels=recall.levels, n.round=3, 
f.criterion="F", metric=NULL, flat.file=flat.file, ann.file=ann.file, dag.file=dag.file, 
ind.test.set=ind.test.set, ind.dir=ind.dir, flat.dir=flat.dir, ann.dir=ann.dir, dag.dir=dag.dir, 
hierScore.dir=hierScore.dir, perf.dir=perf.dir);