In MarianSchoen/DMC: Comparison of different Deconvolution Models in several scenarios

options(width = 1000)
knitr::opts_chunk$set(echo = FALSE, warning = FALSE, message = FALSE)
temp.dir <- params$tempdir
plot.dir <- paste(temp.dir, "report_plots", sep = '/')
celltype.order <- params$celltype.order
celltype.order.sim <- params$celltype.order.sim

benchmark.name <- strsplit(temp.dir, "/")[[1]][
  length(strsplit(temp.dir, "/")[[1]])
]
# read input and processed data
input_raw <- read_data(paste(temp.dir, "/input_data/raw.h5", sep = ""))
sc.counts <- input_raw$sc.counts
sc.pheno <- input_raw$sc.pheno
real.counts <- input_raw$bulk.counts
real.props <- input_raw$bulk.props
validation_set <- read_data(
  paste(temp.dir, "/input_data/validation_set.h5", sep = "")
)
training_set <- read_data(
  paste(temp.dir, "/input_data/training_set.h5", sep = "")
)
training.exprs <- training_set$sc.counts
training.pheno <- training_set$sc.pheno
test.exprs <- validation_set$sc.counts
test.pheno <- validation_set$sc.pheno
sim.bulks <- list(
  bulks = validation_set$real.counts,
  props = validation_set$real.props
)
# load input parameters
input_params <- read_misc_input(
  paste(temp.dir, "input_data/params.h5", sep = "/")
)
genesets <- input_params$genesets
algorithm.names <- input_params$algorithms
function.call <- input_params$function.call
grouping <- input_params$grouping

Deconvolution Benchmark

Real bulks

Score plot

plotfile <- paste0(plot.dir, "/score_plot_real.pdf")
if(file.exists(plotfile)){
  knitr::include_graphics(plotfile)
}

Show/hide description

Deconvolution result for real bulks per algorithm and cell type.
Celltype order (left to right): r celltype.order

The plotted value is the pearson correlation coefficient, coded by rectangle size and color (small to big, red to green with increasing performance).
Negative and undefined scores were set to 0. The first column contains the average performance of each algorithm across all cell types. In case of multiple repetitions (default), the individual scores for each cell type are averaged over all repetitions. The score specified for each algorithm corresponds to the 'overall' column. Uncertainty (standard deviation) of the overall performance in case multiple runs.

Scatter plots of quantities

plotfile <- paste0(plot.dir, "/scatter_plots_real.pdf")
if(file.exists(plotfile)){
  knitr::include_graphics(plotfile)
}

Show/hide description

Scatterplot of the real cell type quantities (x-axis) and the estimated quantities (y-axis) per cell type for each algorithm. Dashed line marks the identity.

Runtime plot

plotfile <- paste0(plot.dir, "/runtime_plot_real.pdf")
if(file.exists(plotfile)){
  knitr::include_graphics(plotfile)
}

Show/hide description

Runtime per algorithm. Due to possible large differences in runtime the scale on the x-axis is transformed to logarithmic scale (log10). In case of multiple repetitions, the plotted times are averages over all repetitions.

Additional Plots

Show/hide

Condition number plots

plotfile <- paste0(plot.dir, "/condition_number_plots_real.pdf")
if(file.exists(plotfile)){
  knitr::include_graphics(plotfile)
}

Condition number plots:

1) Average condition numbers of the reference matrices used by different algorithms (models).

2) Average score vs condition number of the model

3) Standard deviation of the score of the model across multiple repetitions vs the variation of the model's condition number across repetitions

Bootstrapped Bulks

plotfile <- paste0(plot.dir, "/bootstrap_plot_real.pdf")
if(file.exists(plotfile)){
  knitr::include_graphics(plotfile)
}

Bootstrapping performed to test the robustness of the chosen correlation. After deconvolution, a number of bulks is repeatedly chosen at random (with replacement) and the score is calculated each time. This gives an estimate of how robust the deconvolution result is. A large variance indicates that the correlation may be driven by outliers.

Simulations

Bulk simulations

Score plot

plotfile <- paste0(plot.dir, "/score_plot_simulated.pdf")
if(file.exists(plotfile)){
  knitr::include_graphics(plotfile)
}

Show/hide description

Deconvolution result for simulated bulks per algorithm and cell type.
Celltype order (left to right): r celltype.order.sim
The value is the pearson correlation coefficient, coded by rectangle size and color (small to big, red to green with increasing performance). Negative and undefined scores were set to 0. The overall column contains the average performance of each algorithm across all cell types. In case of multiple repetitions, the plotted scores are averages over all repetitions. The score specified for each algorithm corresponds to the 'overall' column. Uncertainty (standard deviation) of the overall performance is specified for each algorithm as well. Bulks were simulated by taking the sum over a fixed number of randomly sampled single-cell profiles. The distribution of cell types in the artificial bulks does not necessarily match the distribution of cell types in the single cell data.

Scatter plot of quantities

plotfile <- paste0(plot.dir, "/scatter_plots_simulated.pdf")
if(file.exists(plotfile)){
  knitr::include_graphics(plotfile)
}

Show/hide description

Comparison of the real cell type quantities in the simulated bulks (x-axis) and the estimated quantities per cell type (y-axis) for each algorithm. Dashed line marks the identity.

\pagebreak

Training set size simulation

Score boxplots

plotfile <- paste0(plot.dir, "/sample_plots_simulated.pdf")
if(file.exists(plotfile)){
  knitr::include_graphics(plotfile)
}

Show/hide description

Variability of performance of algorithms using different amounts of randomly selected training profiles for learning and reference profile creation. Overall denotes the average score of all cell types. Bulks used for deconvolution are the same as in the simulated score plot above.

Geneset simulation

Runtime lineplot

plotfile <- paste0(plot.dir, "/gene_time_plot_simulated.pdf")
if(file.exists(plotfile)){
  knitr::include_graphics(plotfile)
}

Show/hide description

Runtime of algorithms depending on the amount of genes supplied for deconvolution. Each algorithm can perform a separate feature selection on the given gene set to obtain the features used in the model.
Gene sets are ordered by the size of the intersect of features available in the gene set and the single-cell data.

Score lineplots

plotfile <- paste0(plot.dir, "/gene_score_plots_simulated.pdf")
if(file.exists(plotfile)){
  knitr::include_graphics(plotfile)
}

Show/hide description

Performance of algorithms depending on the gene set supplied for deconvolution. Each algorithm could perform a separate feature selection on the gene set to obtain the features used in the model. Gene sets are ordered by the size of the intersect of features available in the gene set and the single-cell data. Score is the pearson correlation coefficient, averaged over multiple repetitions. Errorbars show standard deviation of the score across repetitions.

Subtype simulation

Score plot

plotfile <- paste0(plot.dir, "/subtype_plot.pdf")
if(file.exists(plotfile)){
  knitr::include_graphics(plotfile)
}

Show/hide description

Average performance of the different algorithms depending on cell type granularity. Cell types were simulated from single-cell data using hierarchical clustering on a umap embedding of the data. The number of distinct cell types increases from left to right. Quantities of subtypes of the same cell type obtained by deconvolution are combined and their sum is compared to the quantities of the coarse cell types (classification that was contained in the single-cell pheno data).

Input data

Show/hide

Parameters of the benchmark call:

function.call.list <- as.list(function.call)

adjusted.names <- sapply(
  X = names(function.call.list),
  FUN = function(x) {
      if (x != "") {
        return(paste0(x, " ="))
      }else{
        return(x)
      }
    },
  USE.NAMES = FALSE
  )
print(paste(adjusted.names, function.call.list))

Single-cell data contained r nrow(sc.counts) features for r ncol(sc.counts) cells.

Corresponding phenotype data contained labels for r length(unique(sc.pheno[[celltype.col]])) cell types:

  t <- table(sc.pheno[[celltype.col]])
  print(paste(names(t), t, sep = ": "))

Used algorithms: r algorithm.names

r ncol(real.counts) real bulk profiles were available, with r length(intersect(rownames(real.counts), rownames(sc.counts))) overlapping features with provided single-cell data. (Pre-trained models are not considered here.)

Ground-truth cell type quantities were available for r nrow(real.props) cell types.

    if (exists("real.props") && !is.null(real.props) && length(real.props) > 0) {
    print(t(apply(real.props, 1, summary)))
    }

\pagebreak

MarianSchoen/DMC documentation built on Aug. 2, 2022, 3:05 p.m.