In wiesenfa/challengeR: Analyzing assessment data of biomedical image analysis competitions and visualization of results

Ranking stability: Ranking variability via bootstrap approach

A blob plot of bootstrap results over the different tasks separated by algorithm allows another perspective on the assessment data. This gives deeper insights into the characteristics of tasks and the ranking uncertainty of the algorithms in each task.

\bigskip

#stabilityByAlgorithm.bootstrap.list
if (n.tasks<=6 & n.algorithms<=10 ){
  stabilityByAlgorithm(boot_object,
                       ordering=ordering_consensus,
                       max_size = 9,
                       size=4,
                       shape=4,
                       single = F) + 
    scale_color_manual(values=cols) +
    guides(color = 'none')
} else {
  pl=stabilityByAlgorithm(boot_object,
                          ordering=ordering_consensus,
                          max_size = 9,
                          size=4,
                          shape=4,
                          single = T)
  for (i in 1:length(pl)) print(pl[[i]] +
                                  scale_color_manual(values=cols) +
                                  guides(size = guide_legend(title="%"),color="none")
                                )
}

\newpage

An alternative representation is provided by a stacked frequency plot of the observed ranks, separated by algorithm. Observed ranks across bootstrap samples are displayed with coloring according to the task. For algorithms that achieve the same rank in different tasks for the full assessment data set, vertical lines are on top of each other. Vertical lines allow to compare the achieved rank of each algorithm over different tasks.

\bigskip

if (n.tasks<=6 & n.algorithms<=10 ){
  stabilityByAlgorithm(boot_object,
                       ordering=ordering_consensus,
                       stacked = TRUE,
                       single = F) 
} else {
  pl=stabilityByAlgorithm(boot_object,
                          ordering=ordering_consensus,
                          stacked = TRUE,
                          single = T) %++%
    theme(legend.position = ifelse(n.tasks>20,
                                   yes = "bottom",
                                   no = "right"))
  print(pl)
}

wiesenfa/challengeR documentation built on Aug. 25, 2023, 6:43 a.m.