In wiesenfa/challengeR: Analyzing assessment data of biomedical image analysis competitions and visualization of results

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)

Introduction

This tutorial intends to give customized scripts to generate reports quickly, without going through all the installation and usage steps given in the README in detail.

The tutorial contains the following scripts, which are included in the "vignettes" directory:

• SingleTask_aggregate-then-rank.R
• MultiTask_rank-then-aggregate.R
• MultiTask_test-then-rank.R

How to use the tutorial scripts in RStudio:

1. Specify where the report should be generated.

setwd("myWorkingDirectoryFilePath")

1. Open the script.

2. Click "Source".

3. The report will be generated in the previously specified working directory.

4. Check out the report, adapt the script to fit your configuration.

Usage

Each script contains the following steps, as described in the README:

1. Load package

2. Load data (generated randomly)

3. Perform ranking

4. Uncertainty analysis (bootstrapping)

5. Generate report

The scrips will be now explained in more detail:

• SingleTask_aggregate-then-rank.R: In this script a single-task evaluation will be performed. The applied ranking method is "metric-based aggregation". It begins by aggregating metric values across all test cases for each algorithm. This aggregate is then used to compute a rank for each algorithm.

• MultiTask_rank-then-aggregate.R: In this script a multi-task evaluation will be performed. The applied ranking method is "case-based aggregation". It begins with computing a rank for each test case for each algorithm (”rank first”). The final rank is based on the aggregated test-case ranks. Distance-based approaches for rank aggregation can also be used.

• MultiTask_test-then-rank.R: In this script a multi-task evaluation will be performed. The applied ranking method is "significance ranking". In a complementary approach, statistical hypothesis tests are computed for each possible pair of algorithms to assess differences in metric values between the algorithms. Then ranking is performed according to the resulting relations or according to the number of significant one-sided test results. In the latter case, if algorithms have the same number of significant test results then they obtain the same rank. Various test statistics can be used.

For more hints, see the README and the package documentation.

wiesenfa/challengeR documentation built on Aug. 25, 2023, 6:43 a.m.