as.challenge: Constructs a challenge object

In wiesenfa/challengeR: Analyzing assessment data of biomedical image analysis competitions and visualization of results

Constructs a challenge object

Description

Constructs an S3 object to represent the configuration of an assessment data set originating from a benchmarking competition (so-called "challenge").

Usage

as.challenge(
  object,
  case,
  algorithm,
  value,
  by = NULL,
  taskName = NULL,
  annotator = NULL,
  smallBetter = FALSE,
  na.treat = NULL,
  check = TRUE
)

Arguments

object	A data frame containing the assessment data.
case	A string specifying the name of the column that contains the case identifiers.
algorithm	A string specifying the name of the column that contains the algorithm identifiers.
value	A string specifying the name of the column that contains the performance values.
by	A string specifying the name of the column that contains the task identifiers. Required for multi-task data set.
taskName	A string specifying the task name for single-task data set that does not contain a task column. This argument is optional for a single-task data set and is ignored for a multi-task data set.
annotator	If multiple annotators annotated the test cases, a string specifying the name of the column that contains the annotator identifiers. Only applies to rang-then-aggregate. Use with caution: Currently not tested.
smallBetter	A boolean specifying whether small performance values indicate better algorithm performance.
na.treat	Indicates how missing perfomance values are treated if sanity check is enabled. It can be 'na.rm', numeric value or function. For a numeric value or function, NAs will be replaced by the specified values. For 'na.rm', rows that contain missing values will be removed.
check	A boolean to indicate to perform a sanity check of the specified data set and arguments if set to TRUE.

Value

An S3 object to represent the configuration of an assessment data set.

Assessment data set

The toolkit provides visualization approaches for both challenges designed around a single task (single-task challenges) and for challenges comprising multiple tasks (multi-task challenges). For a single-task challenge, the assessment data set (argument object) requires the following columns:

• test case identifier (string or numeric)

• algorithm identifier (string or numeric)

• performance value (numeric)

For a multi-task challenge, the assessment data set (argument object) requires the following columns:

• task identifier (string or numeric)

• test case identifier (string or numeric)

• algorithm identifier (string or numeric)

• performance value (numeric)

Sanity check

It is highly recommended that the sanity check is not disabled when the data set is provided initially. It checks that:

• performance values are numeric (if not, raises error)

• algorithm performances are observed for all cases (if not, adds them as NA and emits a message)

• cases appear only once for the same algorithm (if not, raises error)

If the argument na.treat for treatment of NA is specified, NAs will be handled respectively.

It might be reasonable to disable the sanity check for further computations (e.g., for performance reasons during bootstrapping (bootstrap.ranked.list) where cases are actually allowed to appear more than once for the same algorithm).

Examples

# single-task data set

# multi-task data set

wiesenfa/challengeR documentation built on Aug. 25, 2023, 6:43 a.m.