doIMRMC: MRMC analysis of the area under the ROC curve

In iMRMC: Multi-Reader, Multi-Case Analysis Methods (ROC, Agreement, and Other Metrics)

MRMC analysis of the area under the ROC curve

Description

doIMRMC takes ROC data as a data frame and runs a multi-reader multi-case analysis based on U-statistics as described in the following papers Gallas2006_Acad-Radiol_v13p353 (single-modality), Gallas2008_Neural-Networks_v21p387 (multiple modalities, arbitrary study designs), Gallas2009_Commun-Stat-A-Theor_v38p2586 (framework paper).

Usage

doIMRMC(
  data = NULL,
  fileName = NULL,
  workDir = NULL,
  iMRMCjarFullPath = NULL,
  stripDatesForTests = FALSE
)

Arguments

data	This data.frame contains the following variables: readerID Factor with levels like "reader1", "reader2", ... caseID Factor with levels like "case1", "case2", ... modalityID Factor with levels like "modality1", "modality2", ... score num reader score Each row of this data frame corresponds to an observation. For every caseID, there must be a row corresponding to the truth observation. The readerID for a truth observation is "truth". The modalityID for a truth observation is "truth". The score for a truth observation must be either 0 (signal-absent) or 1 (signal-present).
fileName	This character string identifies the location of an iMRMC input file. The input file is identical to data except there is a free text section to start, then a line with "BEGIN DATA:", then the data frame info.
workDir	This character string determines the directory where intermediate results are written. If this parameter is not set, the program writes the intermediate results to the directory specified by tempdir() and then deletes them.
iMRMCjarFullPath	This character string identifies the location of the iMRMC.jar file this jar file can be downloaded from https://github.com/DIDSR/iMRMC/releases this R program supports version iMRMC-v3p2.jar
stripDatesForTests	Since results include a date and time stamp, these need to be stripped out when doing the package tests. This parameter flags whether or not the dates should be stripped out.

Details

In detail, this procedure reads the name of an input file from the local file system, or takes a data frame and writes it to the local file system formatted for the iMRMC program (found at https://github.com/DIDSR/iMRMC/releases), it executes a java app, the iMRMC engine, which writes the results to the local files system, it reads the analysis results from the local file system, packs the analysis results into a list object, deletes the data and analysis results from the local file system, and returns the list object.

This software requires Java(>=8).

The examples took too long for CRAN to accept. So here is an example:

# Create a sample configuration file
config <- sim.gRoeMetz.config()
# Simulate an MRMC ROC data set
dFrame.imrmc <- sim.gRoeMetz(config)
# Analyze the MRMC ROC data
result <- doIMRMC(dFrame.imrmc)

Value

[list] iMRMC outputs. The objects of this list are described in detail in the iMRMC documentation which can be found at <http://didsr.github.io/iMRMC/000_iMRMC/userManualHTML/index.htm>

Here is a quick summary:

• perReader data.frame containing the performance results for each reader. Key variables of this data frame are AUCA, AUCB, AUCAminusAUCB and the corresponding variances, confidence intervals, degrees of freedom and p-values.

• Ustat data.frame containing the reader-average performance results. The analysis results are based on U-statistics and the papers listed above. Key variables of this data frame are AUCA, AUCB, AUCAminusAUCB and the corresponding variances, confidence intervals, degrees of freedom and p-values.

• MLEstat data.frame containing the reader-average performance results. The analysis results are based on V-statistics, which approximates the true distribution with the empirical distribution. The empirical distribution equals the nonparametric MLE estimate of the true distribution, which is also equivalent to the ideal bootstrap estimate. Please refer to the papers listed above. Key variables of this data frame are AUCA, AUCB, AUCAminusAUCB and the corresponding variances, confidence intervals, degrees of freedom and p-values.

• ROC list containing ROC curves There is an ROC curve for every combination of reader and modality. For every modality, there are also four average ROC curves. These are discussed in Chen2014_Br-J-Radiol_v87p20140016. The diagonal average averages the reader-specific ROC curves along y = -x + b for b in (0,1). The horizontal average averages the reader specific ROC curves along y = b for b in (0,1). The vertical average averages the reader specific ROC curves along x = b for b in (0,1). The pooled average ignores readerID and pools all the scores together to create one ROC curve.

• varDecomp list containing different decompositions of the total variance. Please refer to Gallas2009_Commun-Stat-A-Theor_v38p2586 (framework paper). The different decompositions are BCK, BDG, DBM, MS, OR.

iMRMC documentation built on May 31, 2023, 8:36 p.m.