Using ANOVA to Estimate Limits of Agreement for MRMC study"

In iMRMC: Multi-Reader, Multi-Case Analysis Methods (ROC, Agreement, and Other Metrics)

library(knitr)
library(iMRMC)
knitr::opts_chunk$set(echo = TRUE)

Here we demonstrate how to run the limits of agreement functions (laWRBM and laBRBM), and we show the output.

1. Definition

Let $X_{ijk}$ denote the score for case $k$ $(k=1,...,K)$ from the reader $j$ $(j=1,..,J)$ under modality $i$ $(i=1,2)$ in a Multi-reader Multi-case study, where $i=1$ and $i=2$ indicate test modality and reference modality respectively. The difference score between the two modalities is given by $D_{jj'k}^{12}=X_{1jk}-X_{2j'k}$.

If $j=j'$, the difference score $D_{jj,k}^{12}$, or simply $Y_{jk}$, denotes the within-reader between-modality (WRBM) difference. That is, the difference score from the same reader under different modalities. Given the mean difference $\overline{D_{WR}^{12}}=E[Y_{jk}]$ and the variance of the difference $V_{WR}^{12}=Var[Y_{jk}]$, the WRBM limits of agreement are defined as [ \overline{D_{WR}^{12}}\pm2\sqrt{V_{WR}^{12}}. ]

If $j\neq j'$, the difference score $D_{jj'k}^{12}$ denotes the between-reader between-modality (BRBM) difference. Similar to above, we have a mean difference $\overline{D_{BR}^{12}}=E[D_{jj'k}^{12}]$ and a variance of the difference $V_{BR}^{12}=Var[D_{jj'k}^{12}]$. Then, the BRBM limits of agreement are defined as [ \overline{D_{BR}^{12}}\pm2\sqrt{V_{BR}^{12}}. ]

To construct the WRBM and BRBM limits of agreement, we need to estimate $\overline{D_{WR}^{12}}$, $V_{WR}^{12}$, $\overline{D_{BR}^{12}}$, and $V_{BR}^{12}$. The two mean differences are easy to estimate: [ \hat{\overline{D_{WR}^{12}}}=\frac{1}{JK}\sum_j\sum_kD_{jj,k}^{12}=\frac{1}{JK}\sum_j\sum_k(X_{1jk}-X_{2jk})=\overline{X_1..}-\overline{X_2..}, ] where sums are over all samples implied by the index unless otherwise described. Also, $\overline{X_i..}=\frac{1}{JK}\sum_j\sum_kX_{ijk}$ for $i=1,2$ denotes the average score across all the readers and cases for a single modality. Similarly, [ \hat{\overline{D_{BR}^{12}}}=\frac{1}{J(J-1)K}\sum_j\sum_{j\neq j'}\sum_kD_{jj'k}^{12}=\frac{1}{J(J-1)K}\sum_j\sum_{j\neq j'}\sum_k(X_{1jk}-X_{2j'k})=\overline{X_1..}-\overline{X_2..}. ] Therefore, $\hat{\overline{D_{WR}^{12}}}=\hat{\overline{D_{BR}^{12}}}=\overline{X_1..}-\overline{X_2..}$. The WRBM and BRBM limits of agreement will be different only by the variances $V_{WR}^{12}$ and $V_{BR}^{12}$. In the following two sections, we will discuss how to use two-way random effect ANOVA to estimate $V_{WR}^{12}$ and three-way mixed effect ANOVA to estimate $V_{BR}^{12}$.

2. Using two-way random effect ANOVA to estimate $V_{WRBM}$

To estimate $V_{WR}^{12}=Var[Y_{jk}]$, we build up a two-way random effect model for the WRBM difference $Y_{jk}$: [ Y_{jk} = \mu+R_j+C_k+\varepsilon_{jk}, ] where $R_j\sim N(0,\sigma_R^2)$, $C_k\sim N(0,\sigma_C^2)$, and $\varepsilon_{jk}\sim N(0,\sigma_\varepsilon^2)$ are independent random variables. Then, the variance of $Y_{jk}$ can be expressed as [ V_{WR}^{12}=Var[Y_{jk}]=Var(R_j+C_k+\varepsilon_{jk})=\sigma_R^2+\sigma_C^2+\sigma_\varepsilon^2 ]

Here is the two-way random effect ANOVA table:

Source DF Sum of Square (SS) Mean Square (MS) E(MS)
------- ----------- ---------------------------- ---------------------- -------- Reader $J-1$ $SSR=K\sum_j(\overline{Y_{j.}}-\overline{Y_{..}})^2$ $MSR=SSR/(J-1)$ $\sigma_\varepsilon^2+K\sigma_R^2$ Case $K-1$ $SSC=J\sum_k(\overline{Y_{.k}}-\overline{Y_{..}})^2$ $MSR=SSC/(K-1)$ $\sigma_\varepsilon^2+J\sigma_C^2$ Error $(J-1)(K-1)$ $SSE=SST-SSR-SSC$ $MSE=SSE/(J-1)(K-1)$ $\sigma_\varepsilon^2$ Total $JK-1$ $SST=\sum_j\sum_k(Y_{jk}-\overline{Y_{..}})^2$ ------- ----------- ---------------------------- ---------------------- --------

In the table above, $\overline{Y_{j.}}=\frac{1}{K}\sum_kY_{jk}$, $\overline{Y_{.k}}=\frac{1}{J}\sum_jY_{jk}$, $\overline{Y_{..}}=\overline{D_{WR}^{12}}$ are the marginals and overall mean of difference score. Hence, the sum of squares (SS) and mean squares (MS) can be calculated from the data for each effect. From the last column of the ANOVA table, we can find the relationship between the variance components ($\sigma_R^2$, $\sigma_C^2$, $\sigma_\varepsilon^2$) and the mean squares. The unbiased estimates of the the variance components are [ \begin{aligned} \hat\sigma_\varepsilon^2&=MSE,\ \hat\sigma_R^2&=\frac{MSR-MSE}{K}, and \ \hat\sigma_C^2&=\frac{MSC-MSE}{J}. \end{aligned} ]

Putting it all together, the estimate of the variance of $Y_{jk}$, [ \hat V_{WR}^{12}=\hat Var[Y_{jk}]=\hat\sigma_R^2+\hat\sigma_C^2+\hat\sigma_\varepsilon^2=\frac{1}{JK}(JMSR+KMSC+(JK-J-K)*MSE) ]

The following is an example demonstrating the use of the function laWRBM to compute the WRBM limits of agreement for simulated MRMC agreement data. The example starts by simulating MRMC agreement data with the function sim.NormalIG.Hierarchical and finishes by printing the output.

# Simulate MRMC data
config <- sim.NormalIG.Hierarchical.config(modalityID = c("testA","testB"))
set.seed(1)
data.sim <- sim.NormalIG.Hierarchical(config)

# Using ANOVA to calculate WRBM limits of agreement
laWRBM_result <- laWRBM(data.sim)
print(laWRBM_result)

The output is a list of two objects. The first object in the list is a data frame with key summary statistics. In order, the key summary statstics are the mean difference, the variance of the mean difference, the variance of WRBM differences $\hat V_{WR}^{12}$ (1 observation), the confidence interval of the mean difference, and the limits of agreement.

The second object in the list is a data frame containing the typical ANOVA table.

3. Using three-way mixed effect ANOVA to calculate $V_{WRBM}$

To estimate $V_{BR}^{12}=Var[D_{jj'k}^{12}]$, we build up a three-way mixed effect model for the score $X_{ijk}$: [ X_{ijk} = \mu+m_i+R_j+C_k+RC_{jk}+mR_{ij}+mC_{ik}+\varepsilon_{ijk}, ] where $m_i$ denotes the fixed effect for modality, $\sum_im_i=0$, and the other variables are independently normally distributed: $R_j\sim N(0,\sigma_R^2)$, $C_k\sim N(0,\sigma_C^2)$, $RC_{jk}\sim N(0,\sigma_{RC}^2)$, $mR_{ij}\sim N(0,\sigma_{mR}^2)$, $mC_{ik}\sim N(0,\sigma_{mC}^2)$, and $\varepsilon_{ijk}\sim N(0,\sigma_\varepsilon^2)$.

The mixed effect model we applied is the unrestricted mixed model. Unlike the restricted model, we do not force the sum over modalities of the modality-reader terms to equal zero, and we do not force the sum over modalities of the modality-case terms to equal zero. This makes the model easier to work with.

Under this model, the BRBM difference $D_{jj'k}^{12}$ can be expressed as [ D_{jj'k}^{12}=X_{1jk}-X_{2j'k}=m_1-m_2+R_j-R_j'+RC_{jk}-RC_{j'k}+mR_{1j}-mR_{2j'}+mC_{1k}-mC_{2k}+\varepsilon_{1jk}-\varepsilon_{2j'k}. ] The variance of $D_{jj'k}^{12}$ is the following: [ V_{BR}^{12}=Var[D_{jj'k}^{12}]=2\sigma_R^2+2\sigma_{RC}^2+2\sigma_{mR}^2+2\sigma_{mC}^2+2\sigma_\varepsilon^2. ]

The three-way mixed effect ANOVA table is given by

Source DF Sum of Square (SS) E(MS)
------- ----------- ---------------------------- ----------------------- Modality $I-1$ $SSM=JK\sum_i(\overline{X_{i..}}-\overline{X_{...}})^2$ $\sigma_\varepsilon^2+K\sigma_{mR}^2+J\sigma_{mC}^2+\frac{JK}{I-1}\sum_im_i^2$ Reader $J-1$ $SSR=IK\sum_j(\overline{X_{.j.}}-\overline{X_{...}})^2$ $\sigma_\varepsilon^2+I\sigma_{RC}^2+K\sigma_{mR}^2+IK\sigma_R^2$ Case $K-1$ $SSC=IJ\sum_k(\overline{X_{..k}}-\overline{X_{...}})^2$ $\sigma_\varepsilon^2+I\sigma_{RC}^2+J\sigma_{mC}^2+IJ\sigma_C^2$ Reader:Case $(J-1)(K-1)$ $SSRC=I\sum_j\sum_k(\overline{X_{.jk}}-\overline{X_{.j.}}-\overline{X_{..k}}+\overline{X_{...}})^2$ $\sigma_\varepsilon^2+I\sigma_{RC}^2$ Reader:Modality $(J-1)(I-1)$ $SSMR=K\sum_i\sum_j(\overline{X_{ij.}}-\overline{X_{i..}}-\overline{X_{.j.}}+\overline{X_{...}})^2$ $\sigma_\varepsilon^2+K\sigma_{mR}^2$ Case:Modality $(K-1)(I-1)$ $SSMC=J\sum_i\sum_k(\overline{X_{i.k}}-\overline{X_{i..}}-\overline{X_{..k}}+\overline{X_{...}})^2$ $\sigma_\varepsilon^2+J\sigma_{mC}^2$ Error $df_{E}$ $SSE=SST-other SS$ $\sigma_\varepsilon^2$ Total $IJK-1$ $SST=\sum_i\sum_j\sum_k(X_{ijk}-\overline{X_{...}})^2$ ------- ----------- ---------------------------- -----------------------

where $\overline{X_{i..}}$, $\overline{X_{.j.}}$, $\overline{X_{..k}}$, $\overline{X_{ij.}}$, $\overline{X_{i.k}}$, $\overline{X_{.jk}}$, and $\overline{X_{...}}$ are marginal means and overall means of the score $X_{ijk}$. The $df_{E}$ denotes the degrees of freedom for the error, $df_{E}=IJK-IJ-JK-IK+I+J+K-1$. Similar to the two-way ANOVA table, the last column shows the relationship between the variance components and the mean squares. Consequently, the unbiased estimates of the variance components are [ \begin{aligned} \hat\sigma_\varepsilon^2&=MSE,\ \hat\sigma_{RC}^2&=\frac{MSRC-MSE}{I},\ \hat\sigma_{mC}^2&=\frac{MSMC-MSE}{J},\ \hat\sigma_{mR}^2&=\frac{MSMR-MSE}{K},\ \hat\sigma_{R}^2&=\frac{MSR-MSRC-MSMR+MSE}{IK},\ \hat\sigma_{C}^2&=\frac{MSC-MSRC-MSMC+MSE}{IJ}.\ \end{aligned} ]

Putting it all together, the estimate of the variance BRBM differences is [ \begin{aligned} \hat V_{BR}^{12}&=\hat Var[D_{jj'k}^{12}]=2\hat\sigma_R^2+2\hat\sigma_{RC}^2+2\hat\sigma_{mR}^2+2\hat\sigma_{mC}^2+2\hat\sigma_\varepsilon^2\ &=\frac{2}{IJK}(JMSR+J(K-1)MSRC+J(I-1)MSMR\ &\ \ \ \ +IKMSMC+(IJK-IJ-IK-JK+J)*MSE). \end{aligned} ]

The following is an example demonstrating the use of the function laBRBM to compute the BRBM limits of agreement for simulated MRMC agreement data. The example uses the previously simulated MRMC agreement data. The output is a list of the same two objects as for laWRBM.

# Using ANOVA to calculate BRBM limits of agreement
laBRBM_result <- laBRBM(data.sim)
print(laBRBM_result)

4. Relationship between the two-way random effect ANOVA and the three-way mixed effect ANOVA

Since the WRBM difference score, $Y_{jk}=X_{1jk}-X_{2jk}$, is a linear combination of individual scores, the variance of $Y_{jk}$ can also be expressed by the mean squares in the three-way mixed effect ANOVA. First, we put the three-way ANOVA model into the WRBM difference score definition: [ Y_{jk}=X_{1jk}-X_{2jk}=m_1-m_2+mR_{1j}-mR_{2j}+mC_{1k}-mC_{2k}+\varepsilon_{1jk}-\varepsilon_{2jk}. ] Then, the variance of $Y_{jk}$ is [ V_{WR}^{12}=Var[Y_{jk}]=2\sigma_{mR}^2+2\sigma_{mC}^2+2\sigma_\varepsilon^2. ] Now, if we insert the unbiased estimates of the variance components, we get the estimate of $V_{WRBM}$ [ \hat V_{WR}^{12}=2\hat\sigma_{mR}^2+2\hat\sigma_{mC}^2+2\hat\sigma_\varepsilon^2 =\frac{2}{JK}(JMSMR+KMSMC+(JK-J-K)*MSE) ] When we compare this result to the one from the two-way random effect ANOVA, we notice that there is a linear relationship between the sums of squares in the two ANOVA models. In the following we use the subscripts "$2w$" and "$3w$" to denote the MS or SS for the two-way and three-way ANOVA models, respectively. [ \begin{aligned} SSR_{2w}&=K\sum_j(\overline{Y_{j.}}-\overline{Y_{..}})^2\ &=K\sum_j\left(\overline{X_{1j.}}-\overline{X_{2j.}}-\overline{X_{1..}}+\overline{X_{2..}}\right)^2\ &=K\sum_j[(\overline{X_{1j.}}-\overline{X_{1..}}-\overline{X_{.j.}}+\overline{X_{...}})^2+(\overline{X_{2j.}}-\overline{X_{2..}}-\overline{X_{.j.}}+\overline{X_{...}})^2\ &\ \ \ \ \ -2(\overline{X_{1j.}}-\overline{X_{1..}}-\overline{X_{.j.}}+\overline{X_{...}})(\overline{X_{2j.}}-\overline{X_{2..}}-\overline{X_{.j.}}+\overline{X_{...}})]\ &=K\sum_j\left[\sum_i(\overline{X_{ij.}}-\overline{X_{i..}}-\overline{X_{.j.}}+\overline{X_{...}})^2 +2\cdot\frac{1}{2}\cdot\frac{1}{2}(\overline{Y_{j.}}-\overline{Y_{..}})^2\right]\ &=SSMR_{3w}+\frac{1}{2}SSR_{2w} \end{aligned} ] Thus, [ SSR_{2w}=2SSMR_{3w} ] Similarly, [ SSC_{2w}=2SSMC_{3w} ] [ SST_{2w}=2\sum_i\sum_j\sum_k(X_{ijk}-\overline{X_{i..}}-\overline{X_{.jk}}+\overline{X_{...}})^2 ] For the total sum of square in the three-way ANOVA, [ \begin{aligned} SST_{3w}&=\sum_i\sum_j\sum_k(X_{ijk}-\overline{X_{...}})^2\ &=\sum_i\sum_j\sum_k\left[(X_{ijk}-\overline{X_{i..}}-\overline{X_{.jk}}+\overline{X_{...}})+(\overline{X_{i..}}-\overline{X_{...}})+(\overline{X_{.jk}}-\overline{X_{...}})\right]^2\ &=\sum_i\sum_j\sum_k(X_{ijk}-\overline{X_{i..}}-\overline{X_{.jk}}+\overline{X_{...}})^2+SSM_{3w}+I\sum_j\sum_k(\overline{X_{.jk}}-\overline{X_{...}})^2 \end{aligned} ] The last term on the right-hand-side of the formula above can be decomposed as [ \begin{aligned} &\ \ \ \ \ I\sum_j\sum_k(\overline{X_{.jk}}-\overline{X_{...}})^2\ &=I\sum_j\sum_k\left[(\overline{X_{.jk}}-\overline{X_{.j.}}-\overline{X_{..k}}+\overline{X_{...}})+(\overline{X_{.j.}}-\overline{X_{...}})+(\overline{X_{..k}}-\overline{X_{...}})\right]^2\ &=SSRC_{3w}+SSR_{3w}+SSC_{3w} \end{aligned} ] Thus, [ \begin{aligned} SSE_{2w} &= SST_{2w}-SSR_{2w}-SSC_{2w}\ &=2(SST_{3w}-SSM_{3w}-SSR_{3w}-SSC_{3w}-SSRC_{3w})-2SSMR_{3w}-2SSMC_{3w}\&=2SSE_{3w} \end{aligned} ]

Since $I=2$, the degrees of freedom is $(J-)(I-1)=J-1$ for $SSMR_{3w}$ and $(K-1)(I-1)=K-1$ for $SSMC_{3w}$. $df_E=IJK-IJ-JK-IK+I+J+K-1=JK-J-K+1$ is the degree of freedom for $SSE_{3w}$. We also have the same mean square relationship as that for the sum of squares: [ \begin{aligned} MSR_{2w}&=2MSMR_{3w},\ MSC_{2w}&=2MSMR_{3w},\ MSE_{2w}&=2MSE_{3w}.\ \end{aligned} ] Pulling together the degrees of freedom and the mean squares, we see that the variance estimate is the same from both models: [ \begin{aligned} \hat V_{WRBM}&=\frac{2}{JK}(JMSMR_{3w}+KMSMC_{3w}+(JK-J-K)MSE_{3w})\ &=\frac{1}{JK}(JMSR_{2w}+KMSC_{2w}+(JK-J-K)MSE_{2w}). \end{aligned} ]

iMRMC documentation built on Sept. 11, 2024, 7:12 p.m.