vignettes/vignette_4_quality.md

Quality assessment of multivariable prediction models

For systematic reviews of multivariable prediction models the “Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis” (TRIPOD) and “Prediction model Risk Of Bias ASsessment Tool” (PROBAST) guidelines should be used to assess the reporting and methodological quality of the predictive models developed.

Displaying this output for papers can be a challenging task, particularly as plots. predictr provides several functions to aid in this.

Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD)

Let’s use an inbuilt example from predictr of papers which have been assessed against the TRIPOD statement.

Note there are 3 key categories of columns here:

  1. ID column (“score”): This is needed to identify which paper is referred to in each row and must be unique for each row.

  2. Type column (“type”): This is needed to identify what type of models are done in the paper (is it derivation and/or validation of a model). This is essential as this changes what items are needed to be assessed in the TRIPOD statement.

  3. This must only contain 3 values: “d” = derivation only, “dv” = derivation and validation" or “v” = validation only.

  4. “item_” columns: These refer to each respective item listed in the TRIPOD statement (item 1 to 22).

  5. The names of these columns must be exactly the same as this example or else the subsequent function will not work.

  6. This must only contain 4 values as specified in the TRIPOD statement (“Yes”, “Partial”, “No”, or “Not applicable”).

predictr::example_tripod %>% head(10) %>% knitr::kable()
score type item\_1 item\_2 item\_3a item\_3b item\_4a item\_4b item\_5a item\_5b item\_5c item\_6a item\_6b item\_7a item\_7b item\_8 item\_9 item\_10a item\_10b item\_10c item\_10d item\_10e item\_11 item\_12 item\_13a item\_13b item\_13c item\_14a item\_14b item\_15a item\_15b item\_16 item\_17 item\_18 item\_19a item\_19b item\_20 item\_21 item\_22 n B1 dvi Partial (P) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) No (N) Yes (Y) No (N) Yes (Y) No (N) Yes (Y) Yes (Y) Yes (Y) Partial (P) No (N) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Partial (P) No (N) Yes (Y) Yes (Y) Yes (Y) Yes (Y) No (N) Partial (P) 1 B2 d No (N) Yes (Y) Yes (Y) Yes (Y) Yes (Y) No (N) Yes (Y) Yes (Y) Yes (Y) Yes (Y) No (N) Yes (Y) No (N) Yes (Y) No (N) Yes (Y) Yes (Y) Not applicable (NA) Partial (P) Not applicable (NA) Yes (Y) Not applicable (NA) Yes (Y) Yes (Y) Not applicable (NA) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Partial (P) Not applicable (NA) Yes (Y) Not applicable (NA) Yes (Y) Yes (Y) Yes (Y) Yes (Y) 2 A1 dvi Partial (P) Partial (P) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Partial (P) Yes (Y) Yes (Y) Yes (Y) No (N) Yes (Y) No (N) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) No (N) Partial (P) Partial (P) Partial (P) Yes (Y) No (N) Yes (Y) Yes (Y) Partial (P) Partial (P) Yes (Y) No (N) Yes (Y) Yes (Y) Yes (Y) Yes (Y) No (N) Yes (Y) 1 F1 d Partial (P) Partial (P) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Partial (P) Partial (P) Partial (P) No (N) Yes (Y) No (N) Yes (Y) No (N) Yes (Y) Partial (P) Not applicable (NA) Partial (P) Not applicable (NA) Yes (Y) Not applicable (NA) Partial (P) Partial (P) Not applicable (NA) Yes (Y) Yes (Y) Yes (Y) Partial (P) No (N) Not applicable (NA) Partial (P) Not applicable (NA) Yes (Y) Yes (Y) No (N) No (N) 1 A2 dvi Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Partial (P) Yes (Y) Partial (P) Yes (Y) No (N) Yes (Y) No (N) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) No (N) Yes (Y) No (N) Partial (P) Yes (Y) No (N) Partial (P) Yes (Y) No (N) Partial (P) Yes (Y) No (N) Yes (Y) Yes (Y) Yes (Y) Yes (Y) No (N) Yes (Y) 2 M1 dvi Partial (P) Partial (P) Yes (Y) Partial (P) Yes (Y) No (N) Partial (P) Partial (P) Partial (P) Yes (Y) Yes (Y) Partial (P) Yes (Y) Yes (Y) No (N) Partial (P) Partial (P) Partial (P) Partial (P) No (N) No (N) No (N) Partial (P) Partial (P) No (N) No (N) Partial (P) No (N) No (N) Yes (Y) No (N) Partial (P) Partial (P) Yes (Y) Partial (P) Yes (Y) No (N) 1 M2 d Partial (P) Partial (P) Yes (Y) Partial (P) Yes (Y) No (N) Yes (Y) Yes (Y) Partial (P) Yes (Y) Yes (Y) Partial (P) Yes (Y) Yes (Y) No (N) Partial (P) Partial (P) Not applicable (NA) Partial (P) Not applicable (NA) No (N) Not applicable (NA) Partial (P) Partial (P) Not applicable (NA) Yes (Y) Partial (P) No (N) No (N) Yes (Y) Not applicable (NA) Partial (P) Not applicable (NA) Yes (Y) Partial (P) Yes (Y) No (N) 2 J1 dvi Partial (P) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Partial (P) Yes (Y) Partial (P) Yes (Y) No (N) Partial (P) No (N) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) No (N) Yes (Y) No (N) Partial (P) Yes (Y) No (N) Yes (Y) Yes (Y) Partial (P) Partial (P) Yes (Y) No (N) Yes (Y) Partial (P) Yes (Y) Yes (Y) No (N) Yes (Y) 1 S1 d Partial (P) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Partial (P) Yes (Y) No (N) Yes (Y) No (N) Yes (Y) No (N) Yes (Y) Yes (Y) Not applicable (NA) Yes (Y) Not applicable (NA) No (N) Not applicable (NA) Yes (Y) Yes (Y) Not applicable (NA) Yes (Y) Yes (Y) Yes (Y) Partial (P) Yes (Y) Not applicable (NA) Yes (Y) Not applicable (NA) Yes (Y) Yes (Y) No (N) Yes (Y) 1 A3 dvi Partial (P) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Partial (P) Yes (Y) Partial (P) Yes (Y) Yes (Y) Partial (P) Yes (Y) Yes (Y) Partial (P) Yes (Y) Yes (Y) Yes (Y) Yes (Y) No (N) Partial (P) Partial (P) Partial (P) Partial (P) No (N) Yes (Y) Yes (Y) Yes (Y) Partial (P) Yes (Y) No (N) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) 3

Once your TRIPOD data matches the format of predictr::example_tripod, this can be put into the tripod_format() function to prepare this for plotting.

tripod_prep <- predictr::example_tripod %>%
  tripod_format(id = "score", type = "type")

knitr::kable(head(tripod_prep, 10))
id type item value B1 Derivation + Validation (Internal) 1 Partial B1 Derivation + Validation (Internal) 2 Yes B1 Derivation + Validation (Internal) 3a Yes B1 Derivation + Validation (Internal) 3b Yes B1 Derivation + Validation (Internal) 4a Yes B1 Derivation + Validation (Internal) 4b Yes B1 Derivation + Validation (Internal) 5a Yes B1 Derivation + Validation (Internal) 5b Yes B1 Derivation + Validation (Internal) 5c Yes B1 Derivation + Validation (Internal) 6a Yes

Now we have the data in the required format, we can then plot it using tripod_plot(). This produces a high quality ggplot of the TRIPOD results.

 predictr::example_tripod %>%
   predictr::tripod_format(id = "score", type = "type") %>%
   predictr::tripod_plot()

Prediction model Risk Of Bias ASsessment Tool (PROBAST)

Let’s use an inbuilt example from predictr of papers which have been assessed against the PROBAST tool.

Like the example above for TRIPOD, there are 4 key categories of columns here:

  1. ID column (“score”): This is needed to identify which paper is referred to in each row and must be unique for each row.

  2. Type column (“type”): This is needed to identify what type of models are done in the paper (is it derivation and/or validation of a model). This is essential as this changes what items are needed to be assessed in the PROBAST tool.

  3. This must only contain 3 values: “d” = derivation only, “dv” = derivation and validation" or “v” = validation only.

  4. “rob_” columns: These refer to each respective risk of bias (ROB) assessment using PROBAST for each domain (1 = “Participants”, 2 = “Predictors”, 3 = “Outcome”, and 4 = “Analysis”) and the respective screening questions.

  5. The names of these columns must be exactly the same as this example or else the subsequent function will not work.

  6. This must only contain 5 values as specified in the PROBAST tool (“yes,” “probably yes,” “probably no,” “no,” or “no information”).

  7. “app_” columns: These refer to each respective applicability assessment using PROBAST for each domain (1 = “Participants”, 2 = “Predictors”, and 3 = “Outcome”).

  8. The names of these columns must be exactly the same as this example or else the subsequent function will not work.

  9. This must only contain 3 values as specified in the PROBAST tool (low, unclear, high). Note this assessment is specifically “concerns regarding applicability”, rather than whether the model is applicable.

  10. The 4th domain (“Analysis”) is not assessed in PROBAST regarding “concerns regarding applicability”.

predictr::example_probast %>% head(10) %>% knitr::kable()
score type rob\_1\_1 rob\_1\_2 rob\_2\_1 rob\_2\_2 rob\_2\_3 rob\_3\_1 rob\_3\_2 rob\_3\_3 rob\_3\_4 rob\_3\_5 rob\_3\_6 rob\_4\_1 rob\_4\_2 rob\_4\_3 rob\_4\_4 rob\_4\_5 rob\_4\_6 rob\_4\_7 rob\_4\_8 rob\_4\_9 app\_1 app\_2 app\_3 n B1 d Probably yes (PY) Probably yes (PY) Probably yes (PY) Probably yes (PY) Yes (Y) Yes (Y) No (N) Yes (Y) Probably yes (PY) Probably no (PN) Probably yes (PY) Yes (Y) Yes (Y) No information (NI) No information (NI) No (N) No information (NI) Yes (Y) No (N) Yes (Y) Low Low Low 1 B2 dvi Probably yes (PY) Probably yes (PY) Probably yes (PY) Probably yes (PY) Yes (Y) Yes (Y) No (N) Yes (Y) Probably yes (PY) Probably no (PN) Probably yes (PY) Yes (Y) Yes (Y) No information (NI) No information (NI) No (N) No information (NI) Yes (Y) No (N) Yes (Y) Low Low Low 2 A1 dve Probably yes (PY) Probably yes (PY) Yes (Y) Probably yes (PY) Yes (Y) Yes (Y) No (N) Yes (Y) Probably yes (PY) No information (NI) Yes (Y) Yes (Y) Yes (Y) No (N) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) High Low High 1 F1 dvie Probably yes (PY) No (N) Probably yes (PY) Probably no (PN) Yes (Y) Yes (Y) No (N) Yes (Y) Probably yes (PY) Probably no (PN) Yes (Y) Yes (Y) Yes (Y) No information (NI) No information (NI) No (N) No information (NI) Yes (Y) Yes (Y) Yes (Y) Low Low Low 1 A2 ve Probably yes (PY) Probably yes (PY) Yes (Y) Probably yes (PY) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Probably yes (PY) No information (NI) Yes (Y) Yes (Y) Yes (Y) No (N) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Low Low Low 2 M1 d Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Probably yes (PY) Yes (Y) Yes (Y) Probably yes (PY) Yes (Y) No information (NI) No information (NI) Yes (Y) No information (NI) Yes (Y) No information (NI) Yes (Y) Low Low High 1 M2 dvi Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) Yes (Y) No information (NI) No information (NI) No (N) No information (NI) Yes (Y) No (N) No information (NI) High Low Low 2 J1 dve Probably yes (PY) Yes (Y) Yes (Y) Probably yes (PY) Yes (Y) Yes (Y) No (N) Yes (Y) Yes (Y) Probably no (PN) Yes (Y) Yes (Y) Yes (Y) No (N) No information (NI) No (N) No (N) Yes (Y) No (N) Yes (Y) High Low High 1 S1 dvie Yes (Y) No (N) Probably yes (PY) Probably yes (PY) Yes (Y) Probably yes (PY) Yes (Y) Yes (Y) Probably yes (PY) Probably no (PN) Yes (Y) Yes (Y) Yes (Y) No (N) No information (NI) No (N) No information (NI) Yes (Y) No (N) Yes (Y) Low Low Low 1 A3 ve Yes (Y) Yes (Y) Probably yes (PY) Probably yes (PY) Yes (Y) Probably yes (PY) Yes (Y) Yes (Y) Probably yes (PY) Probably no (PN) Yes (Y) Yes (Y) Yes (Y) No (N) Yes (Y) Probably no (PN) Yes (Y) Yes (Y) Probably no (PN) Yes (Y) Low Low Low 3

Once your TRIPOD data matches the format of predictr::example_tripod, this can be put into the tripod_format() function to prepare this for plotting.

probast_prep <- predictr::example_probast %>%
  probast_format(id = "score", type = "type")

knitr::kable(head(probast_prep, 10))
id type domain subdomain assess item value B1 Derivation Risk of Bias Participants Low 1 PY B1 Derivation Risk of Bias Participants Low 2 PY B1 Derivation Risk of Bias Predictors Low 1 PY B1 Derivation Risk of Bias Predictors Low 2 PY B1 Derivation Risk of Bias Predictors Low 3 Y B1 Derivation Risk of Bias Outcome High 1 Y B1 Derivation Risk of Bias Outcome High 2 N B1 Derivation Risk of Bias Outcome High 3 Y B1 Derivation Risk of Bias Outcome High 4 PY B1 Derivation Risk of Bias Outcome High 5 PN

Now we have the data in the required format, we can then plot it using probast_plot(). This produces a high quality ggplot of the PROBAST results.

 predictr::example_probast %>%
  probast_format(id = "score", type = "type") %>%
  probast_plot()

For PROBAST, the risk of bias (ROB) assessment is a composite assessment across all screening questions in the domain. This is calculated automatically by the probast_plot() function. However, if you want that ROB screening question data for a supplementary table, then you can use the probast_table() function.

 predictr::example_probast %>%
  probast_format(id = "score", type = "type") %>%
  probast_table() %>%  head(10) %>% knitr::kable()
id type Participants 1 Participants 2 Predictors 1 Predictors 2 Predictors 3 Outcome 1 Outcome 2 Outcome 3 Outcome 4 Outcome 5 Outcome 6 Analysis 1 Analysis 2 Analysis 3 Analysis 4 Analysis 5 Analysis 6 Analysis 7 Analysis 8 Analysis 9 Overall NA B1 Derivation PY PY PY PY Y Y N Y PY PN PY Y Y NI NI N NI Y N Y NA B2 Derivation + Validation (Internal) PY PY PY PY Y Y N Y PY PN PY Y Y NI NI N NI Y N Y NA A1 Derivation + Validation (External) PY PY Y PY Y Y N Y PY NI Y Y Y N Y Y Y Y Y Y NA F1 Derivation + Validation (Both) PY N PY PN Y Y N Y PY PN Y Y Y NI NI N NI Y Y Y NA A2 Validation (External) PY PY Y PY Y Y Y Y PY NI Y Y Y N Y Y Y Y Y Y NA M1 Derivation Y Y Y Y Y Y Y Y PY Y Y PY Y NI NI Y NI Y NI Y NA M2 Derivation + Validation (Internal) Y Y Y Y Y Y Y Y Y Y Y Y Y NI NI N NI Y N NI NA J1 Derivation + Validation (External) PY Y Y PY Y Y N Y Y PN Y Y Y N NI N N Y N Y NA S1 Derivation + Validation (Both) Y N PY PY Y PY Y Y PY PN Y Y Y N NI N NI Y N Y NA A3 Validation (External) Y Y PY PY Y PY Y Y PY PN Y Y Y N Y PN Y Y PN Y NA

kamclean/predictr documentation built on Aug. 14, 2022, 4:35 a.m.