For systematic reviews of multivariable prediction models the “Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis” (TRIPOD) and “Prediction model Risk Of Bias ASsessment Tool” (PROBAST) guidelines should be used to assess the reporting and methodological quality of the predictive models developed.
Displaying this output for papers can be a challenging task,
particularly as plots. predictr
provides several functions to aid in
this.
Let’s use an inbuilt example from predictr
of papers which have been
assessed against the TRIPOD statement.
Note there are 3 key categories of columns here:
ID column (“score”): This is needed to identify which paper is referred to in each row and must be unique for each row.
Type column (“type”): This is needed to identify what type of models are done in the paper (is it derivation and/or validation of a model). This is essential as this changes what items are needed to be assessed in the TRIPOD statement.
This must only contain 3 values: “d” = derivation only, “dv” = derivation and validation" or “v” = validation only.
“item_” columns: These refer to each respective item listed in the TRIPOD statement (item 1 to 22).
The names of these columns must be exactly the same as this example or else the subsequent function will not work.
This must only contain 4 values as specified in the TRIPOD statement (“Yes”, “Partial”, “No”, or “Not applicable”).
predictr::example_tripod %>% head(10) %>% knitr::kable()
score
type
item\_1
item\_2
item\_3a
item\_3b
item\_4a
item\_4b
item\_5a
item\_5b
item\_5c
item\_6a
item\_6b
item\_7a
item\_7b
item\_8
item\_9
item\_10a
item\_10b
item\_10c
item\_10d
item\_10e
item\_11
item\_12
item\_13a
item\_13b
item\_13c
item\_14a
item\_14b
item\_15a
item\_15b
item\_16
item\_17
item\_18
item\_19a
item\_19b
item\_20
item\_21
item\_22
n
B1
dvi
Partial (P)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
No (N)
Yes (Y)
No (N)
Yes (Y)
No (N)
Yes (Y)
Yes (Y)
Yes (Y)
Partial (P)
No (N)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Partial (P)
No (N)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
No (N)
Partial (P)
1
B2
d
No (N)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
No (N)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
No (N)
Yes (Y)
No (N)
Yes (Y)
No (N)
Yes (Y)
Yes (Y)
Not applicable (NA)
Partial (P)
Not applicable (NA)
Yes (Y)
Not applicable (NA)
Yes (Y)
Yes (Y)
Not applicable (NA)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Partial (P)
Not applicable (NA)
Yes (Y)
Not applicable (NA)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
2
A1
dvi
Partial (P)
Partial (P)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Partial (P)
Yes (Y)
Yes (Y)
Yes (Y)
No (N)
Yes (Y)
No (N)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
No (N)
Partial (P)
Partial (P)
Partial (P)
Yes (Y)
No (N)
Yes (Y)
Yes (Y)
Partial (P)
Partial (P)
Yes (Y)
No (N)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
No (N)
Yes (Y)
1
F1
d
Partial (P)
Partial (P)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Partial (P)
Partial (P)
Partial (P)
No (N)
Yes (Y)
No (N)
Yes (Y)
No (N)
Yes (Y)
Partial (P)
Not applicable (NA)
Partial (P)
Not applicable (NA)
Yes (Y)
Not applicable (NA)
Partial (P)
Partial (P)
Not applicable (NA)
Yes (Y)
Yes (Y)
Yes (Y)
Partial (P)
No (N)
Not applicable (NA)
Partial (P)
Not applicable (NA)
Yes (Y)
Yes (Y)
No (N)
No (N)
1
A2
dvi
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Partial (P)
Yes (Y)
Partial (P)
Yes (Y)
No (N)
Yes (Y)
No (N)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
No (N)
Yes (Y)
No (N)
Partial (P)
Yes (Y)
No (N)
Partial (P)
Yes (Y)
No (N)
Partial (P)
Yes (Y)
No (N)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
No (N)
Yes (Y)
2
M1
dvi
Partial (P)
Partial (P)
Yes (Y)
Partial (P)
Yes (Y)
No (N)
Partial (P)
Partial (P)
Partial (P)
Yes (Y)
Yes (Y)
Partial (P)
Yes (Y)
Yes (Y)
No (N)
Partial (P)
Partial (P)
Partial (P)
Partial (P)
No (N)
No (N)
No (N)
Partial (P)
Partial (P)
No (N)
No (N)
Partial (P)
No (N)
No (N)
Yes (Y)
No (N)
Partial (P)
Partial (P)
Yes (Y)
Partial (P)
Yes (Y)
No (N)
1
M2
d
Partial (P)
Partial (P)
Yes (Y)
Partial (P)
Yes (Y)
No (N)
Yes (Y)
Yes (Y)
Partial (P)
Yes (Y)
Yes (Y)
Partial (P)
Yes (Y)
Yes (Y)
No (N)
Partial (P)
Partial (P)
Not applicable (NA)
Partial (P)
Not applicable (NA)
No (N)
Not applicable (NA)
Partial (P)
Partial (P)
Not applicable (NA)
Yes (Y)
Partial (P)
No (N)
No (N)
Yes (Y)
Not applicable (NA)
Partial (P)
Not applicable (NA)
Yes (Y)
Partial (P)
Yes (Y)
No (N)
2
J1
dvi
Partial (P)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Partial (P)
Yes (Y)
Partial (P)
Yes (Y)
No (N)
Partial (P)
No (N)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
No (N)
Yes (Y)
No (N)
Partial (P)
Yes (Y)
No (N)
Yes (Y)
Yes (Y)
Partial (P)
Partial (P)
Yes (Y)
No (N)
Yes (Y)
Partial (P)
Yes (Y)
Yes (Y)
No (N)
Yes (Y)
1
S1
d
Partial (P)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Partial (P)
Yes (Y)
No (N)
Yes (Y)
No (N)
Yes (Y)
No (N)
Yes (Y)
Yes (Y)
Not applicable (NA)
Yes (Y)
Not applicable (NA)
No (N)
Not applicable (NA)
Yes (Y)
Yes (Y)
Not applicable (NA)
Yes (Y)
Yes (Y)
Yes (Y)
Partial (P)
Yes (Y)
Not applicable (NA)
Yes (Y)
Not applicable (NA)
Yes (Y)
Yes (Y)
No (N)
Yes (Y)
1
A3
dvi
Partial (P)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Partial (P)
Yes (Y)
Partial (P)
Yes (Y)
Yes (Y)
Partial (P)
Yes (Y)
Yes (Y)
Partial (P)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
No (N)
Partial (P)
Partial (P)
Partial (P)
Partial (P)
No (N)
Yes (Y)
Yes (Y)
Yes (Y)
Partial (P)
Yes (Y)
No (N)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
3
Once your TRIPOD data matches the format of predictr::example_tripod
,
this can be put into the tripod_format()
function to prepare this for
plotting.
tripod_prep <- predictr::example_tripod %>%
tripod_format(id = "score", type = "type")
knitr::kable(head(tripod_prep, 10))
id
type
item
value
B1
Derivation + Validation (Internal)
1
Partial
B1
Derivation + Validation (Internal)
2
Yes
B1
Derivation + Validation (Internal)
3a
Yes
B1
Derivation + Validation (Internal)
3b
Yes
B1
Derivation + Validation (Internal)
4a
Yes
B1
Derivation + Validation (Internal)
4b
Yes
B1
Derivation + Validation (Internal)
5a
Yes
B1
Derivation + Validation (Internal)
5b
Yes
B1
Derivation + Validation (Internal)
5c
Yes
B1
Derivation + Validation (Internal)
6a
Yes
Now we have the data in the required format, we can then plot it using
tripod_plot()
. This produces a high quality ggplot of the TRIPOD
results.
predictr::example_tripod %>%
predictr::tripod_format(id = "score", type = "type") %>%
predictr::tripod_plot()
Let’s use an inbuilt example from predictr
of papers which have been
assessed against the PROBAST tool.
Like the example above for TRIPOD, there are 4 key categories of columns here:
ID column (“score”): This is needed to identify which paper is referred to in each row and must be unique for each row.
Type column (“type”): This is needed to identify what type of models are done in the paper (is it derivation and/or validation of a model). This is essential as this changes what items are needed to be assessed in the PROBAST tool.
This must only contain 3 values: “d” = derivation only, “dv” = derivation and validation" or “v” = validation only.
“rob_” columns: These refer to each respective risk of bias (ROB) assessment using PROBAST for each domain (1 = “Participants”, 2 = “Predictors”, 3 = “Outcome”, and 4 = “Analysis”) and the respective screening questions.
The names of these columns must be exactly the same as this example or else the subsequent function will not work.
This must only contain 5 values as specified in the PROBAST tool (“yes,” “probably yes,” “probably no,” “no,” or “no information”).
“app_” columns: These refer to each respective applicability assessment using PROBAST for each domain (1 = “Participants”, 2 = “Predictors”, and 3 = “Outcome”).
The names of these columns must be exactly the same as this example or else the subsequent function will not work.
This must only contain 3 values as specified in the PROBAST tool (low, unclear, high). Note this assessment is specifically “concerns regarding applicability”, rather than whether the model is applicable.
The 4th domain (“Analysis”) is not assessed in PROBAST regarding “concerns regarding applicability”.
predictr::example_probast %>% head(10) %>% knitr::kable()
score
type
rob\_1\_1
rob\_1\_2
rob\_2\_1
rob\_2\_2
rob\_2\_3
rob\_3\_1
rob\_3\_2
rob\_3\_3
rob\_3\_4
rob\_3\_5
rob\_3\_6
rob\_4\_1
rob\_4\_2
rob\_4\_3
rob\_4\_4
rob\_4\_5
rob\_4\_6
rob\_4\_7
rob\_4\_8
rob\_4\_9
app\_1
app\_2
app\_3
n
B1
d
Probably yes (PY)
Probably yes (PY)
Probably yes (PY)
Probably yes (PY)
Yes (Y)
Yes (Y)
No (N)
Yes (Y)
Probably yes (PY)
Probably no (PN)
Probably yes (PY)
Yes (Y)
Yes (Y)
No information (NI)
No information (NI)
No (N)
No information (NI)
Yes (Y)
No (N)
Yes (Y)
Low
Low
Low
1
B2
dvi
Probably yes (PY)
Probably yes (PY)
Probably yes (PY)
Probably yes (PY)
Yes (Y)
Yes (Y)
No (N)
Yes (Y)
Probably yes (PY)
Probably no (PN)
Probably yes (PY)
Yes (Y)
Yes (Y)
No information (NI)
No information (NI)
No (N)
No information (NI)
Yes (Y)
No (N)
Yes (Y)
Low
Low
Low
2
A1
dve
Probably yes (PY)
Probably yes (PY)
Yes (Y)
Probably yes (PY)
Yes (Y)
Yes (Y)
No (N)
Yes (Y)
Probably yes (PY)
No information (NI)
Yes (Y)
Yes (Y)
Yes (Y)
No (N)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
High
Low
High
1
F1
dvie
Probably yes (PY)
No (N)
Probably yes (PY)
Probably no (PN)
Yes (Y)
Yes (Y)
No (N)
Yes (Y)
Probably yes (PY)
Probably no (PN)
Yes (Y)
Yes (Y)
Yes (Y)
No information (NI)
No information (NI)
No (N)
No information (NI)
Yes (Y)
Yes (Y)
Yes (Y)
Low
Low
Low
1
A2
ve
Probably yes (PY)
Probably yes (PY)
Yes (Y)
Probably yes (PY)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Probably yes (PY)
No information (NI)
Yes (Y)
Yes (Y)
Yes (Y)
No (N)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Low
Low
Low
2
M1
d
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Probably yes (PY)
Yes (Y)
Yes (Y)
Probably yes (PY)
Yes (Y)
No information (NI)
No information (NI)
Yes (Y)
No information (NI)
Yes (Y)
No information (NI)
Yes (Y)
Low
Low
High
1
M2
dvi
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
Yes (Y)
No information (NI)
No information (NI)
No (N)
No information (NI)
Yes (Y)
No (N)
No information (NI)
High
Low
Low
2
J1
dve
Probably yes (PY)
Yes (Y)
Yes (Y)
Probably yes (PY)
Yes (Y)
Yes (Y)
No (N)
Yes (Y)
Yes (Y)
Probably no (PN)
Yes (Y)
Yes (Y)
Yes (Y)
No (N)
No information (NI)
No (N)
No (N)
Yes (Y)
No (N)
Yes (Y)
High
Low
High
1
S1
dvie
Yes (Y)
No (N)
Probably yes (PY)
Probably yes (PY)
Yes (Y)
Probably yes (PY)
Yes (Y)
Yes (Y)
Probably yes (PY)
Probably no (PN)
Yes (Y)
Yes (Y)
Yes (Y)
No (N)
No information (NI)
No (N)
No information (NI)
Yes (Y)
No (N)
Yes (Y)
Low
Low
Low
1
A3
ve
Yes (Y)
Yes (Y)
Probably yes (PY)
Probably yes (PY)
Yes (Y)
Probably yes (PY)
Yes (Y)
Yes (Y)
Probably yes (PY)
Probably no (PN)
Yes (Y)
Yes (Y)
Yes (Y)
No (N)
Yes (Y)
Probably no (PN)
Yes (Y)
Yes (Y)
Probably no (PN)
Yes (Y)
Low
Low
Low
3
Once your TRIPOD data matches the format of predictr::example_tripod
,
this can be put into the tripod_format()
function to prepare this for
plotting.
probast_prep <- predictr::example_probast %>%
probast_format(id = "score", type = "type")
knitr::kable(head(probast_prep, 10))
id
type
domain
subdomain
assess
item
value
B1
Derivation
Risk of Bias
Participants
Low
1
PY
B1
Derivation
Risk of Bias
Participants
Low
2
PY
B1
Derivation
Risk of Bias
Predictors
Low
1
PY
B1
Derivation
Risk of Bias
Predictors
Low
2
PY
B1
Derivation
Risk of Bias
Predictors
Low
3
Y
B1
Derivation
Risk of Bias
Outcome
High
1
Y
B1
Derivation
Risk of Bias
Outcome
High
2
N
B1
Derivation
Risk of Bias
Outcome
High
3
Y
B1
Derivation
Risk of Bias
Outcome
High
4
PY
B1
Derivation
Risk of Bias
Outcome
High
5
PN
Now we have the data in the required format, we can then plot it using
probast_plot()
. This produces a high quality ggplot of the PROBAST
results.
predictr::example_probast %>%
probast_format(id = "score", type = "type") %>%
probast_plot()
For PROBAST, the risk of bias (ROB) assessment is a composite assessment
across all screening questions in the domain. This is calculated
automatically by the probast_plot()
function. However, if you want
that ROB screening question data for a supplementary table, then you can
use the probast_table()
function.
predictr::example_probast %>%
probast_format(id = "score", type = "type") %>%
probast_table() %>% head(10) %>% knitr::kable()
id
type
Participants 1
Participants 2
Predictors 1
Predictors 2
Predictors 3
Outcome 1
Outcome 2
Outcome 3
Outcome 4
Outcome 5
Outcome 6
Analysis 1
Analysis 2
Analysis 3
Analysis 4
Analysis 5
Analysis 6
Analysis 7
Analysis 8
Analysis 9
Overall NA
B1
Derivation
PY
PY
PY
PY
Y
Y
N
Y
PY
PN
PY
Y
Y
NI
NI
N
NI
Y
N
Y
NA
B2
Derivation + Validation (Internal)
PY
PY
PY
PY
Y
Y
N
Y
PY
PN
PY
Y
Y
NI
NI
N
NI
Y
N
Y
NA
A1
Derivation + Validation (External)
PY
PY
Y
PY
Y
Y
N
Y
PY
NI
Y
Y
Y
N
Y
Y
Y
Y
Y
Y
NA
F1
Derivation + Validation (Both)
PY
N
PY
PN
Y
Y
N
Y
PY
PN
Y
Y
Y
NI
NI
N
NI
Y
Y
Y
NA
A2
Validation (External)
PY
PY
Y
PY
Y
Y
Y
Y
PY
NI
Y
Y
Y
N
Y
Y
Y
Y
Y
Y
NA
M1
Derivation
Y
Y
Y
Y
Y
Y
Y
Y
PY
Y
Y
PY
Y
NI
NI
Y
NI
Y
NI
Y
NA
M2
Derivation + Validation (Internal)
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
Y
NI
NI
N
NI
Y
N
NI
NA
J1
Derivation + Validation (External)
PY
Y
Y
PY
Y
Y
N
Y
Y
PN
Y
Y
Y
N
NI
N
N
Y
N
Y
NA
S1
Derivation + Validation (Both)
Y
N
PY
PY
Y
PY
Y
Y
PY
PN
Y
Y
Y
N
NI
N
NI
Y
N
Y
NA
A3
Validation (External)
Y
Y
PY
PY
Y
PY
Y
Y
PY
PN
Y
Y
Y
N
Y
PN
Y
Y
PN
Y
NA
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.