add.evidence: Add evidence classes to "umbrella" objects
In metaumbrella: Umbrella Review Package for R

add.evidence

R Documentation

Add evidence classes to “umbrella” objects

Description

Add evidence classes to the factors included in an umbrella review.

Usage

add.evidence(
  x,
  criteria = "Ioannidis",
  eq_range_or = c(0.8, 1.25),
  eq_range_g = c(-0.1, 0.1),
  class_I = c(n_studies = NA, total_n = NA, n_cases = NA, p_value = NA, I2 = NA,
    imprecision = NA, rob = NA, amstar = NA, egger_p = NA, esb_p = NA, JK_p = NA, pi =
    NA, largest_CI = NA),
  class_II = c(n_studies = NA, total_n = NA, n_cases = NA, p_value = NA, I2 = NA,
    imprecision = NA, rob = NA, amstar = NA, egger_p = NA, esb_p = NA, JK_p = NA, pi =
    NA, largest_CI = NA),
  class_III = c(n_studies = NA, total_n = NA, n_cases = NA, p_value = NA, I2 = NA,
    imprecision = NA, rob = NA, amstar = NA, egger_p = NA, esb_p = NA, JK_p = NA, pi =
    NA, largest_CI = NA),
  class_IV = c(n_studies = NA, total_n = NA, n_cases = NA, p_value = NA, I2 = NA,
    imprecision = NA, rob = NA, amstar = NA, egger_p = NA, esb_p = NA, JK_p = NA, pi =
    NA, largest_CI = NA),
  verbose = TRUE
)

Arguments

`x`	an object of class “umbrella”.
`criteria`	the evidence criteria. It must be "GRADE", "Ioannidis" or "Personalized".
`eq_range_or`	a vector of the bounds of equivalence ranges for OR/RR/HR/IRR (only required for GRADE) criteria.
`eq_range_g`	a vector of the bounds of equivalence ranges for SMD (only required for GRADE) criteria.
`class_I`	a vector or list of threshold values required for reaching Class I in the Personalized criteria (see details below).
`class_II`	a vector or list of threshold values required for reaching Class II in the Personalized criteria (see details below).
`class_III`	a vector or list of threshold values required for reaching Class III in the Personalized criteria (see details below).
`class_IV`	a vector or list of threshold values required for reaching Class IV in the Personalized criteria (see details below).
`verbose`	logical variable indicating whether text outputs and messages should be generated. We recommend turning this option to FALSE only after having carefully read all the generated messages.

Details

The add.evidence() function performs a stratification of evidence according to three criteria.

"Ioannidis" classification

This classification allows to stratify evidence according to the criteria described in Fusar-Poli & Radua (2018). This classification proposes to stratify evidence in five ordinal classes: "Class I", "Class II", "Class III", "Class IV", "Class ns". The criteria for each class are the following:

Class I: number of cases > 1000, p-value of the meta-analysis < 10^-6, I^2 < 0.5, 95% prediction interval excluding the null, p-value of the Egger test > .05 and p-value of the excess of statistical significance test > .05.
Class II: number of cases > 1000, p-value of the meta-analysis < 10^-6, largest study with a statistically significant effect and class I criteria not met.
Class III: number of cases > 1000, p-value of the meta-analysis < 10^-3 and class I-II criteria not met.
Class IV: p-value of the meta-analysis < 0.05 and class I-III criteria not met.
Class ns: p-value of the meta-analysis >= 0.05. To apply this classification with R and Z effect size measures, you should indicate both the 'n_sample' AND the 'n_cases'.

"GRADE" classification

This classification allows to stratify evidence according to four ordinal classes: "High", "Moderate", "Low", "Very low". Importantly, this classification should not be taken as an equivalent to the subjective approach underlying the standard GRADE classification. However, in line with the standard GRADE approach, this classification uses a downgrading procedure in which all factors start with a "High" evidence class that could then be downgraded according to the 5 following criteria. Importantly, when the number of studies is low (k < 5), then the meta-analysis starts with a 'Moderate' rating to account for the difficulty of identifying heterogeneity and publication bias with such a limited number of studies.

All calculations are made automatically, but users should input in their dataset, i) the overall risk of bias of each study ('rob' column), ii) the risk of selective reporting ('rob1_report', or 'rob2_report' columns) and iii) the risk of indirectness ('indirectness' column). If this information is left empty, each criterion will be assumed to be at high risk.

Risk of Bias (Limitations):
- No downgrade: >=75% of participants included in low-risk studies
- One downgrade: 50%-75% of participants included in low-risk studies
- Two downgrades: <=50% of participants included in low-risk studies
The pooled percentage of participants is calculated as a weighted mean, with weights attributed to each study being equal to the weight each study receives in the meta-analysis.
Heterogeneity:
- Two downgrades: Substantial discrepancy between the 95% CI and 95% PI (e.g., bounds of the 95% CI and 95% PI not of the same sign and in different equivalence ranges).
- One downgrade: Small/moderate discrepancy between the 95% CI and 95% PI (e.g., bounds of the 95% CI and 95% PI of the same sign, but in different equivalence ranges).
When 95% PI is not reliably estimable, the assessment relies on the I² statistic and the percentage of studies with contradicting results:
- Two downgrades: I² >= 50% and >=10% of studies with statistically significant results in the opposite direction compared to the pooled effect size
- One downgrade: I² >= 30% and >=10% of studies with statistically significant results in the opposite direction compared to the pooled effect size
Indirectness: The number of downgrades and the criteria are left to the user's discretion, as these factors vary significantly depending on the scope of the review. Examples of criteria may include heterogeneity in participants' age or undefined control groups:
- No downgrade: No concerns regarding indirectness
- One downgrade: Serious concerns (e.g., "serious" indirectness)
- Two downgrades: Very serious concerns (e.g., "very serious" indirectness)
Imprecision:
- Two downgrades: The 95% CI of the pooled effect size includes both null (SMD = 0; RR/OR = 1) and large (SMD >= 0.80; OR/RR >= 5) effects AND the meta-analysis does not have the sample size required to detect small effects (eSMD = 0.20) with 80% statistical power (n < 394 per arm)
- One downgrade: The 95% CI of the pooled effect size includes both null and large effects
- Two downgrades: The meta-analysis does not have the sample size required to detect moderate effects (eSMD = 0.50) with 80% statistical power (n < 64 per arm)
- One downgrade: The meta-analysis does not have the sample size required to detect small effects (eSMD = 0.20) with 80% statistical power (n < 394 per arm)
Publication Bias:
- One downgrade: p-value of Egger's test < 0.10, OR excess significance bias p-value < 0.10, OR more than 50% of participants included in trials with high reporting bias

This classification is not available for R and Z effect size measures.

The GRADE classification implementation in this package was developed through the collaborative efforts of: Dr Corentin J. Gosling, Dr Miguel Garcia-Argibay, Prof Richard Delorme, Prof Marco Solmi, Prof Andrea Cipriani, Prof Christoph U. Correll, Dr Cinzia Del Giovane, Prof Paolo Fusar-Poli, Prof Henrik Larsson, Edoardo Ostinelli, Prof Jae Il Shin, Prof DongKeon Yon, Prof Joaquim Radua, Prof John P. Ioannidis and Prof Samuele Cortese

"Personalized" classification

Because the "Ioannidis" and "GRADE" classifications do not necessarily provide a rating system that perfectly matches the requirements of your umbrella review, the add.evidence() function offers the possibility to use a "Personalized" criteria to stratify the evidence according to 13 criteria. This Personalized criteria proposes to stratify the evidence in 5 ordinal classes: "Class I", "Class II", "Class III", "Class IV" and "Class V". "Class I" is the highest class that could be achieved and "Class V" is the lowest.
The overall class achieved by a factor is equal to the lowest class achieved by all the criteria used to stratify evidence. For example, if users choose to stratify the evidence according to 3 criteria (the p-value of the meta-analysis, the inconsistency, the publication bias), and that the classes achieved by these 3 criteria are respectively "Class I", "Class III" and "Class IV", the overall class reached by the factor will be "Class IV".
To determine the class that should be assigned to a factor, users have to indicate - for each class - a vector/list of threshold values for all the criteria that are used to stratify the evidence. A description of the criteria and their corresponding inputs is provided below:

n_studies: a number of studies included in the meta-analysis. If the number of studies included in the meta-analysis is strictly superior to the threshold value indicated in studies, the class for which this value is indicated can be reached.
total_n: a total number of participants included in the meta-analysis. If the total number of participants included in the meta-analysis is strictly superior to the threshold value indicated in total_n, the class for which this value is indicated can be reached.
n_cases: a number of cases included in the meta-analysis. If the number of cases included in the meta-analysis is strictly superior to the threshold value indicated in cases, the class for which this value is indicated can be reached.
p_value: a p-value of the pooled effect size under the random-effects model. If the p-value of the pooled effect size is strictly inferior to the threshold value indicated in p.value, the class for which this value is indicated can be reached.
I2: an i-squared (I^2) value. If the I^2 value of the meta-analysis is strictly inferior to the threshold value indicated in I2, the class for which this value is indicated can be reached.
imprecision: a SMD value that will be used to calculate the statistical power of the meta-analysis. If the number of participants included in the meta-analyses allows to obtain a statistical power strictly superior to 80% for the SMD value indicated in imprecision, the class for which this value is indicated can be reached.
rob: a percentage of participants included in studies at low risk of bias. Note that the approach to determining whether a study is at low risk of bias is left to the user. If the percentage of participants included in studies at low risk of bias is strictly superior to the threshold value indicated in rob, the class for which this value is indicated can be reached.
amstar: an AMSTAR rating on the methodological quality of the meta-analysis. If the AMSTAR value of the meta-analysis is strictly superior to the threshold value indicated in amstar, the class for which this value is indicated can be reached.
egger_p: a p-value of an Egger's test for publication bias. If the p-value of the Egger's test is strictly superior to the threshold value indicated in egger_p, the class for which this value is indicated can be reached.
esb_p: a p-value of a test for excess of statistical significance bias (ESB). If the p-value of the test is strictly superior to the threshold value indicated in esb_p, the class for which this value is indicated can be reached.
JK_p: the largest p-value obtained in the jackknife meta-analysis (JK). If the largest p-value obtained in the jackknife meta-analysis is strictly inferior to the threshold value indicated in JK_p, the class for which this value is indicated can be reached.
pi: a "notnull" value indicates that users request the 95% prediction interval of the meta-analysis to exclude the null value to achieve the class for which it is indicated.
largest_CI: a "notnull" value indicates that users request the 95% confidence interval of the largest study included in the meta-analysis to exclude the null value to achieve the class for which it is indicated.

Value

Return an object of class “umbrella” with the evidence classes added.

References

Fusar-Poli, P., & Radua, J. (2018). Ten simple rules for conducting umbrella reviews. Evidence-Based Mental Health, 21, 95-100.

Examples

### perform calculations required for an umbrella review
df <- subset(df.SMD, factor == "Surgical")
umb.full <- umbrella(df)

### stratify evidence according to the Ioannidis classification
evid_ioannidis <- add.evidence(umb.full, criteria = "Ioannidis")
summary(evid_ioannidis)

### stratify evidence according to the Personalized classification with
### the number of studies and cases, the inconsistency as criteria.
### - a class I can be reached if the number of studies is > 10, the number of cases is > 500 and
###   the I2 is < 25%.
### - a class II can be reached if the number of studies is > 5, the number of cases is > 400 and
###   the I2 is < 50%.
### - a class III can be reached if the number of cases is > 300 and the I2 is < 75%.
### - a class IV can be reached if the number of cases is > 100.
### - else, if the number of cases is <= 100, a class V is assigned.
evid_perso1 <- add.evidence(umb.full, criteria = "Personalized",
   class_I = c(n_studies = 10, n_cases = 500, I2 = 25),
   class_II = c(n_studies = 5, n_cases = 400, I2 = 50),
   class_III = c(n_cases = 300, I2 = 75),
   class_IV = c(n_cases = 100))
summary(evid_perso1)

metaumbrella documentation built on April 12, 2025, 1:32 a.m.