In mcguinlu/robvis: Visualize the Results of Risk-of-Bias (ROB) Assessments

``` {r, echo = FALSE, warning=FALSE, message=FALSE } library(robvis) library(cowplot)

knitr::opts_chunk$set(eval=FALSE)

# Front Matter
__Title: __  
Risk-Of-Bias VISualization (`robvis`): an R package and Shiny web app for visualizing risk-of-bias assessments

 

__Authors and Affiliations__

Luke A McGuinness^1,2^ (ORCID: 0000-0001-8730-9761), 
Julian PT Higgins^1,2^ (ORCID: 0000-0002-8323-2514)

(1) MRC Integrative Epidemiology Unit at the University of Bristol, Bristol, UK
(2) Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, UK

 

__Corresponding author:__  

Luke McGuinness; Bristol Medical School, University of Bristol,
Canynge Hall, 39 Whatley Road, Bristol, BS8 2PS, United Kingdom; [e-mail]

 

__Article type__

_Computational Tools and Methods_

 

__Funding__


LAM is supported by an NIHR Doctoral Research Fellowship (DRF-2018-11-ST2-048). Attendance at the Evidence Synthesis Hackathon in April 2019 was supported by a grant from the Evidence Synthesis Hackathon itself and the University of Bristol Alumni Foundation. JPTH is a National Institute for Health Research (NIHR) Senior Investigator (NF-SI-0617-10145). JPTH is supported by NIHR Bristol Biomedical Research Centre at University Hospitals Bristol NHS Foundation Trust and the University of Bristol. JPTH is supported by the NIHR Applied Research Collaboration West (ARC West) at University Hospitals Bristol NHS Foundation Trust. JPTH is a member of the MRC Integrative Epidemiology Unit at the University of Bristol. The views expressed in this article are those of the authors and do not necessarily represent those of the NHS, the NIHR, MRC, or the Department of Health and Social Care.

 

__Keywords__

Risk of bias; Evidence synthesis; Data visualization; R


#####


# Abstract
Despite a major increase in the range and number of software offerings now available to help researchers produce evidence syntheses, there is currently no generic tool for producing figures to display and explore the risk-of-bias  assessments that routinely take place as part of systematic review. However, tools such as the R programming environment and `Shiny` (an R package for building interactive web apps) have made it straightforward to produce new tools to help in producing evidence syntheses. We present a new tool, `robvis` (Risk-Of-Bias VISualization), available as an R package and web app, which facilitates rapid production of publication-quality risk-of-bias assessment figures. We present a timeline of the tool’s development and its key functionality.

#####

# Introduction

Synthesis of evidence from the totality of relevant research is becoming more important than ever in informing policy across an increasingly wide range of fields.[@donnelly2018] Risk of bias assessment - evaluation of the internal validity of studies included in a systematic review - often forms a key part of the evidence synthesis process, particularly in the health sciences.[@cochranechpt7] A well-developed family of tools is widely used, which have in common the characteristic that they evaluate specific domains of bias rather being constructed as a checklist or a quantitative score.[@cochranechpt7] These tools include the RoB (Risk of Bias) 2 tool for randomized trials,[@sterne2019rob] the ROBINS-I (Risk Of Bias In Non-randomized Studies - of Interventions) tool,[@sterne2016robins] the QUADAS 2 (Quality and Applicability of Diagnostic Accuracy Studies) tool [@whiting2011quadas] and the ROBIS (Risk Of Bias in Systematic Reviews) tool.[@whiting2016robis] Within each bias domain a judgement is reached about the strength of the study in that regard: for example, the first domain in the Cochrane RoB 2 tool deals with bias arising from the randomization process.[@sterne2019rob] Accessible graphics summarizing the results of these domain-based risk-of-bias assessments are included in reports of systematic reviews. A convenient plot in many reviews is a “traffic light” plot, which tabulates the judgement for each study in each domain. For larger numbers of studies, when such a table becomes unmanageable, a popular alternative is a (possibly weighted) bar plot, which shows the proportion of information with each judgement for each domain.[@higgins2008assessing]

Researchers can face a number of barriers in creating these plots. While some evidence synthesis platforms, such as Cochrane’s Review Manager,[@cochrane2014review] can produce these visualizations, not all researchers use these systems to conduct their systematic reviews, and copying the risk-of-bias data into these systems solely to produce the plots is inefficient and error prone. On the other hand, creating the figures "by hand", through software such as MS PowerPoint or Adobe Illustrator, may lead to unintentional errors and require the plots to be redrawn when a review is updated. Additionally, while the field of evidence synthesis software has grown rapidly in recent years,[@marshall2015systematic] this growth has not been equally distributed across the different aspects of the systematic review process. For example, a recent review found several software offerings aimed specifically at the abstract screening stage of the review process,[@harrison2020software] but no similar time- and error-reducing tool has been proposed for visualizing the results of risk-of-bias assessments.

It is now straightforward to produce such a tool, thanks to the availability of powerful computing offerings including R, RStudio and `Shiny` (an R package for building interactive web apps) [@rref; @rstudioref; @shinyref] Here, we present `robvis` (Risk Of Bias VISualization),[@mcguinness2019a] an R package and `Shiny` web-app that allows users to create publication-ready risk-of-bias plots quickly and easily. While primarily designed for use with the major risk-of-bias assessment tools used in health research (ROB2, ROBINS-I & QUADAS-2), the tool allows users to visualize the results from any domain-based risk-of-bias assessment or quality appraisal tool.

The tool is open-source and available to use free of charge. Users can download a stable version of the R package from CRAN ([https://cran.r-project.org/package=robvis](https://cran.r-project.org/package=robvis)); or access and contribute to the open-source code that powers the package via GitHub ([https://github.com/mcguinlu/robvis](https://github.com/mcguinlu/robvis)). Extended guidance for the tool, including a step-by-step walk-through for those new to the R programming environment, is also available via the “Doing Meta-Analysis in R” online guide.[@mathias_harrer_2019_2551803] Below, we discuss the tool's development and key functionality.



# Development

Development of `robvis` began in April 2019 at the Evidence Synthesis Hackathon (ESH), an event which brings together interested researchers, practitioners and coders to discuss and develop new open-source evidence synthesis technologies. Test versions of both the R package and the web app were made available in early June 2019, with attendees of the ESH and members of the Bristol Appraisal and Review of Research (BARR) group at the University of Bristol being invited to test the tool and provide feedback. This feedback, along with other feature suggestions from the wider evidence synthesis community captured via GitHub issues, was incorporated and the first release version of the package was uploaded to CRAN in November 2019. The tool has been well received and is beginning to be cited in the evidence synthesis literature.[@gibb2019consistent; @habadi2019prevalence; @veloso2020effectiveness;@simillis2020; @tanneru2020]


# Functionality
## R package
### Tool templates and example data sets

At the time of writing, the tool includes templates for three major tools: the Cochrane RoB 2 tool for assessing randomized trials,[@sterne2019rob] the ROBINS-I tool for assessing non-randomized studies of interventions,[@sterne2016robins] and the QUADAS-2 tool for assessing diagnostic accuracy studies.[@whiting2011quadas] These templates automatically apply the correct risk-of-bias domain names to the figures, and label the judgement levels appropriately (e.g. “Low”, “Some Concerns”, “High” in the case of the RoB 2 tool). In addition, `robvis` contains a general template that can be used to visualize the result of any domain-based assessment tool. The generic template has greater flexibility than the tool-specific templates, allowing a user-specified number of domains and custom domain titles to be used. It is suitable for use with the original version of the Cochrane risk-of-bias tool for randomized trials,[@higgins2011] in which flexibility in the specification of domains was permitted. Since users of more recent tools such as RoB 2 and ROBINS-I are not permitted to modify the domains, we strongly encourage use of the in-built templates for these.

In order to help users familiarize themselves with the package and its functionality, `robvis` contains built-in example data-sets for each template. We illustrate the example data for the RoB 2 tool for assessing risk of bias in randomized trials in Table  \@ref(tab:robdata).

### Data import

`robvis` expects the risk-of-bias data file to be arranged in a specific way (see Table 1 for an example). The first column should contain the unique study/result identifier. The second-to-last column should contain the overall risk-of-bias judgments. The final column contains some measure of the result’s precision (e.g. the weight assigned to that result in a meta-analysis, or the sample size of the analysis that produced the result). This weight column is used to create the summary bar plot, as current guidance recommends dividing the bars to show the proportion of information at each level of risk of bias, determined by the cumulative weight at that level, rather than simply showing the number of studies/results in each category.[@cochranechpt7] If a measure of precision is not available, or to reproduce ‘equally’ weighted bar charts as have traditionally been presented in Cochrane Reviews to date, these weights may all be specified to be 1.


### Functions

`robvis` contains two main functions. The first, `rob_traffic_light()`, creates a _traffic light plot_. This displays every risk-of-bias judgement in a matrix, with domains along the horizontal and results/studies down the vertical, similar to the data set. The second function, `rob_summary()`, creates a _weighted bar plot_. This shows the proportion of information with each risk-of-bias judgement separately for each domain in the assessment tool specified.

A worked example using these functions is outlined below, illustrating the simple steps involved in creating risk-of-bias plots using `robvis`. A detailed description of the additional options that can be used with each function is presented in Table 2. All examples produced in this article are created using the stable version available from CRAN.

To install and load the package in R, enter the following into the console:

 

```r

install.packages("robvis")

library(robvis)

 

Using the example data set (data_rob2) that is built into the package and is presented in Table \@ref(tab:robdata) for reference, the traffic light plot shown in Figure \@ref(fig:trafficplot) is created using:

 

rob_traffic_light(data = data_rob2,
                  tool = "ROB2",
                  colour = "colourblind", 
                  psize = 15)

 

Similarly, using the same data set, the summary barplot shown in Figure \@ref(fig:summaryplot) is created using:

 

rob_summary(data = data_rob2,
            tool = "ROB2", 
            overall = TRUE)

 

Further customization

The ggplot2 package in R, based on the “The Grammar of Graphics”, allows users to create detailed graphics and was used to create the templates found in robvis.[@ggplot2ref; @wilkinson2012grammar] As a result, both robvis functions return a ggplot object, meaning they can be easily customized further using the ggplot2 framework. For example, to add a title to the summary bar plot:

 

    library(ggplot2)


    plot <- rob_summary(data_rob2, tool = "ROB2")

    plot +
    ggtitle("Summary Bar Plot")

 

For a full discussion of the range of post-production modifications that can be made to the plots via the ggplot2 package, we refer the reader to the extensive guidance available.[@wilkinson2012grammar]

Shiny web app

Although robvis was originally designed for use in the command-line–based R programming environment, we developed a web app to make the tool accessible to those without knowledge of R. This is available via www.riskofbias.info.[@robviswebapp] The app was built using Shiny,[@shinyref] an R package which makes it easy to produce interactive web-apps, and provides a graphical user interface (GUI) for the robvis package, allowing users to interact with the functions presented above without the need to download R or type any commands. Users can upload their data as either an Excel spreadsheet (recommended) or a comma-separate values (CSV) file, or can manually enter it directly into the app. Uploaded data are passed through a number of quality control checks that ensure the app will work correctly, and users are prevented from producing the plots until any issues identified have been addressed. Finally, users can customize their plots by defining a colour scheme and other parameters, all through the online app.

Discussion

robvis facilitates the rapid production of two common risk-of-bias assessment figures at publication quality. By implementing robvis both as an R package and a Shiny web app, its functionality is available to evidence synthesists with varying levels of ability in R. robvis serves as an example of the advantages of “packaging” the R scripts that evidence synthesists often create for personal use.[@wickham2015r] It is likely that several other evidence synthesists have written scripts to produce similar risk-of-bias plots to those presented here - in fact, we personally know of at least one other research group that has done so. This duplication of time and effort is inefficient, and creating and sharing well-documented R packages represents one way to reduce this inefficiency. Taking this approach one step further, Shiny apps represent a straightforward way to provide a user-friendly GUI for a newly created R package within a very short time-frame, expanding the potential pool of users of the package to anyone with an internet connection.

Creating a package using R has a number of particular advantages. R provides access to a range of powerful tools including the ggplot infrastructure as demonstrated above, and RMarkdown, which enables creation of documents that can be rendered in a range of formats such as PDF, HTML, or Word.[@xie2018r] Furthermore, and focusing specifically on evidence synthesis, building new tools as packages in R allows for easy integration with the range of existing evidence synthesis packages. Recently, the metaverse project,[@variousauthors2020] of which robvis is a part, has begun to curate a collection of R packages that cover different aspects of the systematic review and meta-analysis process which, when taken together, form a coherent end-to-end open-source alternative to commercial offerings such as Covidence or Review Manager. Key offerings in this suite of packages include litsearcher, which facilitates systematic search strategy development, revtools, a package for managing the review process and performing title and abstract screening, metaDigitise, a package for automatic extraction of data from figures in research papers, and metafor, a package for conducting meta-analyses in R.[@grames2019automated; @metaforref; @pick2018; @westgate2019revtools]

While robvis is a stable package, a range of additional functionality could be added. At present, the number of tools with a specific template included in robvis is limited - adding additional templates is a priority. For example, a template for ROBIS, the risk-of-bias assessment tool for systematic reviews,[@whiting2016robis] is in development. Additionally, the robvis tool does not yet allow for the production of paired forest plots, where the risk-of-bias judgement is presented alongside each specific result included in the meta-analysis.[@cochranechpt7] This was initially considered to be beyond the scope of the tool, as it involves the visualization of something other than risk-of-bias assessments. However, following user-driven demand, this functionality is in development and will be available in the near future. Finally, we would like to add similar functionality to that provided by the metafor::reporter() function, which generates a brief paragraph of text describing the results of a meta-analysis. The future robvis::reporter() function would provide a boilerplate description of the assessment tool used and the key domains at risk of bias.

Back Matter

Acknowledgments

The following people provided valuable feedback that contributed to the development of the robvis tool: Emily Kothe, Eliza Grames, Matthew Page, Alexandra Bannach-Brown, Kyle Hamilton, Charles Gray, Vincent Cheng, Wouter van Amsterdamn, Neal Haddaway and Martin Westgate.

Code/Data availability statement

The software and data presented in this paper are freely available on GitHub: https://github.com/mcguinlu/robvis

Highlights

• Risk-of-bias assessment is a key element of the systematic review workflow.
• No other tool exists explicitly for the purpose of visualizing risk-of-bias results.
• Here we present robvis, an open-source R package and Shiny web app for creating publication-ready risk-of-bias assessment figures.
• robvis forms part of the metaverse, a collection of R packages designed to provide an evidence synthesis workflow in R.

Tables/Figures

plotdata <- robvis::data_rob2

plotdata$Weight <- as.numeric(round(plotdata$Weight,2)) 

plotdata$Weight <- gsub("\\.00", "", plotdata$Weight)

knitr::kable(plotdata, caption = "Example dataset for the ROB 2 tool contained within `robvis`. Data can be imported to the tool from with an Excel spreadsheet or a CSV file.")

``` {r robarguments, echo = FALSE, eval = TRUE}

arguments <- data.frame(stringsAsFactors=FALSE, argument = c("data", "tool", "colour", "overall", "weighted", "psize"), rob_traffic_light = c("X", "X", "X", "", "", "X"), rob_summary = c("X", "X", "X", "X", "X", ""), Description = c( "Defines the dataframe containing the summary (domain) level risk-of-bias assessments. See the text and Table 1 for the format expected by robvis", "Defines the risk of bias assessment tool used. The RoB2 (tool=\"ROB2\"), ROBINS-I (tool=\"ROBINS-I\"), and QUADAS-2 (tool=\"QUADAS-2\") assessments tools are currently supported. Other tools can be visualized using the generic template (tool = \"Generic\") [Note 1]", "Defines the colour scheme for the plot. The default is colour = \"cochrane\" which uses the \"Cochrane\" (red, yellow, green) colours, while a preset option for a colour-blind friendly palette is also available (colour = \"colourblind\"). Alternatively, users can specify their own colour scheme e.g. colour = c(\"#f442c8\", \"#bef441\", \"#000000\")", "Defines whether to include an additional bar showing the distribution of overall risk of bias judgments in the summary barplot figure. Default is overall = FALSE.", "Defines whether weights should be used to produce the summary barplot figure. Default is weighted = TRUE, in line with current Cochrane Collaboration guidance.", "Defines the size of the points in the traffic light plot. Default is psize = 20."))

knitr::kable(arguments,escape = T, caption = "Description of the arguments available in the two main functions of the robvis R package. ‘X’ indicates that the option is available for the respective function.", align = c("l","c","c","l"), col.names = c("Argument","rob_traffic_light()","rob_summary()", "Description"))

Note 1: This option (`tool = "Generic"`) reflects the general template name used in the current development version of `robvis`, which will become the standard for all future iterations of the package. However, in the current CRAN version, the generic template is accessed using `tool = "ROB1"`.

######

``` {r trafficplot, echo = FALSE, eval = TRUE, fig.cap = "Example risk of bias traffic light plot of ROB2 assessments created using `robvis` and the colourblind palette."}

plotdata <- data_rob2

tfplot <- robvis::rob_traffic_light(data = plotdata, "ROB2", psize = 15, colour = "colourblind")

ggplot2::ggsave(
          filename = "figs/Fig1-rob-traffic-light-plot.tiff",
          plot = tfplot,
          width = 7,
          height = 8,
          dpi = 600
          )

ggplot2::ggsave(
          filename = "figs/Fig1-rob-traffic-light-plot.png",
          plot = tfplot,
          width = 7,
          height = 8,
          dpi = 1000
          )


knitr::include_graphics("figs/Fig1-rob-traffic-light-plot.png")

`` {r summaryplot, echo = FALSE, eval = TRUE, fig.cap = "Example risk of bias summary plot of ROB2 assessments created usingrobvis` and the standard Cochrane palette."}

sumplot <- robvis::rob_summary(plotdata, "ROB2", overall = TRUE) + ggplot2::theme(plot.background = ggplot2::element_rect(colour = "black"))

ggplot2::ggsave( filename = "figs/Fig2-rob-summary-barplot.tiff", plot = sumplot, width = 8, height = 2.41, dpi = 600 )

ggplot2::ggsave( filename = "figs/Fig2-rob-summary-barplot.png", plot = sumplot, width = 8, height = 2.41, dpi = 1000 )

knitr::include_graphics("figs/Fig2-rob-summary-barplot.png")

```

Bibliography

mcguinlu/robvis documentation built on July 4, 2023, 11:15 p.m.