In bikeactuary/repoRter.nih: R Interface to the 'NIH RePORTER Project' API

opts_knit$set(upload.fun = function(file) {RWordPress::uploadFile(file)$url })
options(width = 95, cli.unicode = FALSE, cli.width = 80)

Introduction

The US National Institute of Health (NIH) received funding of approximately \$42 billion in fiscal year 2022; \$31 billion (72%) of this was awarded by the NIH in the form of research grant funding to hospitals, medical colleges, non-profits, businesses, and other organizations based in the U.S. and abroad.^[https://nexus.od.nih.gov/all/2021/04/21/fy-2020-by-the-numbers-extramural-investments-in-research] The NIH maintains a publicly available database called "RePORTER" to track this substantial flow of grant funding and makes it available to the public via a web-based query interface as well as an API.

"The NIH RePORTER APIs is designed to programmatically expose relevant scientific awards data from both NIH and non-NIH federal agencies for the consumption of project teams or external 3rd party applications to support reporting, data analysis, data integration or to satisfy other business needs as deemed pertinent." r tufte::quote_footer('--NIH RePORTER v2 API Documentation')

This data can have significant value for many audiences, including researchers, investors, industry, watchdogs/public advocates, and R users. But constructing queries and retrieving results programmatically involves some coding overhead which can be a challenge for those not familiar with RESTful APIs and JSON; it takes some effort even for those who are. The repoRter.nih package aims to simplify this task for the typical analyst scripting in R.

Getting Started

Installation

This package (latest stable release) can be installed from CRAN the usual way:

install.packages("repoRter.nih")

The current dev version can be installed from github, on the dev branch:

devtools::install_github('bikeactuary/repoRter.nih@dev')

I welcome R developers more capable than myself to collaborate on improving the source code, documentation, and unit testing in this package.

Basic Workflow

library(repoRter.nih)

The make_req() method is used to generate a valid JSON request object. The req can subsequently be passed to the RePORTER Project API and results retrieved via the get_nih_data() method.

Generating the request:

# all projects funded by the Paycheck Protection Act, Coronavirus Response and
# Relief Act, and American Rescue Plan, in fiscal year 2021
req <- make_req(criteria =
                  list(fiscal_years = 2021,
                       covid_response = c("C4", "C5", "C6")))

Sending the request and retrieving results:

res <- get_nih_data(req)
class(res)

A tibble is returned containing 43 columns. This data is not flat - several columns are nested data.frames and lists (of variable length vectors and data.frames of varying height).

res %>% glimpse(width = getOption("cli.width"))

Criteria-Field Translation

A dataset (nih_fields) is provided with this package to assist in translating between field names used in the payload criteria, column names in the return data, and field names used in the include_fields, exclude_fields, and sort_field arguments.

data("nih_fields")
nih_fields %>% print

Some fields can not be used as filtering criteria - these will show NA in the payload_name column.

Generating Requests

Most of the detail (and function documentation) is around the many parameters available in RePORTER to filter/search project records. Let's get into some of the capabilities.

Default Request

If no arguments are supplied, the default behavior of make_req() is to generate a request for all projects funded in fiscal_years = lubridate::year(Sys.Date()). Limiting requests to a single year is often necessary (depending on additional filtering criteria used) due to a RePORTER restriction that a maximum of 10K records may be returned from any result set. There are currently ~2.6M projects in the database going back to fiscal year 1985, and each fiscal year tends to have 70-100K projects, so the 10K limit can be restrictive to the user wanting a broad search.

req <- make_req()

The method prints a helpful message to the console in addition to returning the JSON. Set message = FALSE if you wish to suppress this message.

Limiting Data Retrieved

You can limit both the width and height of the result set retrieved from the API.

Fields

We probably will not need to fetch every field every time. The include_fields argument is provided to specify a limited set of fields to be returned. Alternatively, fields may be excluded using exclude_fields.

Records (projects)

This package provides the ability to retrieve only a limited number of result pages via the max_pages argument. This can be useful for developing/testing your queries (and for reducing time to render package documentation). Each page has a record count equal to limit - so setting max_pages = 5 with the default limit = 500 (the maximum permitted by RePORTER) in make_req() will result in up to 2,500 total records returned.

Ex. 1 - Limiting results and selecting fields

data("nih_fields")
fields <- nih_fields %>%
  filter(response_name %in% 
           c("appl_id", "subproject_id", "project_title", "fiscal_year",
             "award_amount", "is_active", "project_start_date")) %>%
  pull(include_name)

req <- make_req(include_fields = fields,
                limit = 500,
                message = FALSE) # default
res <- get_nih_data(query = req,
                    max_pages = 1)

res %>% glimpse(width = getOption("cli.width"))

Some Vanilla Criteria

Many criteria are passed as vectors within the criteria list argument. We will cover some of the most useful examples:

Ex. 2 - Organization search

We can refine our query results by providing filtering criteria to make_req(), and by extension to the API. Suppose we want all currently active projects, funded in fiscal years 2017 through 2021, with a specific organization in mind (though we don't know exactly how its name will appear in RePORTER):

req <- make_req(criteria = 
                  list(
                    fiscal_years = 2010:2011,
                    include_active_projects = TRUE,
                    org_names = c("Yale", "New Haven")
                  ),
                include_fields = c("Organization", "FiscalYear", "AwardAmount"),
                message = FALSE)

Here we are asking for any orgs containing the strings "yale" or "new haven" (ignoring case) - there are implied wildcards on either end of the strings we provide. This is the same as org_name LIKE '%yale%' OR org_name LIKE '%new haven%' in a SQL WHERE clause.

res <- get_nih_data(req, max_pages = 1)
res %>% glimpse(width = getOption("cli.width"))

Notice the column organization is a nested data frame - it has 17 columns and always a single record. Setting flatten_result = TRUE in the call to get_nih_data() will flatten all such return fields, prefixing the original field name and returning with clean names (see janitor::clean_names()).

res <- get_nih_data(req,
                    max_pages = 1,
                    flatten_result = TRUE)

res %>% glimpse(width = getOption("cli.width"))

Most users will prefer the flattened format above. It looks like Yale is busy, but it is not the only org matching our search.

res %>% 
  group_by(organization_org_name) %>%
  summarise(project_count = n())

The org_names_exact_match criteria can be used as an alternative when we know the exact org name as it appears in RePORTER, if we want only that org's projects returned.

Ex. 3 - Geographic search

We can also filter projects by the geographic location (country/state/city) of the applicant organization.

## A valid request but probably not what we want
req <- make_req(criteria = 
                  list(
                    fiscal_years = 2010:2011,
                    include_active_projects = TRUE,
                    org_cities = "New Haven",
                    org_states = "WY"
                  ),
                include_fields = c("Organization", "FiscalYear", "AwardAmount"),
                message = FALSE ## suppress printed message
)

res <- get_nih_data(req,
                    max_pages = 5,
                    flatten_result = TRUE)

Multiple criteria are usually connected by logical "AND" - there are no orgs based in the city of New Haven in Wyoming state (because it doesn't exist.)

req <- make_req(criteria =
                  list(
                    fiscal_years = 2015,
                    include_active_projects = TRUE,
                    org_states = "WY"
                  ),
                include_fields = c("ApplId", "Organization", "FiscalYear", "AwardAmount"),
                sort_field = "AwardAmount",
                sort_order = "desc",
                message = FALSE)

res <- get_nih_data(req,
                    flatten_result = TRUE)

res %>% glimpse(width = getOption("cli.width"))

Why are there projects from more recent years than 2015? Because the include_active_projects flag adds in active projects that match all criteria aside from fiscal_years (this appears to be the intended behavior by RePORTER).

Ex. 3 - Coronavirus/Covid-19 research

We already provided one example of this search criteria above. Let's mix it up and request all Covid response projects.

## all projects funded by the Paycheck Protection Act, Coronavirus Response and Relief Act,
## and American Rescue Plan, in fiscal year 2021
req <- make_req(criteria =
                  list(covid_response = c("All")),
                include_fields = nih_fields %>%
                  filter(payload_name %in% c("award_amount_range", "covid_response"))
                %>% pull(include_name))

res <- get_nih_data(req)
res$covid_response %>% class()
res$covid_response[[1]]

covid_response is a nested list (with character vectors of variable length) within the return tibble. We can use flatten_result = TRUE here - elements of each vector will be collapsed to a single string delimited by ";", massaging the list to a single character vector.

## all projects funded by the Paycheck Protection Act, Coronavirus Response and Relief Act,
## and American Rescue Plan, in fiscal year 2021
req <- make_req(criteria =
                  list(covid_response = c("All")),
                message = FALSE)

res <- get_nih_data(req,
                    flatten_result = TRUE)

unique(res$covid_response)

Some projects are being funded from multiple sources. Summarizing all Covid-related project awards:

library(ggplot2)

res %>%
  left_join(covid_response_codes, by = "covid_response") %>%
  mutate(covid_code_desc = case_when(!is.na(fund_src) ~ paste0(covid_response, ": ", fund_src),
                                     TRUE ~ paste0(covid_response, " (Multiple)"))) %>%
  group_by(covid_code_desc) %>%
  summarise(total_awards = sum(award_amount) / 1e6) %>%
  ungroup() %>%
  arrange(desc(covid_code_desc)) %>%
  mutate(prop = total_awards / sum(total_awards),
         csum = cumsum(prop),
         ypos = csum - prop/2 ) %>%
  ggplot(aes(x = "", y = prop, fill = covid_code_desc)) +
  geom_bar(stat="identity") +
  geom_text_repel(aes(label =
                        paste0(dollar(total_awards,
                                      accuracy = 1,
                                      suffix = "M"),
                               "\n", percent(prop, accuracy = .01)),
                      y = ypos),
                  show.legend = FALSE,
                  nudge_x = .8,
                  size = 3, color = "grey25") +
  coord_polar(theta ="y") +
  theme_void() +
  theme(legend.position = "right",
        legend.title = element_text(colour = "grey25"),
        legend.text = element_text(colour="blue", size=6, 
                                   face="bold"),
        plot.title = element_text(color = "grey25"),
        plot.caption = element_text(size = 6)) +
  labs(caption = "Data Source: NIH RePORTER API v2") +
  ggtitle("Legislative Source for NIH Covid Response Project Funding")

A second dataset is provided to translate the covid_response codes; it includes both the long-form and a shorter version of the source name.

data("covid_response_codes")
covid_response_codes %>% print

Additional Resources

The full vignette contains a few more advanced examples, including boolean search functionality on the project title, abstract, and tagged project terms. It also includes an example which allows the user to retrieve complete result sets above the 10,000 record limit through by applying some basic statistics and a little programming.

The RePORTER web interface and official API documentation are useful for getting familiar with available search parameters

h/t to Chris whose code on github was all I could find existing in R and served as a starting point for this work

knitr::knit_exit()

bikeactuary/repoRter.nih documentation built on Feb. 6, 2023, 8:05 p.m.