In seandavi/OmicIDXR: Query the OmicIDX genomic data index API

Introduction to the OmicIDX API

OmicIDX parses and then serves public genomics repository metadata. These metadata are growing quickly, updated often, and are now very large when taken as a whole. Because of the interrelated nature of the metadata and the myriad approaches and use cases that exist, including search, bulk download, and even data mining, we serve the data via a GraphQL endpoint.

Currently, OmicIDX contains the SRA and Biosample metadata sets. These overlap with each other, but SRA metadata contains deeper metadata than Biosample data on the same samples. Biosample, on the other hand, contains many more samples and currently includes metadata about a subset of NCBI GEO, all SRA samples, and some additional samples from projects like Genbank.

GraphQL for accessing OmicIDX data

GraphQL is a query language for APIs and a runtime for fulfilling those queries with existing data. GraphQL provides a complete and understandable description of the data in the API, gives clients the power to ask for exactly what they need and nothing more, makes it easier to evolve APIs over time, and enables powerful developer tools.

GraphQL has only a single url, called the endpoint, which allows access to all data in the API. GraphQL is also a query language. It is the GraphQL query that is submitted to the GraphQL endpoint that results in data being returned.

What is a GraphQL query?

A GraphQL query looks a bit like JSON, except without quotes or commas. Here is an example GraphQL query for a fictitious GraphQL API.

{ 
  allCharacters {
    name
  }
}

If we had a server that contained Star Wars trivia, the response from the server might look like:

{ "data": {
    "allCharacters": [
      { 
        "name":"Luke"
      },
      { 
        "name": "Darth"
      },
      ...
    ]
  }
}

If we changed the query to:

{ 
  allCharacters {
    name
    mass
  }
}

the response would now look like:

{ "data": {
    "allCharacters": [
      { 
        "name":"Luke",
        "mass": 80
      },
      { 
        "name": "Darth",
        "mass": 140
      },
      ...
    ]
  }
}

How do I know what is in the GraphQL endpoint?

The GraphQL schema describes the data model(s) contained in the GraphQL endpoint. GraphQL is strongly typed, has the concept of relationships between data types, and is self-documenting. In fact, one can use the GraphQL endpoint to discover what is in the endpoint. I will not go into the details right now, but this introspection capability makes possible some powerful tooling. One of the most ubiguitous is the so-called GraphiQL (note the i in the name) tool.

Exercise 1

Navigate to GraphiQL and follow along with the video (no sound).[^urlchange]

[^urlchange]: The urls in this document are subject to change.

Querying OmicIDX programmatically

GraphQL is quite easy to work with programmatically. All queries are made via a post request to the GraphQL endpoint. The POST request needs to be JSON encoded and must include the "query" key. A simple example of the JSON for a basic query might look like:

{ "query": "{ heros { name weight } }"}

Note that the query string (ie., "{ heros { name weight } }") is just a string. It is not formated as JSON itself.

• Current OmicIDX GraphQL endpoint[^urlchange]: http://graphql-omicidx.cancerdatasci.org/graphql

For example, let us get the first 500 SRA studies (500 is a limit to the number of results that we can retrieve in one go. The GraphQL might look like this:

{
  allSraStudies {
    edges {
      node {
        accession
        title
        abstract
      }
    }
  }
}

From exercise 1, you know that you can copy this query into the GraphiQL browser and get results. How about using curl? Note that I am doing some gymnastics below to make this work in one command line. In practice, one would probably save the GraphQL query as a file (in json format) and then post that file[^curlnote].

[^curlnote]: *For examples of how to post JSON using curl, see this stackoverflow post

curl --silent \
  -X POST \
  -H "Content-Type: application/json" \
  http://graphql-omicidx.cancerdatasci.org/graphql \
  --data @- << EOF

  { "query":"{
  allSraStudies(first: 1) {
    edges {
      node {
        accession
        title
        abstract
      }
    }
  }
}"
}
 EOF

Note that the return value is simply JSON. All our normal tools for working with JSON are available to us to process and manipulate the results.

Using OmicIDX from R

As a review, we know that:

• GraphQL queries look like JSON, but are not quite the same.
• We can use the GraphiQL graphical tool to help us write our query, including autocompleting fields, etc.
• GraphQL queries are POSTed to the graphql endpoint and results are returned as JSON.

In order to query from R, then, we need to:

1. Define our query.
2. POST the query, encoded as a simple JSON data structure.
3. Deal with the resulting JSON data that comes back as a result of the post in #2.

Defining the query

Again, the easiest way to define the query is to experiment witht he GraphiQL query tool. Once a query works as expected, the query can be reused from R.

In this case, we are going to continue with the query we worked with above:

{
  allSraStudies(first: 1) {
    edges {
      node {
        accession
        title
        abstract
      }
    }
  }
}

Performing the query

The CRANpkg('httr') package performs http requests, including POST requests.

With our query defined, we need to prepare the POST body which will be sent to the server as JSON. In R, basicc lists are the information equivalent of JSON objects.

post_body = list(query = "
{
  allSraStudies(first: 20) {
    edges {
      node {
        accession
        title
        sraExperimentsByStudyAccession {
          totalCount
        }
      }
    }
  }
}
")

The URL for the OmicIDX endpoint is: http://graphql-omicidx.cancerdatasci.org/graphql.[^urlchange]

endpoint = "http://graphql-omicidx.cancerdatasci.org/graphql"

I am going to lead the reader a bit here and jump to a "handler" function that will convert the returned JSON into convenient R data structures.

handler = function(response) {
  jsonlite::fromJSON(httr::content(response, as='text'),flatten = TRUE)
}

Finally, we are ready to perform our query.

resp = httr::POST(endpoint, body = post_body, encode = 'json')
resp

And finally, convert the response to convenient R data structure.

result = handler(resp)
knitr::kable(result$data[[1]]$edges)

We can change our query a bit to get some more details.

study_experiment_counts_query = "
{
  allSraStudies(first: 10) {
    edges {
      node {
        accession
        title
        sraExperimentsByStudyAccession {
          totalCount
        }
      }
    }
  }
}
"

And streamline our code a bit as a function.

# Borrow "handler" function from above
# Borrow "endpoint" URL from above
graphql_query = function(gql) {
  .handler = function(response) {
    jsonlite::fromJSON(httr::content(response, as='text'),flatten = TRUE)$data[[1]]$edges
  }
  post_body = list(query = gql)
  resp = httr::POST(endpoint, body = post_body, encode = 'json')
  return(.handler(resp))
}

knitr::kable(graphql_query(study_experiment_counts_query))

One might be interested in filtering the results, also. For example, to search study titles for "cancer" (case insensitive) we can add a filter to the query.

cancer_studies_query = '
{
  allSraStudies(first: 10 filter: {title: {includesInsensitive: "cancer"}}) {
    edges {
      node {
        accession
        title
      }
    }
  }
}
'
knitr::kable(graphql_query(cancer_studies_query))

To check the total count of available studies with "cancer" in the title, we can again write a query for that. Note that this query

cancer_studies_query = '
{
  allSraStudies(first: 10 filter: {title: {includesInsensitive: "cancer"}}) {
    totalCount
  }
}
'

A keen eye will notice that there is not an edges component in this query. Our handler above returns the edges component, though. So, we are going to drop back to basic httr code here, again.

httr::content(
  httr::POST(endpoint, body = list(query=cancer_studies_query), encode='json'),
  as="parsed"
)

Related objects

One of the unique aspects of GraphQL is the ability to treat data as a graph, with data entities related to each other logically linked. For example, we can fetch SRA experiments and link them each to the study under which they were performed.

experiments_with_study = '
{
  allSraExperiments(first: 5) {
    edges{
      node{
        accession
        title
        libraryStrategy
        sraStudyByStudyAccession{
          title
          accession
        }
      }
    }
  }
}
'

The graphql query from R still works the same way, so we execute as before.

result = graphql_query(experiments_with_study)
knitr::kable(result)

Fetching all results

TODO

• Describe cursors

Using variables

TODO

• Describe adding variables to POST body

Conclusion

TODO

seandavi/OmicIDXR documentation built on May 21, 2019, 11:42 a.m.