How to create new _AnnotationHub_ resources

How to write recipes for new AnnotationHub resources

Overview of the process

This vignette is for those wishing to process online resources into AnnotationHub resources. There are four steps involved in creating resources.

Setup

This vignette is intended for users who are comfortable with R. Be sure to use Bioc-devel. Install the AnnotationHubData package using biocLite("AnnotationHubData").

Introducing AnnotationHubMetadata Objects

The AnnotationHubData package provides a place where we store code to processes online resources into AnnotationHub resources. AnnotationHubMetadata objects are used to describe an online resource, including a function ('recipe') that creates the resource.

The steps involved include writing a recipe are:

  1. Write a function which takes the metadata about the resource and processes them into AnnotationHubMetadata objects
  2. Optional step : Write an additional function specifying how original source files are pre-processed to R or other objects. Not all source files are pre-processed; one might pre-process data to make it easier or faster for the user to access the resource.
  3. Optional step: Write a function specifying how a resource is to be read in to the user's R session after being downloaded to a user's local cache.

Step 1: Writing AnnotationHubMetadata Generating Functions

The following example function takes files from the latest release of inparanoid and processes them into a list of AnnotationHubMetadata objects (e.g., using Map()).

The function body can contain hard-coded information, but many recipes will 'scrape' internet resources, creating many AnnotationHub resources; in the example below, .inparanoidMetadataFromUrl() visits the inparanoid web site to find current data. The Recipe: field indicates the function that will perform optional step 2, transforming the original source file(s) to an object that is more convenient for user input. This function returns a list of AnnotationHubMetadata objects.

makeinparanoid8ToAHMs <-
   function(currentMetadata, justRunUnitTests=FALSE, BiocVersion=biocVersion())
{
    baseUrl <- 'http://inparanoid.sbc.su.se/download/current/Orthologs_other_formats'
    ## Make list of metadata in a helper function
    meta <- .inparanoidMetadataFromUrl(baseUrl)
    ## then make AnnotationHubMetadata objects.
    Map(AnnotationHubMetadata,
        Description=meta$description,
        Genome=meta$genome,
        SourceFile=meta$sourceFile,
        SourceUrl=meta$sourceUrl,
        SourceVersion=meta$sourceVersion,
        Species=meta$species,
        TaxonomyId=meta$taxonomyId,
        Title=meta$title,
        RDataPath=meta$rDataPath,
        MoreArgs=list(
          Coordinate_1_based = TRUE,
          DataProvider = baseUrl,
          Maintainer = "Marc Carlson <mcarlson@fhcrc.org>",
          RDataClass = "SQLiteFile",
          RDataDateAdded = Sys.time(),
          RDataVersion = "0.0.1",
          Recipe = "AnnotationHubData:::inparanoid8ToDbsRecipe",
          Tags = c("Inparanoid", "Gene", "Homology", "Annotation")))
}

Here is a listing of AnntotationHubMetadata arguments:

Step 2: Recipes for Pre-Processing Resources

A (optional) recipe function transformed the source data into an object served by AnnotationHub to the user. It takes a single AnnotationHubMetadata object as an argument. Below is a recipe that generates an inparanoid database object from the metadata stored in it's AnnotationHubMetadata object. Note that the recipe will be invoked on the hub, so should output data to the location specified in the input AnnotationHubMetadata object.

inparanoid8ToDbsRecipe <-
    function(ahm)
{
    require(AnnotationForge)
    inputFiles <- metadata(ahm)$SourceFile
    dbname <- makeInpDb(dir=file.path(inputFiles,""),
                        dataDir=tempdir())
    db <- loadDb(file=dbname)
    outputPath <- file.path(metadata(ahm)$AnnotationHubRoot,
                            metadata(ahm)$RDataPath)
    saveDb(db, file=outputPath)
    outputFile(ahm)
}

Note for Steps 1 and 2

While writing this function, care has to be taken for a couple of fields:

Case 1 (common) - The AnnotationHub resource is downloaded directly to the user cache without any pre-processing, then

  1. SourceUrls specify the original resource locatoin = Location_Prefix + RDataPath
  2. Recipe = NA_character_

Example -

SourceUrls = "http://hgdownload.cse.ucsc.edu/goldenPath/hg38/liftOver/hg38ToRn5.over.chain.gz"
Location_Prefix = "http://hgdownload.cse.ucsc.edu/",
RDataPath = "goldenPath/hg38/liftOver/hg38ToRn5.over.chain.gz"
Recipe = NA_character_

Case 2 - The AnnotationHub resource requires pre-processing

  1. SourceUrls should merely document the original location of the untouched file
  2. Location_Prefix + RDataPath should be equal to the file path on the amazon machine where all pre-processed files are stored.
  3. Recipe = helper function which tells us how to pre-process the original file

Example -

SourceUrls="http://hgdownload.cse.ucsc.edu/goldenPath/hg38/liftOver/hg38ToRn5.over.chain.gz",
Location_Prefix = "http://s3.amazonaws.com/annotationhub/",
RDataPath="chainfile/dummy.Rda"

If this seems confusing, please note how in both of these cases, the sourceUrl needs to reflect the location that the resource will actually come from once when the client is in use.

Step 3: Post-Processing Resources

One can post-process a file when it is instantiated into AnnotationHub from the user's cache. An example, would be a BED file, downloaded to the user's cache but input into R as a GRanges using rtrackler::import. Implement this by defining a class that extends AnnotationHubResource and that implements a get1() method.

setClass("BEDFileResource", contains="AnnotationHubResource")

setMethod(".get1", "BEDFileResource",
    function(x, ...)
{
    .require("rtracklayer")
    yy <- getHub(x)
    dat <- rtracklayer::BEDFile(cache(yy))
    rtracklayer::import(dat, format="bed", genome=yy$genome, ...)
})

The class and method definition typically need to be added to AnnotationHub, and require coordination with us.

Step 4: Testing and Next Steps

At this point make sure that the AnnotationHubMetadata generating function produces a list of AnnotationHubMetadata objects and that the recipe (if needed) produces an appropriate output path. Contact us to add your recipe to the production hub.

Session Information

sessionInfo()


Try the AnnotationHubData package in your browser

Any scripts or data that you put into this service are public.

AnnotationHubData documentation built on April 17, 2021, 6:05 p.m.