The synapser
package provides an interface to Synapse, a collaborative workspace for reproducible data intensive research projects, providing support for:
The synapser
package lets you communicate with the Synapse platform to create collaborative data analysis projects and access data using the R programming language. Other Synapse clients exist for Python, Java, and the web browser.
If you're just getting started with Synapse, have a look at the Getting Started guides for Synapse.
Good example projects are:
synapser
is available as a ready-built package for Microsoft Windows and Mac OSX. For Linux systems, it is available to install from source. It can be installed or upgraded using the standard install.packages()
command, adding the Sage Bionetworks R Archive Network (RAN) to the repository list, e.g.:
install.packages("synapser", repos = c("http://ran.synapse.org", "https://cloud.r-project.org"))
Alternatively, edit your ~/.Rprofile
and configure your default repositories:
options(repos = c("http://ran.synapse.org", "https://cloud.r-project.org"))
after which you may run install.packages
without specifying the repositories:
install.packages("synapser")
For a detailed installation guide see installation vignette. Please refer to the Troubleshooting guide for more information.
To use Synapse, you'll need to register for an account. The Synapse website can authenticate using a Google account. If you authenticate using a Google account, you'll need to create a personal access token to log in to Synapse through the programmatic clients. See the Manage Synapse Credentials vignette for more information.
Once that's done, you'll be able to load the library and login:
library(synapser) synLogin()
For more ways to manage your Synapse credentials, please see the Manage Synapse Credentials vignette, and the native reference documentation:
?synLogin ?synLogout
To make the example below print useful information, we prepare a file:
# use hex_digits to generate random string hex_digits <- c(as.character(0:9), letters[1:6]) projectName <- sprintf("My unique project %s", paste0(sample(hex_digits, 32, replace = TRUE), collapse = "")) project <- Project(projectName) project <- synStore(project) # Create some files filePath <- tempfile() connection <- file(filePath) writeChar("a \t b \t c \n d \t e \t f \n", connection, eos = NULL) close(connection) file <- File(path = filePath, parent = project) # Add a version comment file$properties$versionComment <- 'Some sort of comment about the new version of the file.' file <- synStore(file) synId <- file$properties$id
Synapse identifiers are used to refer to projects and data which are represented by entity objects. For example, the entity above represents a tab-delimited file containing a 2 by 3 matrix. Getting the entity retrieves an object that holds metadata describing the matrix, and also downloads the file to a local cache:
fileEntity <- synGet(synId)
View the entity's metadata in the R console:
print(fileEntity)
This is one simple way to read in a small matrix (we load just the first few rows):
read.table(fileEntity$path, nrows = 2)
View the entity in the browser:
synOnweb(synId)
By default the download location will always be in the Synapse cache. You can specify the downloadLocation parameter.
entity <- synGet("syn00123", downloadLocation = "/path/to/folder")
For more details see the native reference documentation, e.g.:
?synGet ?synOnweb
You can create your own projects and upload your own data sets. Synapse stores entities in a hierarchical or tree structure. Projects are at the top level and must be uniquely named:
project <- Project(projectName) project <- synStore(project)
Creating a folder:
dataFolder <- Folder("Data", parent = project) dataFolder <- synStore(dataFolder)
Adding files to the project:
filePath <- tempfile() connection <- file(filePath) writeChar("this is the content of the file", connection, eos = NULL) close(connection) file <- File(path = filePath, parent = dataFolder) file <- synStore(file)
You can print the properties of an entity (such as the file we just created):
file$properties
Most other properties are immutable, but you can change an entity's name:
file$properties$name <- "different name"
Update Synapse with the change:
file <- synStore(file) file$properties
You can list all children of an entity:
children <- synGetChildren(project$properties$id) as.list(children)
You can also filter by type:
filesAndFolders <- synGetChildren(project$properties$id, includeTypes = c("file", "folder")) as.list(filesAndFolders)
You can avoid reading all children into memory at once by iterating through one at a time:
children <- synGetChildren(project$properties$id) tryCatch({ while (TRUE) { child <- nextElem(children) print(child) } }, error = function(e) { print("Reached end of list.") })
You can move files to a different parent:
newFolder <- Folder("New Parent", parent = project) newFolder <- synStore(newFolder) file <- synMove(file, newFolder)
Content can be deleted:
synDelete(file)
Deletion of a project will also delete its contents, in this case the folder:
folderId <- dataFolder$properties$id synDelete(project) tryCatch( synGet(folderId), error = function(e) { message(sprintf("Retrieving a deleted folder causes: %s", as.character(e))) }, silent = TRUE )
In addition to simple data storage, Synapse entities can be annotated with key/value metadata, described in markdown documents (wikis), and linked together in provenance graphs to create a reproducible record of a data analysis pipeline.
For more details see the native reference documentation, e.g.:
?Project ?Folder ?File ?Link ?synStore
# (We use a time stamp just to help ensure uniqueness.) projectName <- sprintf("My unique project created on %s", format(Sys.time(), "%a %b %d %H%M%OS4 %Y")) project <- Project(projectName) # This will erase all existing annotations project$annotations <- list(annotationName = "annotationValue") project <- synStore(project)
project <- synGet(project$properties$id) project$annotations synGetAnnotations(project)
Synapse provides tools for tracking 'provenance', or the transformation of raw data into processed results, by linking derived data objects to source data and the code used to perform the transformation.
The Activity object represents the source of a data set or the data processing steps used to produce it. Using W3C provenance ontology terms, a result is generated by a combination of data and code which are either used or executed.
act <- Activity( name = "clustering", description = "whizzy clustering", used = c("syn1234", "syn1235"), executed = "syn4567")
Here, syn1234 and syn1235 might be two types of measurements on a common set of samples. Some whizzy clustering code might be referred to by syn4567.
Alternatively, you can build an activity up piecemeal:
act <- Activity(name = "clustering", description = "whizzy clustering") act$used(c("syn12345", "syn12346")) act$executed("syn4567")
The used and executed can reference entities in Synapse or URLs.
Entity examples:
act$used("syn12345") act$used(project) act$used(target = "syn12345", targetVersion = 2)
URL examples:
act$used("http://mydomain.com/my/awesome/data.RData") act$used(url = "http://mydomain.com/my/awesome/data.RData", name = "Awesome Data") act$used(url = "https://github.com/joe_hacker/code_repo", name = "Gnarly hacks", wasExecuted = TRUE)
The activity can be passed in when storing an Entity to set the Entity’s provenance:
project <- synGet(project$properties$id) project <- synStore(project, activity = act)
We’ve now recorded that 'project' is the output of syn4567 applied to the data stored in syn1234 and syn1235.
The synStore() has shortcuts for specifying the used and executed lists directly. For example, when storing a data entity, it’s a good idea to record its source:
project <- synStore( project, activityName = "data-r-us", activityDescription = "downloaded from data-r-us", used = "http://data-r-us.com/excellent/data.xyz")
For more information:
?Activity ?synDeleteProvenance
Tables can be built up by adding sets of rows that follow a user-defined schema and queried using a SQL-like syntax. Please visit the Table vignettes for more information.
Wiki pages can be attached to an Synapse entity (i.e. project, folder, file, etc). Text and graphics can be composed in markdown and rendered in the web view of the object.
Creating a Wiki
project <- synGet(project$properties$id) content <- " # My Wiki Page Here is a description of my **fantastic** project! " # attachment filePath <- tempfile() connection <- file(filePath) writeChar("this is the content of the file", connection, eos = NULL) close(connection) wiki <- Wiki(owner = project, title = "My Wiki Page", markdown = content, attachments = list(filePath)) wiki <- synStore(wiki)
Updating a Wiki
project <- synGet(project$properties$id) wiki <- synGetWiki(project) wiki.markdown <- " # My Wiki Page Here is a description of my **fantastic** project! Let's *emphasize* the important stuff. " wiki <- synStore(wiki)
For more information:
?Wiki ?synGetWiki
An Evaluation is a Synapse construct useful for building processing pipelines and for scoring predictive modeling and data analysis challenges.
Creating an Evaluation:
eval <- Evaluation( name = sprintf("My unique evaluation created on %s", format(Sys.time(), "%a %b %d %H%M%OS4 %Y")), description = "testing", contentSource = project$properties$id, submissionReceiptMessage = "Thank you for your submission!", submissionInstructionsMessage = "This evaluation only accepts files.") eval <- synStore(eval)
Retrieving the created Evaluation:
eval <- synGetEvaluation(eval$id) eval
Submitting a file to an existing Evaluation:
# first create a file to submit filePath <- tempfile() connection <- file(filePath) writeChar("this is my first submission", connection, eos = NULL) close(connection) file <- File(path = filePath, parent = project) file <- synStore(file) # submit the created file submission <- synSubmit(eval, file)
List submissions:
submissions <- synGetSubmissionBundles(eval) as.list(submissions)
Retrieving submission by id:
# Not evaluating this section because of SYNPY-235 submission <- synGetSubmission(submission$id) submission
Retrieving the submission status:
submissionStatus <- synGetSubmissionStatus(submission) submissionStatus
To view the annotations:
submissionStatus$submissionAnnotations
To update an annotation:
submissionStatus$annotations["doubleAnnos"] <- list(c("rank" = 3)) synStore(submissionStatus)
Query an evaluation:
queryString <- sprintf("query=select * from evaluation_%s LIMIT %s OFFSET %s'", eval$id, 10, 0) synRestGET(paste("/evaluation/submission/query?", URLencode(queryString), sep = ""))
To learn more about writing an evaluation query, please see: http://docs.synapse.org/rest/GET/evaluation/submission/query.html
For more information, please see:
?synGetEvaluation ?synSubmit ?synGetSubmissionBundles ?synGetSubmission ?synGetSubmissionStatus
By default, data sets in Synapse are private to your user account, but they can easily be shared with specific users, groups, or the public.
Retrieve the sharing setting on an entity:
synGetAcl(project, principal_id = "273950")
The first time an entity is shared, an ACL object is created for that entity. Let's make project public:
acl <- synSetPermissions(project, principalId = 273949, accessType = list("READ")) acl
Now public can read:
synGetAcl(project, principal_id = 273950)
Get permissions will obtain more human-readable view of an entity's permissions
permissions = synGetPermissions(project) permissions$can_view
?synGetAcl ?synSetPermissions ?synGetPermissions
synDelete(project)
A file view can be defined by its scope. It allows querying for FileEntity within the scope using a SQL-like syntax. Please visit the Views vignettes for more information.
These methods enable access to the Synapse REST(ish) API taking care of details like endpoints and authentication. See the REST API documentation.
?synRestGET ?synRestPOST ?synRestPUT ?synRestDELETE
We provide some utility functions in the synapserutils package:
Please visit the synapserutils Github repository for instructions on how to download.
For more information see the Synapse User Guide.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.