knitr::opts_chunk$set( comment = "#> ", error = FALSE, cache = TRUE, warning = FALSE, tidy = TRUE )
The Berkeley Ecoengine (http://ecoengine.berkeley.edu) provides an open API to a wealth of museum data contained in the Berkeley natural history museums. This R package provides a programmatic interface to this rich repository of data allowing for the data to be easily analyzed and visualized or brought to bear in other contexts. This vignette provides a brief overview of the package's capabilities.
The API documentation is available at http://ecoengine.berkeley.edu/developers/. As with most APIs it is possible to query all the available endpoints that are accessible through the API itself. Ecoengine has something similar.
suppressPackageStartupMessages(library(ecoengine)) suppressPackageStartupMessages(library(pander))
library(ecoengine) ee_about()
pandoc.table(ee_about(), justify = "left")
The data functions in the package include ones that query obervations, checklists, photos, and vegetation records. These data are all formatted as a common S3
class called ecoengine
. The class includes 4 slots.
Total results on server
] A total result count (not necessarily the results in this particular object but the total number available for a particlar query)Args
] The arguments (So a reader can replicate the results or rerun the query using other tools.) Type
] The type (photos
, observation
, or checklist
) Number of results retrieved
] The data. Data are most often coerced into a data.frame
. To access the data simply use result_object$data
. The default print
method for the class will summarize the object.
For the sake of speed, results are paginated at 1000
results per page. It is possible to request all pages for any query by specifying page = all
in any function that retrieves data. However, this option should be used if the request is reasonably sized. With larger requests, there is a chance that the query might become interrupted and you could lose any data that may have been partially downloaded. In such cases the recommended practice is to use the returned observations to split the request. You can always check the number of requests you'll need to retreive data for any query by running ee_pages(obj)
where obj
is an object of class ecoengine
.
request <- ee_photos(county = "Santa Clara County", quiet = TRUE, progress = FALSE) # Use quiet to suppress messages. Use progress = FALSE to suppress progress bars which can clutter up documents. ee_pages(request) # Now it's simple to parallelize this request # You can parallelize across number of cores by passing a vector of pages from 1 through the total available.
x <- ee_observations(quiet = TRUE, progress = FALSE)
The database contains over 2 million records (r format(x$results, nsmall = 0)
total). Many of these have already been georeferenced. There are two ways to obtain observations. One is to query the database directly based on a partial or exact taxonomic match. For example
pinus_observations <- ee_observations(scientific_name = "Pinus", page = 1, quiet = TRUE, progress = FALSE) pinus_observations
For additional fields upon which to query, simply look through the help for ?ee_observations
. In addition to narrowing data by taxonomic group, it's also possible to add a bounding box (add argument bbox
) or request only data that have been georeferenced (set georeferenced = TRUE
).
lynx_data <- ee_observations(genus = "Lynx",georeferenced = TRUE, quiet = TRUE, progress = FALSE) lynx_data # Notice that we only for the first 1000 rows. # But since 795 is not a big request, we can obtain this all in one go. lynx_data <- ee_observations(genus = "Lynx", georeferenced = TRUE, page = "all", progress = FALSE) lynx_data
Other search examples
animalia <- ee_observations(kingdom = "Animalia") Artemisia <- ee_observations(scientific_name = "Artemisia douglasiana") asteraceae <- ee_observationss(family = "asteraceae") vulpes <- ee_observations(genus = "vulpes") Anas <- ee_observations(scientific_name = "Anas cyanoptera", page = "all") loons <- ee_observations(scientific_name = "Gavia immer", page = "all") plantae <- ee_observations(kingdom = "plantae") # grab first 10 pages (250 results) plantae <- ee_observations(kingdom = "plantae", page = 1:10) chordata <- ee_observations(phylum = "chordata") # Class is clss since the former is a reserved keyword in SQL. aves <- ee_observations(clss = "aves")
Additional Features
As of July 2014, the API now allows you exclude or request additional fields from the database, even if they are not directly exposed by the API. The list of fields are:
id
, record
, source
, remote_resource
, begin_date
, end_date
, collection_code
, institution_code
, state_province
, county
, last_modified
, original_id
, geometry
, coordinate_uncertainty_in_meters
, md5
, scientific_name
, observation_type
, date_precision
, locality
, earliest_period_or_lowest_system
, latest_period_or_highest_system
, kingdom
, phylum
, clss
, order
, family
, genus
, specific_epithet
,
infraspecific_epithet
, minimum_depth_in_meters
, maximum_depth_in_meters
, maximum_elevation_in_meters
, minimum_elevation_in_meters
, catalog_number
, preparations
, sex
, life_stage
, water_body
, country
, individual_count
, associated_resources
To request additional fields
Just pass then in the extra
field with multiple ones separated by commas.
aves <- ee_observations(clss = "aves", extra = "kingdom,genus") names(aves$data)
Similarly use exclude
to exclude any fields that might be returned by default.
aves <- ee_observations(clss = "aves", exclude = "source,remote_resource") names(aves$data)
Mapping observations
The development version of the package includes a new function ee_map()
that allows users to generate interactive maps from observation queries using Leaflet.js.
lynx_data <- ee_observations(genus = "Lynx", georeferenced = TRUE, page = "all", quiet = TRUE) ee_map(lynx_data)
The ecoengine also contains a large number of photos from various sources. It's easy to query the photo database using similar arguments as above. One can search by taxa, location, source, collection and much more.
photos <- ee_photos(quiet = TRUE, progress = FALSE) photos
The database currently holds r format(photos$results, nsmall = 0)
photos. Photos can be searched by state province, county, genus, scientific name, authors along with date bounds. For additional options see ?ee_photos
.
charles_results <- ee_photos(authors = "Charles Webber", quiet = TRUE, progress = FALSE) charles_results # Let's examine a couple of rows of the data charles_results$data[1:2, ]
view_photos(charles_results)
This will launch your default browser and render a page with thumbnails of all images returned by the search query. You can do this with any ecoengine
object of type photos
. Suggestions for improving the photo browser are welcome.
Other photo search examples
# All the photos in the CDGA collection all_cdfa <- ee_photos(collection_code = "CDFA", page = "all", progress = FALSE) # All Racoon pictures racoons <- ee_photos(scientific_name = "Procyon lotor", quiet = TRUE, progress = FALSE)
There is a wealth of checklists from all the source locations. To get all available checklists from the engine, run:
all_lists <- ee_checklists() head(all_lists[, c("footprint", "subject")])
Currently there are r nrow(all_lists)
lists available. We can drill deeper into any list to get all the available data. We can also narrow our checklist search to groups of interest (see unique(all_lists$subject)
). For example, to get the list of Spiders:
spiders <- ee_checklists(subject = "Spiders") spiders
Now we can drill deep into each list. For this tutorial I'll just retrieve data from the the two lists returned above.
library(plyr) spider_details <- ldply(spiders$url, checklist_details) names(spider_details) unique(spider_details$scientific_name)
Our resulting dataset now contains r length(unique(spider_details$scientific_name))
unique spider species.
The search is elastic by default. One can search for any field in ee_observations()
across all available resources. For example,
# The search function runs an automatic elastic search across all resources available through the engine. lynx_results <- ee_search(query = "genus:Lynx") lynx_results[, -3] # This gives you a breakdown of what's available allowing you dig deeper.
lynx_results <- ee_search(query = "genus:Lynx") pander::pandoc.table(lynx_results[[1]], justify = "left")
Similarly it's possible to search through the observations in a detailed manner as well.
all_lynx_data <- ee_search_obs(query = "Lynx", page = "all", progress = FALSE) all_lynx_data
Footprints
ee_footprints()
provides a list of all the footprints.
footprints <- ee_footprints() footprints[, -3] # To keep the table from spilling over
footprints <- ee_footprints() pandoc.table(footprints[, -3], justify = "left")
Data sources
ee_sources()
provides a list of data sources for the specimens contained in the museum.
source_list <- ee_sources() unique(source_list$name)
source_list <- ee_sources() pandoc.table(data.frame(name = unique(source_list$name)), justify = "left")
devtools::session_info()
Please send any comments, questions, or ideas for new functionality or improvements to <karthik.ram@berkeley.edu>. The code lives on GitHub under the rOpenSci account. Pull requests and bug reports are most welcome.
library(httr) x <- content(GET("http://ipinfo.io/"), as = "parsed")
Karthik Ram
r library(lubridate); as.character(lubridate::month(lubridate::now(), label = TRUE))
, r lubridate::year(lubridate::now())
r x$city
, r x$region
Any scripts or data that you put into this service are public.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.