In Huh/collar: Utilities for Accessing and Manipulating GPS Data

Purpose

Data collected by tags attached to animals often relay data to users via satellite modems and email. Email is a cumbersome way to receive large amounts of data. For this reason manufacturers typically create a graphical user interface (GUI), which requires the user to visit a website, click on a series of buttons and download individual files. This workflow is better than email, but onerous if more than a few collars are deployed at a time. A more ideal workflow would be for a server to wake up in the middle of the night contact another server and download data in an automated fashion. Few companies, namely Vectronics, have developed application program interfaces (i.e. API) to solve this problem and facilitate the latter more desirable workflow. This package is a workaround for those using tags from companies that do not supply an API and in the case of Vectronics we offer a simple solution to interacting with the API via R. Writing this package in R (and using S3) is important because of the prevalance of R in wildlife science.

Roadmap

When creating this package we imagined a simple interface that made it easy for users to retrieve their data, but we realize that this is only the beginning and are working to create a series of tools for the standardization of data, visualization and analysis of telemetry derived locations. For more information on where we are headed or to give us your two cents visit the GitHub repository for the project at https://github.com/Huh/collar.

Companies

At the time of this writing we have tools to download data from Vectronics, ATS, Lotek and legacy csv files.

Vectronics

The Vectronics API is documented here. A basic request to API is just a URL like www.google.com except that it also specifies some parameters, which tell the receiving server to do something. Let's take a look

https://api.vectronic-wildlife.com/collar/{collarid}/{datatype}?collarkey={key}

Looking at the request we see that we are going to securely pass data via the encrypted https protocol. Then we see the base url for the server and it specifies port 9443. After that v2/collar/ and we don't need to know much about this chunk as we suspect it is for internal routing. Finally we get to some information that we are required to provide, the collar id. A collar id often looks like 1000001. After passing the collar id we will need to provide the datatype we want to retrieve. Data types depend on the device type, but generally include the following:

1) activity data 2) mortality implant data 3) mortality data 4) proximity data 5) position data 6) separation data 7) trap event data 8) vaginal implant data

The last piece of information we are required to pass is the collar key. A key is a bit of XML (eXtensible Markup Language) that allows Vectronics to identify each unique device and return the relevant data. Thankfully Vectronics provides a test key that we can all use to test the API. If you have your own keys that you want to use go for it, but for this example we provide an example test key in the package. The example key can be located at:

system.file("extdata", "Collar1000001_Registration.keyx", package = "collar")

We are now ready to download data from the test collar. The key_dir argument of the fetch_vectronics function actually needs a directory, not a single key.

Load the package

library(collar)

Call the API

 key_paths <- get_paths(
    system.file(
      "extdata",
      package = "collar"
    )
  )
 vec_dat <- fetch_vectronics(key_paths)

Cool, but what just happened? Let's break down the function call into its pieces. First, I told you that the API required a collar id, a collar key and a data type, but in our call we only supplied paths to key files. The collar package read the key files, extracted the collar id from the collar key, then it extracted the long alphanumeric key and finally it guessed that you wanted GPS location data because no value was supplied. Using these pieces of information the fetch_* function builds a URL and calls the API. The function can handle many keys at once or a single key if desired. The function returns a tibble, which prints nicely to the screen and does not coerce or recycle values in an unreasonable way. It also provides a means of passing a function to make pretty column names. The default removed all non-ASCII characters, like the degree symbol, made all letters lower case and replaced . with _. If you don't like that convention you can write your own function or pass something like make.names or tolower.

The organization of calls to the Vectronic API can be organized into a simple dichotomous key. First, the user needs to decide what kind of data they desire, in the first example gps. Then the next decision is if you want to retrieve the data or a count of how much data is available, this is controlled by the count argument. Once that decision is made decide if all data is required or some subset of data is best. If a subset then choose to implement either data after a positionid or data after some date. This quick summary may be helpful as you get more familiar with the API.

Data ID Limits

More advanced calls to the API can be made to only download subsets of the data. For example, we could assume the 600th value of idPosition came from the last download we performed. Imagine that last time we downloaded data this was the last value we received, we can pass this value to the fetch_vectronics function to limit the download to only new data not previously downloaded. Note: Some users have reported odd behavior if positions were collected, but not transmitted. It is possible that smaller/older values of idPosition were previously missed. Don't assume that you downloaded every point ever collected.

new_dat <- fetch_vectronics(
  key_paths,
  after_data_id = vec_dat$idposition[600]
)

Date Limits

Similarly, we can use dates to limit data downloads. The format of the date must be YYYY-MM-DDTHH:MM:SS and yes, the T is necessary. The scenario is the same, imagine we download data regularly, say daily, and we only want to download data with a scts date greater than some value. Below we create a date to use for downloading by subtracting 10 from the maximum observed date in the vec_dat created by the first download.

Create the date object used to subset the download

after <- format(max(as.Date(vec_dat$acquisitiontime)) - 10, "%Y-%m-%dT%H:%M:%S")

Call the API with the start date

after_scts <- fetch_vectronics(
  key_paths,
  start_date = after,
  which_date = "scts"
)

The same could be accomplished with the acquisition date by changing the call to

after_acq <- fetch_vectronics(
  key_paths,
  start_date = after,
  which_date = "acquisition"
)

If you have more than one collar to download then the length of the dates passed to start_date must be equal to the number of keys found in the key directory. In other words, you have to have one date for each tag.

Data Types

Users can choose to download data of different types using they type argument. The possible options are:

1) gps - GPS location/position data 2) act - Activity data 3) mit - Implant mortality data 4) mor - Mortality data 5) prx - Proximity data 6) sep - Separation data 7) trap- Trap event data 8) vit - Vaginal implant data

Note: The count argument modifies all data requests to return only the quantity of data available per key file.

Definitions of the data types can be found in the API documentation. So far, the calls above have implicitly called the API requesting GPS data, but what if we want activity data? A simple change to the fetch_vectronics call accomplishes this task.

act_dat <- fetch_vectronics(
  key_paths,
  type = "act"
)

The pattern is the same for the remaining data types.

---

ATS

Authentication

Use ats_login with your username and password to log in to your ATS account. There's no need to assign the return value of this function to a variable.

library(collar)
# ATS changed the demo password on us
# This is just a workaround to make the vignette work
ok <- try(ats_login(usr = "mary", pwd = ".")) == TRUE
# Normally this would be fine:
# ats_login(usr = "mary", pwd = ".")

Your login information is stored in a cookie what will remain active for the duration of your R session. If you experience problems retrieving your data you can always run the same code again to log back in to your account.

To log out use ats_logout(). None of the data retrieval functions detailed below will work if you're not logged in.

ats_logout()

Retrieving Events

Use fetch_ats_events to get a list of events (a.k.a. alerts) from the server. No parameters are required:

ok <- try(ats_login(usr = "mary", pwd = ".")) == TRUE
if (ok) {
  alerts <- fetch_ats_events()
  alerts
}

Retrieving A List of Available Devices

As with alerts above, use fetch_ats_devices to see a list of devices (collars) associated with your account:

if (ok) {
  collars <- fetch_ats_devices()
  collars
}

You can also use the filter parameter to filter the list of devices. For example to retrieve a list of devices in mortality status:

if (ok) {
  collars <- fetch_ats_devices("mort")
  collars
}

See ?fetch_ats_devices for the list of available filters.

Retrieving GPS Data

GPS data can be retrieved using fetch_ats_positions, which provides several parameters to limit the data returned. Running this function without parameters may take a very long time depending on the number of positions being returned. Use the device_id parameter to filter by collar:

if (ok) {
  # download position data for collar 44286
  fixes <- fetch_ats_positions(device_id = "044286")
  nrow(fixes)
  # download position data for 3 collars at once
  collars <- c("044286", "044288", "044290")
  fixes <- fetch_ats_positions(device_id = collars)
  nrow(fixes)
  min(fixes$CollarSerialNumber)
  max(fixes$CollarSerialNumber)
}

Use the n parameter to retrieve the last 5 or last 10 fixes (data is only returned for active collars):

if (ok) {
  # download last 10 fixes (all active collars)
  fixes <- fetch_ats_positions(n = 10)
  nrow(fixes)
  # download last 10 fixes for certain collars
  fixes <- fetch_ats_positions(device_id = collars, n = 10)
  nrow(fixes)
}

You can also download new data only using new = TRUE. Start date and end date parameters are included in collar, but due to an internal server error on the ATS website they're currently ignored. These parameters will be enabled in a future version once the error has been resolved.

See ?fetch_ats_positions for more details and a description of the output format.

Retrieving Transmission Data

ATS also allows users to download information about the satellite transmissions that have been sent from ATS collars. This data is used in ats_fetch_positions to determine the GMT offset and detect column substitutions. You can access the data yourself using (you guessed it) fetch_ats_transmissions. As with ats_fetch_positions parameters allow you to filter by collar or include only new data.

if (ok) {
  # get undownloaded transmissions for all collars
  trans <- fetch_ats_transmissions(new = TRUE)
  nrow(trans)
  # get all transmissions for certain collars
  trans <- fetch_ats_transmissions(device_id = collars)
  nrow(trans)
}

As above, don't forget to check out ?fetch_ats_transmissions for more details and a description of the output format.

Retrieving Collar Configuration Details

ATS also provides a very limited set of collar configuration information for download. This information is unlikely to be useful in R, but it's so there so we included it. The function for retrieving this data is called fetch_ats_config:

if (ok) {
  head(fetch_ats_config())
}
ats_logout()

CSV Legacy/Loose Files

Sometimes we just have a bunch of loose csv files from old collars or maybe we received them by email regardless of how we got them we want to use them. The function fetch_csv is a wrapper for just this task. One special and undesirable feature of some collar data is that it has a header of many lines containing metadata of some sort. Some examples are contained in this package. You can run system.file("extdata", "telonics.csv", package = "collar") to see a nice example of an extra large header that makes life difficult.

A clean first example uses a file without a problematic header

lotek_fpath <- system.file(
  "extdata",
  "lotek.csv",
  package = "collar",
  mustWork = TRUE
)

lotek <- fetch_csv(lotek_fpath)

The function fetch_csv allows us to read multiple csv's.

fpaths <- list.files(dirname(lotek_fpath), full.names = T, pattern = "csv$")

my_dat <- fetch_csv(fpaths[1:2])

What if your file has a large header like the Telonics file referenced earlier? Well, we built a function to guess at the number of rows to omit when reading files with headers. Let's try it with the Telonics file provided with this package.

telonics_fpath <- system.file(
  "extdata",
  "telonics.csv",
  package = "collar",
  mustWork = TRUE
)

tel_dat <- fetch_csv(telonics_fpath)

That was ugly, but we have the cllr_remove_header to help with this situation. In order to use the function we just need to know one column name. For example, if you open the csv and look at it we see that one column name is GPS Latitude. Now let's remove the header from our data, note that the column name is unquoted and because it has a space in the name we use back ticks.

clean_tel <- cllr_remove_header(tel_dat, `GPS Latitude`, TRUE)

Now our data no longer have the large header with all the metadata. At this point we may wish to add a unique identifier for each animal.

id_tel <- cllr_add_id(clean_tel, "Dave")

Issues

If you have issues with package or want to see a new feature please leave a note at https://github.com/Huh/collar/issues.

Huh/collar documentation built on Aug. 5, 2022, 11:02 p.m.