In irwinsnet/FIRST_api_R: Tools to Download and Convert FIRST API Data to R Data Frames

The firstapiR package provides R functions that download FIRST Robotics Competition (FRC) data from the FIRST API server via hypertext transfer protocol (HTTP) requests. The firstapiR package formats the resulting data as JSON text, XML text, or as R data frames.

Version 2.0.1

This guide is for version 2.0.1 of the firstapiR package. Guidance for version 1.0.0 of firstapiR is located here .

FIRST - For Inspiration and Recognition in Science and Technology

FIRST engages kids in kindergarten through high school in exciting, mentor-based, research and robotics programs that help them become science and technology leaders, as well as well-rounded contributors to society. Learn more about FIRST by visiting their website.

FRC is one of four FIRST programs. In FRC, high-school students, in cooperation with adult mentors, design and build robots and operate them in competitions. The competitions have detailed rules (check out the 2016 game manual ) and an extensive amount of data is generated at each competition. All scheduling, team, scoring, and ranking data is stored in FIRST servers and is made available to users via the FIRST API.

Getting Access to the FIRST API

To see what the FIRST API can do, visit the FIRST API documentation page . The documentation page provides detailed information on the formats for HTTP request and authorization headers, as well as detailed descriptions of the data provided by each type of API call.

To use the FIRST API (via firstapiR functions or via HTTP requests that you build yourself), you must have a username and authorization key issued by FIRST. To get a username and authorization key, go to the FIRST Community Developers page on TeamForge . Follow the instructions on the page to join the project and FIRST will email you a username and authorization key. Make sure you keep your authorization key secret - don't post the key anywhere that is publicly accessible.

Learning R

R is a functional language intended for statistical computing, data analysis and plotting. I initially considered building this package in Python, but in the end I wanted to learn to use R, so here we are. R seems a bit quirky at first, especially compared to other languages like Java or Python, but it has some useful, built-in features for working with data. R has an extensive array of free add-in packages for manipulating and visualizing data, plus there is an excellent and free IDE available for R, RStudio.

If you are new to R, experimenting with firstapiR isn't the worst way to get started, but there are some resources you should check out:

Download R from the main R website.
Download and use the free version of RStudio.
To learn R, I recommend starting with the Quick R Website
Next, check out the R introduction on the main R documentation page
Everything written by Hadley Wickham is excellent. His Advanced R website, R package website and his testthat and devtools packages were essential to the development of the firstapiR package.

Installing the firstapiR package

Install the devtools package. This is easy to do in RStudio - just select Tools->Install Packages... from the menu bar, type "devtools" in the package name box and hit Install.
The devtools package provides a function for installing R packages from GitHub. Type the following command at the RStudio console prompt:

devtools::install_github("irwinsnet/firstapiR")

If you need the prior version of firstapiR, use this command instead:

devtools::install_github("irwinsnet/firstapiR", ref = "v1.0.0")

You can also download source or binary versions of the firstapiR package from this website. Type help(install.packages) in the R or Rstudio console to see the documentation for installing packages.

Differences between Versions 1.0.0 and 2.0.1

Version 1.0.0 of firstapiR is still available from github per the firstapiR installation instructions above.
The main difference from version 1.0.0 is that there are three shapes (i.e, types) of Schedule, HybridSchedule, and MatchResult data frames. The three shapes are team, alliance, and match. Team data frames have one row per team, requiring six rows to contain all of the data for a single match. The station column specifies the station to which the team was assigned. Alliance data frames have three teams per row, with one row for the blue alliance and another for the red alliance. The alliance column contains Red or Blue, depending on the alliance. Finally, match data frames have six teams per row and one row per match.
The functions GetSchedule(), GetHybridSchedule(), and GetMatchResults() no longer accept the expand_cols argument. These functions can only return a data frame in team shape. Users can convert between team, alliance, and match data frames with the functions ToAllianceShape(), ToMatchShape(), and ToTeamShape().
I've added a MergeReults() function which merges the HybridSchedule and Scores data frames into a single data frame, providing team numbers and detailed performance data all in one table.
I've added the function GetAll(), which downloads everything about a single competition and saves it in a single list.
I've added functions for saving FIRST competition data locally in R data files. The functions SaveData() and ReadData() display file open and save dialogs for saving and loading data to and from .RDS files.
To give users without an authorization key some data to play with, firstapiR now includes all performance data for the 2016 FIRST Robotics Championships. The data is available in R lists named archimedes2016, carver2016, curie2016, etc.
I've fixed a few inconsistencies. Some character columns were changed to factors, some column names were shortened to reduce excess typing, row names are more consistent, etc.

Getting Started - Creating a Session Object

Every HTTP request to the FIRST API requires a username, an authorization key, a four digit year that identifies the season of interest, and a header that specifies whether the data should be returned as XML or JSON. Having to type all of these parameters into every function call would make firstapiR very cumbersome to use, especially from the R console. The firstapiR package addresses this problem with the GetSession() function, which returns a Session object that contains all of the parameters listed above. All other firstapiR functions require a Session object as the first parameter. Here's an example:

# Create a Session object
sn <- firstapiR::GetSession("username", "key")

# Display the Session object.
sn

As you can see the Session object is really just an R list. It's also an instance of class "Session", which will be useful to users who want to do some type checking.

In addition to the username and key, the Session object specifies the season, data format ("data.frame", "xml", or "json") and whether the staging server should be used instead of the production server. Any of these parameters can be included as a named argument when calling GetSession(), or they can be modified later. For example:

# Create a Session and specify XML format
sn <- firstapiR::GetSession("username", "key", format = "xml")

# Change the season to 2015
sn$season <- 2015

# Display the Session object
sn

Using the Session Object

The following example shows how the session object is passed to a firstapiR function:

# Create a Session and use it in a firstapiR function
sn <- firstapiR::GetSession("username", "key")
dist <- firstapiR::GetDistricts(sn)

# Display the class of the object returned by GetDistricts()
class(dist)

# Display the data frame
dist

By default, GetSession creates a Session object with the current year, format = "data.frame", and staging = TRUE. GetDistricts() returns a data frame that is also an instance of class "Districts". GetDistricts() is useful because the three-letter district codes are used as arguments to several other firstapiR functions.

Note: If you run the example above, passing the string "key" to GetSession(), firstapiR skips the HTTP request and pulls data from a local cache. See Why These Examples Work: Local verses HTTP Data below for an explanation of this feature.

Extracting Data from a Data Frame

R provides numerous methods for extracting, filtering, and manipulating data in data frames. Refer to the R resources listed in this vignette for additional information. Individual data elements can be extracted from a data frame using the syntax data_frame$column_name[[row_number]]. For example:

# Retrieve the item in th 6th row of the column named "code"
dist$code[[6]]

Here's one of many slick techniques that are available in R:

# Retrieve the name of any district with code == "PNW"
dist$name[dist$code == "PNW"]

Next Steps

This package provides a detailed help file for every firstapiR function. Make sure the firstapiR package is loaded by running the command library(firstapiR). Then in RStudio, just type the name of the function in the help view search box. Or type help(function_name) at the R console command prompt. Here are the firstapiR functions:

| | | | | |:------------------|:------------------|:---------------|:-------------------| | GetSession() | GetServerStatus() | GetSeason() | GetDistricts() | | GetEvents() | GetTeams() | GetSchedule() | GetHybridSchedule()| | GetMatchResults() | GetScores() | GetAlliances() | GetRankings() | | GetAwards() | GetAwardsList() | GetAll() | MergeResults() | | ReadData() | SaveData() | ToTeamShape() | ToMatchShape() | | ToAllianceShape() |

Table: External Functions in firstapiR Package

In addition to help documents on each function, this package contains one other vignette: Overview of firstapiR Functions.

Advanced Topics

Why These Examples Work: Local verses HTTP Data

Hopefully you were able to figure out that "username" and "key" are not affiliated with a valid FIRST API account. These examples work because for all firstapiR functions, if the key value in the Session object is the literal value "key", firstapiR skips the HTTP requests and extracts cached JSON or XML data from the R/sysdata.rda file. If you insert any other random value into the key parameter, or the username does not match an actual FIRST API username, the FIRST API will return a 401 status code and the firstapiR function will throw an error.

I added the local data feature for several reasons.

Users can experiment with firstapiR functions and the data they provide without a FIRST API user account.
The examples in the help documents and vignettes will work exactly as typed.
Most of firstapiR's functionality can be tested without internet access or a FIRST API account.

Attributes of Returned firstapiR Objects

All of the firstapiR functions add attributes to whatever data they return. These attributes allow you to identify whether the data is local or was extracted directly from the FIRST API server, when the data was extracted from the FIRST API server, and the URL that was sent to the server. To demonstrate, we'll analyze the Districts data frame using R's str() function, which displays the structure of any object:

# Display the structure and attributes of the dist object
str(dist)

The attribute local_test_data is set to TRUE if the object was created from cached data in the R/sysdata.rda file. The local_url attribute contains the actual URL that was sent to the FIRST API server to request the data, and the time_downloaded attribute shows when the data was downloaded from the server.

For non-local, HTTP data, local_test_data will be set to FALSE, and local_url will be NULL. The time_downloaded attribute will contain the time that the HTTP request was received, based on the local system time.

There is also a url attribute. This attribute always contains the URL that is constructed from the arguments passed to the firstapiR function. Normally, this is the actual URL that is sent to the FIRST API server, but when extracting local data, this URL is mostly ignored. In the GetDistricts() example, local_url matches url because GetDistricts has no optional parameters and there is only one way to call this function. Most other firstapiR functions have many optional parameters and a multitude of possible return values. Since it's not practical to cache the entire FIRST API server, when requesting local data (by setting the key parameter to "key"), firstapiR ignores the optional parameters. The local data will be of the same type and format as what was requested. In these situations the url attribute will represent was was requested, and the local_url attribute will describe what was cached and returned.

Use the attr(object, attribute_name) function and syntax to retrive an attribute from an object.

Open the tools/data.R folder (only available in the firstapiR github repository) to see the firstapiR commands that were used to generate the local data.

Testing firstapiR on Your System

The firstapiR package includes several automated tests in the tests/testthat folder. Users can run these tests if they have the testthat package installed on their system. Run all of the tests by running testthat::test_dir(path_to_tests/testthat_folder) at the R console.

If you run the test_dir command, you'll probably notice that several tests are skipped. The tests are set up such that most of them can run without an internet connection or a FIRST API account. This is OK because local data requests use almost all of the firstapiR code that would be used for an actual HTTP request, including evaluation of the function arguments, constructing the URL and HTTP headers, and creating and formatting the data frame from JSON text. (If you specify the data frame format, firstapiR actually requests JSON text from the FIRST API server, because JSON is more compact than XML.) Only the actual HTTP request is skipped.

If you have a username and authorization key assigned by FIRST, then you can run all tests. Store your username and key in character vectors named "username" and "key". Then run test_dir from the R console. The tests will detect that the variables username and key exist and will run tests that make actual HTTP calls to the FIRST API server. If either your username or key are incorrect, the tests will fail due to an HTTP 401 error.