In ge-flight-analytics/refoqa: A Tidyverse Friendly Wrapper for the GE EMS EFOQA API YALL

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%"
)

refoqa

refoqa is a tidyverse friendly R wrapper around the EMS/eFOQA API.

The primary expected workflow for this wrapper is to use the 'Data Sources' Demo App / Developer Tool in EMS Online to navigate the EMS Tree and find all of the fields you want to use.

Then you will use the query json that tool generates to get exactly the fields you wnat to query.

're'foqa because it is a bit of a 're'-write of the other EMS package (rems)

The benefits of this package over Rems are:
1. The functions (like 'filter' or 'select' in Rems) will not clash with dplyr verbs.
2. Credential management is easier.
3. Works with the 'Data Sources' App for interactive and precise field selection.

Installation

You can install the released version of refoqa from github with:

install.packages("devtools")
devtools::install_github("https://github.com/ge-flight-analytics/refoqa.git")

Setup

refoqa expects you to store your EMS/eFOQA credentials and preferred server inside if your .Renviron file. This means you don't have to ever pass your credentials to refoqa by hand! Set it and forget it. Plus this is considered best practice in R so that your credentials stay out of your script, history and git.

In Rstudio run:

usethis::edit_r_environ()

Add in these three lines and replace the example data with your efoqa user, password and server

# Example .Renviron file
EFOQAUSER=example.user
EFOQAPASSWORD=EXAMPLEPASSWORD
EFOQASERVER=https://d2mo-api.us.efoqa.com/api

Restart your R session, and you can test for success by doing:

Sys.getenv("EFOQASERVER")

Some Quick Results

refoqa has a couple of pre-built queries to get you started. To get some general information about the flights on your system try using standard_flight_query()

library(refoqa)
library(tidyverse)

all_flights <- standard_flight_query()

print(head(all_flights))

Specifying the 'Data Source Id' (aka entity or database)

The 'data_source_id' will need to be specified when you want to query for something other than FDW Flights (for example flight safety events). This is what controls what the 'rows' of your query are (flights, events, downloads, etc.)

Let's do an events query now. In the 'data source' app in EMS online, copy the appropriate 'Data Source id'

example_event_data <- standard_event_query(
  event_data_source_id = "[ems-apm][entity-type][events:profile-a7483c449db94a449eb5f67681ee52b0]")

print(head(example_event_data))

A Full Custom Query

Now let's do a full custom query. Run through the full 'Data Source' app selecting the data source and fields. In the end you will get a json query in the 'JSON Editor' tab.

Copy + Paste this json query to create a new json file in your R project. Then pass it to 'database_query_from_json' along with the data source id and it will get executed.

example_query_file <- system.file("extdata", "example_query.json", package = "refoqa")
example_analytics_query_file <- system.file("extdata", "analytic_query_example.json", package = "refoqa")

custom_query_results <- database_query_from_json(data_source_id = "[ems-core][entity-type][foqa-flights]",
                                                 json_file = example_query_file)
print(head(custom_query_results))

Notes

Field Formats

The return type of each field is based on your query. This means that in your json query,

"format": "display" 

Will return all the result as a strings.

"format": "none" 

Will return numbers.

You can set this globally at the base of your json for all fields or overwrite the value for any given field within the 'select' definition.

{
  "select": [
    {
      "fieldId": "[-hub-][field][[[ems-core][entity-type][foqa-flights]][[ems-core][base-field][flight.uid]]]",
      "aggregate": "none",
      "format": "none"
    }
  ],
  "format": "display"
}

'Top 10'

By default the Data Sources App will include a line limiting the results to just the first 10 records:

  "top": 10

Delete this line to get all results.

Alias for Fields

You can set an alias for any database field within the select statement. This lets you shorten the returned name for something very specific 'best available baro corrected altitude 1 at start of event' to something very short and convenient 'altitude'

{
  "select": [
    {
      "fieldId": "[-hub-][field][[[ems-core][entity-type][foqa-flights]][[ems-core][base-field][flight.uid]]]",
      "aggregate": "none",
      "alias": "flight_record
    }
  ]
}

Full Flight Data 'Analytics' Queries

This is super rough right now and may change, but you can query the 'analytics' API endpoint. See the documentation in the EMS Online REST API Explorer for details on the json form of the query.

library(ggplot2, quietly = TRUE)

example_parameter_results <- analytics_query_from_json(flight_id = 3135409, query_json_file = example_analytics_query_file )

ggplot(data = example_parameter_results, aes(x = offset, y = pressure_altitude_ft)) +
  geom_line()

You can run an analytics query on multiple flight records with:

query_as_r_list <- jsonlite::read_json(example_analytics_query_file)
multiple_results <- analytics_query_multiflight(flight_ids=c(3135409, 3135410), query_list = query_as_r_list)

multiple_results %>%
  dplyr::group_by(flight_id) %>%
  dplyr::summarise(max_alt = max(pressure_altitude_ft, na.rm = TRUE))

You can optionally specify start and end times too if you have them. You will have to pipe in the offsets from the database queries above. For example I have a stored query here that gets the timepoint Top of Descent to Touchdown for 10 flights. Note: The format = 'none' option is useful in these timepoint queries

example_timepoint_query_file <- system.file("extdata", "timepoint_query.json", package = "refoqa")

flight_dataframe <-  database_query_from_json(data_source_id = "[ems-core][entity-type][foqa-flights]",
                                              json_file = example_timepoint_query_file)
print(head(flight_dataframe))

And then we can do our same query but pass in these timepoints as the start and end.

query_as_r_list <- jsonlite::read_json(example_analytics_query_file)
tod_to_touchdown <- analytics_query_multiflight(flight_ids=flight_dataframe$flight_record,
                                                query_list = query_as_r_list,
                                                start_offsets = flight_dataframe$top_of_descent,
                                                end_offsets = flight_dataframe$touchdown)

ggplot(data = tod_to_touchdown, aes(x = ground_track_distance_to_touchdown_nm, y = pressure_altitude_ft, color=as.factor(flight_id))) +
  geom_point()

And an option to do this without specifying a flight record if you want to ( refoqa will select a random flight for you). Mostly useful for system maintenance type uses, but it is here if you want it.

example_parameter_results <- analytics_query_with_unspecified_flight( query_list = query_as_r_list )

ggplot(data = example_parameter_results, aes(x = offset, y = pressure_altitude_ft)) +
  geom_line()

Search Analytic Ids

search_for_analytic( "Slat Operating Speed Maximum" )

Helpers for Discretes or Dimensions

The 'data sources' app and json query are not great for handling discretes or dimensions. Here are a few helper functions to help out.

Get Possible Dimension/Discrete States

One way to work around this is to get the EMS ids for the possible discretes. Then you can type these in by hand into your json query. There are some build in id_table queries to get these lists.

airframe_id_table <- get_airframe_id_table()
print( head( airframe_id_table ) )

airframe_engine_id_table <- get_airframe_engine_id_table()
print( head( airframe_engine_id_table ) )

Or you can get a table of possible discrete states for any field you want.

fleet_table <- get_dimension_table(data_source_id = "[ems-core][entity-type][foqa-flights]",
                                   field_id = "[-hub-][field][[[ems-core][entity-type][foqa-flights]][[ems-core][base-field][flight.fleet]]]")
print(head(fleet_table))

Append Dimension/Discrete Filter by String

There is a helper to tack on a filter for a specific string version of a particular field to an input query.

example_query_list <- jsonlite::read_json(example_query_file)

flight_query_for_fleet_03 <- filter_dimension_by_string(
  data_source_id = "[ems-core][entity-type][foqa-flights]",
  query_list = example_query_list,
  field_id = "[-hub-][field][[[ems-core][entity-type][foqa-flights]][[ems-core][base-field][flight.fleet]]]",
  dimension_string = "Fleet 03")

fleet_03_flights <- database_query_from_list( data_source_id = "[ems-core][entity-type][foqa-flights]",
                                              query_list = flight_query_for_fleet_03)
print(head(fleet_03_flights))

And some helpers for specific dimensions

flight_query_for_737 <- filter_airframe_engine_by_string(
  query_list = example_query_list,
  target_airframe_engine_string = "737-800 CFM56-7")

flights_737 <- database_query_from_list( data_source_id = "[ems-core][entity-type][foqa-flights]",
                                              query_list = flight_query_for_737)
print(head(flights_737))

Other Potentially Useful Functions

List APM Profiles

You can get a list of all available profiles on the system with the list_all_apm_profiles() function:

all_profiles <- list_all_apm_profiles()

print( all_profiles[[1]]$name )
example_profile_id <- all_profiles[[1]]$id
print( example_profile_id )

APM Profile Glossary

You can get a glossary of an APM profile in either json/list or csv/dataframe form

library(dplyr, quietly = TRUE)

example_glossary <- apm_profile_glossary( profile_id = example_profile_id, glossary_format = "csv" )

example_glossary <- select(example_glossary, name, record_type, item_id, logical_id)
print(head(example_glossary))

You can also get a list of the events in a profile

example_profile_id <- "a7483c449db94a449eb5f67681ee52b0"

example_events_df <- apm_events_glossary( profile_id = example_profile_id )

print( head( example_events_df ) )

More Built In Queries

A built-in query for a summary of the airframe-engine types on the system

airframe_summary <- standard_airframe_query()

print( head( airframe_summary ) )

A built-in query for a summary of the fleets on the system

fleet_summary <- standard_fleet_query()

print( head( fleet_summary ) )

Misc

Get details on any EMS schema field

field_details <- get_database_field_details(
  data_source_id = "[ems-core][entity-type][foqa-flights]",
  field_id = "[-hub-][field][[[ems-core][entity-type][foqa-flights]][[airframe-engine-field-set][base-field][airframe-engine]]]")
print(names(field_details))

Get a dataframe of all physical parameters for a particular flight record

physical_parameters_for_flight <- get_physical_parameters_for_flight(3135409)

print( head( select( physical_parameters_for_flight, name, units, uid) ) )

Get a dataframe of all physical parameters for all PDCs/LFLs on the system

all_physical_parameters <- get_all_physical_parameters()

print( head( select( all_physical_parameters, name, flight_physical_data_configuration_id ) ) )

Get a dataframe of all logical items (parameters and constants) on the system

all_logical_items <- get_logical_FDW_item_list()

print( head( select( all_logical_items, name, description, item_type ) ) )

ge-flight-analytics/refoqa documentation built on Oct. 15, 2024, 7:15 a.m.