In leonawicz/rtrek: Data Analysis Relating to Star Trek

set.seed(1)
knitr::opts_chunk$set(
  collapse = TRUE, comment = "#>", message = FALSE, warning = FALSE, error = FALSE
)
library(rtrek)

The rtrek package includes some Star Trek datasets, but much more data is available outside the package. You can access other Star Trek data through various APIs.

Technically, there is only one formal API: the Star Trek API (STAPI). rtrek has functions to assist with making calls to this API in order to access specific data. See the STAPI vignette for details.

The focus of this vignette is on accessing data from Memory Alpha. rtrek interfaces with and extracts information from the Memory Alpha and Memory Beta websites. Neither of these sites actually expose an API, but functions in rtrek with querying these websites in an API-like manner. For working with Memory Beta content, see the respective vignette.

The rtrek Memory Alpha API

Memory Alpha is a website that hosts information on all things relating to official canon Star Trek. This strictly pertains to the television series and movies. There are many other officially licensed Star Trek productions, e.g., the many hundreds of novels, but these are not technically canon even though they are often treated as much by many fans. For a broader, licensed works focus, see Memory Beta.

When talking about using rtrek to access data from Memory Alpha, the term data is used loosely. It would be just as accurate to say information, content or text. While the site contains a vast amount of information, it is not structured in tidy tables like a data scientist would love to conveniently encounter. Memory Alpha is a wiki and can be thought of as similar to an encyclopedia. The bulk of its pages consist of articles. While some of these may have interesting html tables contained within, the site largely offers textual data.

Since Memory Alpha does not offer an API, the API-like interfacing provided by rtrek is just a collection of wrappers around web page scraping. In terms of what the relevant functions bring back from Memory Alpha, there are real limitations on the level of generality and quality of formatting that can be achieved across such a massive and diverse collection of articles.

Memory Alpha portals

There are six Memory Alpha web portals available. To see them, call the main function for Memory Alpha access, memory_alpha, and pass it portals as the API endpoint.

memory_alpha("portals")

The data frame returned provides each portal ID and respective "short URL". These relative URLs are given in order to reduce verbosity and redundancy. All absolute URLs begin with https://memory-alpha.fandom.com/wiki/.

In this special case where endpoint = "portals", this table is returned from the package itself because it is already known. The available portals are fixed. There is no accessing of Memory Alpha yet. The URLs shown are also not needed by the user, but are provided alongside the IDs for context.

Using a portal

When using a specific portal at the highest level (portal ID only), the returned data frame contains information about searchable categories available in the portal.

memory_alpha("people")

Again, there are id and url columns. There is also a group (and potentially a subgroup) column. This is only to provide meaningful context for the values in the id column if relevant for a given portal; group is not used for anything and the user can ignore it.

The above call does involve reaching out to Memory Alpha. While the portals are stable, it is expected that content within is regularly updated. Remember that this is not a real API. Since one is not available, what is really going on behind the scenes is the use of xml2 and rvest for web page harvesting.

Some portals have terminal endpoints - in Memory Alpha these are the written articles - at the top level, but typically the top level results for a portal are categories. You can always differentiate categories from articles by the URL, which will begin with Category: in the former case.

Descending through subcategories is done by appending their id values, separated by a forward slash /.

memory_alpha("people/Klingons")

memory_alpha("people/Klingons/Worf")

Note the change in the structure of the final output, which is an article. This is the end of this particular road The result is still a data frame, but now has only one row, the article.

The columns include a text title and three nested datasets. content contains an xml_nodeset object left (mostly) unadulterated by memory_alpha. This contains the article's main content section, including ordered content from a default set of html tags. For more control over article content, see ma_article in the next section. metadata contains a nested data frame of content parsed from the summary card that appears in the top right corner of articles. If this fails to parse for a given article, NULL is returned. categories returns a data frame containing categories in which the article topic falls under and their respective URLs.

Articles

If you already know the article id, You can obtain an article directly using ma_article instead of going through an endpoint with memory_alpha that terminates in the same id. This also offers additional options to control what tags are included in the returned result and whether that result is the original xml_nodeset object or a character vector of only the extracted text. In either case, work is left to the user to do what they intend such as text analysis.

worf <- ma_article("Worf", content_format = "character", content_nodes = c("h2", "h3"))
worf
worf$content[[1]] # Worf article section headings

If browse = TRUE the article page also launches in the browser.

Images

Full resolution source images can be downloaded and imported into R using ma_image if you know the short URL. The easiest way to find URLs is by using a Memory Alpha portal. In the example below, the Memory Alpha images category under Klingons is selected. Look for a picture that includes Worf but also Data.

library(dplyr)
klingons <- memory_alpha("people/Klingons/Memory Alpha images (Klingons)")
klingons

worf_data <- filter(klingons, grepl("Worf", url) & grepl("Data", url))
worf_data

Qapla'! This provides several results.

Technically, this is not the url to an image file. It is a url that redirects you to some other seemingly random article on the website that happens to include the image in it. This is not necessarily a unique instance of the image, nor is there any consistency in what portal or type of article it takes you to. memory_alpha, and ma_article using the short form url, provide the article content associated with the "file" url. See ma_image below for viewing the actual image.

x <- memory_alpha("people/Klingons/Memory Alpha images (Klingons)/Data tries talking to Worf.jpg")
x

x <- ma_article("File:Data_tries_talking_to_Worf.jpg")
x

x$categories

The likely intent is to obtain an image file after browsing the web pages that list images files. Even if you are interactively browsing the website, you have to click several times and scroll through additional articles before you can actually view the image file that was initially presented to you as a clickable link. This is a frustrating user experience and confusing design. If you have a file name you want to view, just use ma_image for this. It returns a ggplot object of the image file rather than an associated article.

ma_image("File:Data_tries_talking_to_Worf.jpg")

ma_image can take the additional arguments, keep = TRUE to retain the downloaded image file, and file to specify the output filename if you do not want it to be derived from the short URL. If you need more control over the plot, set keep = TRUE and then load the image file into R directly to plot separately as needed.

Search

You can perform a Memory Alpha site search using ma_search. This returns a data frame of search results content, including title, truncated text preview, and short URL for the first page of search results.

It does not recursively collate search results through subsequent pages of results. There could be an unexpectedly high number of pages of results depending on the search query. Since the general nature of this search feature seems relatively casual anyway, it aims only to provide a first page preview. As with ma_article, setting browse = TRUE opens the page in the browser.

ma_search("Guinan")

Caveats

Memory Alpha contains almost 50,000 pages at the time of this rtrek version. It is possible that some articles may have idiosyncratic structure that could make them inaccessible by these rtrek functions.

Since this package version is also the first to offer this brand new functionality - and as mentioned, Memory Alpha does not offer an API, leading to a less reliable web-scraping approach, it is unknown what the likelihood is at this time of breaking changes occurring during updates to Memory Alpha by its maintainers.

Jolan Tru.

leonawicz/rtrek documentation built on Sept. 18, 2023, 11:29 p.m.