In crew102/patentsview: An R Client to the PatentsView API

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)

A basic example

Let's start with a basic example of how to use the package's primary function, search_pv():

library(patentsview)

search_pv(
  query = '{"_gte":{"patent_date":"2007-01-01"}}',
  endpoint = "patents"
)

This call to search_pv() sends our query to the "patents" API endpoint. The API has 7 different endpoints, corresponding to 7 different entity types (assignees, CPC subsections, inventors, locations, NBER subcategories, patents, and USPC main classes).[^1] We filtered our results using the API's query language which uses JSON, though we could have used pure R functions instead. The next section briefly touches on those functions, while the writing queries vignette goes into much more depth.

Writing queries

The PatentsView query syntax is documented on their query language page.[^2] However, it can be difficult to get your query right if you're writing it by hand (i.e., just writing the query in a string like '{"_gte":{"patent_date":"2007-01-01"}}'). The patentsview package comes with a simple domain specific language (DSL) to make writing queries a breeze. I recommend using the functions in this DSL for all but the most basic queries, especially if you're encountering errors and don't understand why...Let's rewrite the query from the basic example using one of those functions qry_funs$gte():

qry_funs$gte(patent_date = "2007-01-01")

More complex queries are also possible:

with_qfuns(
  and(
    gte(patent_date = "2007-01-01"),
    text_phrase(patent_abstract = c("computer program", "dog leash"))
  )
)

Check out the writing queries vignette for more details on using the DSL.

Fields

Each endpoint has a different set of queryable and retrievable fields. Queryable fields are those that you can include in your query (e.g., patent_date shown in the first example). Retrievable fields are those that you can get data on. In the first example, we didn't specify which fields we wanted to retrieve so we were given the default set. You can specify which fields you want using the fields argument:

search_pv(
  query = '{"_gte":{"patent_date":"2007-01-01"}}',
  endpoint = "patents", 
  fields = c("patent_number", "patent_title")
)

To list all of the retrievable fields for a given endpoint, use get_fields():

retrvble_flds <- get_fields(endpoint = "patents")
head(retrvble_flds)

You can see an endpoint's list of queryable and retrievable fields by viewing the endpoint's online field list (e.g., the inventor field list table). Note the "Query" column in this table, which indicates whether the field is both queryable and retrievable (Query = Y), or just retrievable (Query = N). The field tables for all of the endpoints can be found in the fieldsdf data frame, which you can load using data("fieldsdf") or View(patentsview::fieldsdf).

Paginated responses

By default, search_pv() returns 25 records per page and only gives you the first page of results. I suggest using these defaults while you're figuring out the details of your request, such as the query syntax you want to use and the fields you want returned. Once you have those items finalized, you can use the per_page argument to download up to 10,000 records per page. You can also choose which page of results you want with the page argument:

search_pv(
  query = qry_funs$eq(inventor_last_name = "chambers"),
  page = 2, per_page = 150 # gets records 150 - 300
) 

You can download all pages of output in one call by setting all_pages = TRUE. This will set per_page equal to 10,000 and loop over all pages of output (downloading up to 10 pages, or 100,000 records total):

search_pv(
  query = qry_funs$eq(inventor_last_name = "chambers"),
  all_pages = TRUE
)

Entity counts

Our last two calls to search_pv() gave the same value for total_patent_count, even though we got a lot more data from the second call. This is because the entity counts returned by the API refer to the number of distinct entities across all downloadable pages of output, not just the page that was returned. Downloadable pages of output is an important phrase here, as the API limits us to 100,000 records per query. For example, we got total_patent_count = 100,000 when we searched for patents published on or after 2007, even though there are way more than 100,000 patents that have been published since 2007. See the FAQs below for details on how to overcome the 100,000 record restriction.

By default, PatentsView uses disambiguted versions of assignees, inventors, and locations, instead of raw data. For example, let's say you search for all inventors whose first name is "john." The PatentsView API is going to return all of the inventors who have a preferred first name of john (as per the disambiguation results), which may not necessarily match their raw first name. You could be getting back inventors whose first name is, say, "jonathan," "johnn," or even "john jay." You can search on the raw inventor names instead of the preferred names by using the fields starting with "raw" in your query (e.g., rawinventor_first_name). Note that the assignee and location raw data fields are not currently being offered by the API. To see the methods behind the disambiguation process, see the PatentsView Inventor Disambiguation Technical Workshop website

7 endpoints for 7 entities

We can get similar data from the 7 endpoints. For example, the following two calls differ only in the endpoint that is chosen:

# Here we are using the patents endpoint
search_pv(
  query = qry_funs$eq(inventor_last_name = "chambers"), 
  endpoint = "patents", 
  fields = c("patent_number", "inventor_last_name", "assignee_organization")
)

# While here we are using the assignees endpoint
search_pv(
  query = qry_funs$eq(inventor_last_name = "chambers"), 
  endpoint = "assignees", 
  fields = c("patent_number", "inventor_last_name", "assignee_organization")
)

Your choice of endpoint determines two things:

1. Which entity your query is applied to. The first call shown above used the patents endpoint, so the API searched for patents that have at least one inventor on them with the last name of "chambers." The second call used the assignees endpoint, so the API searched for assignees that have at least one patent which has an inventor on it whose last name is "chambers."

2. The structure of the data frame that is returned. The first call returned a data frame on the patent level, meaning that each row corresponded to a different patent. Fields that were not on the patent level (e.g., inventor_last_name) were returned in list columns that are named after the subentity associated with the field (e.g., the inventors subentity).[^3] Meanwhile, the second call gave us a data frame on the assignee level (one row for each assignee) because it used the assignees endpoint.

Most of the time you will want to use the patents endpoint. Note that you can still effectively filter on fields that are not at the patent-level when using the patents endpoint (e.g., filter on assignee name or CPC category), due to the fact that patents are relatively low-level entities. For higher level entities like assignees, if you filter on a field that is not at the assignee-level (e.g., inventor name), the API will return data on any assignee that has at least one inventor whose name matches your search.

Casting fields

The API always returns the data fields as strings, even if they would be better stored using a different data type (e.g., numeric). You can cast all of the fields to their preferred type using cast_pv_data():

res <- search_pv(
  query = "{\"patent_number\":\"5116621\"}", 
  fields = c("patent_date", "patent_title", "patent_year")
)

# Right now all of the fields are stored as characters:
res

# Use more appropriate data types:
cast_pv_data(data = res$data)

FAQs

I'm sure my query is well formatted and correct but I keep getting an error. What's the deal?

The API query syntax guidelines do not cover all of the API's behavior. Specifically, there are several things that you cannot do which are not documented on the API's webpage. The writing queries vignette has more details on this.

Does the API have any rate limiting/throttling controls?

Not at the moment.

How do I download more than 100,000 records?

Your best bet is to split your query into pieces based on dates, then concatenate the results together. For example, the below query would return more than 100,000 records for the patents endpoint:

query <- with_qfuns(
  text_any(patent_abstract = 'tool animal')
)

To download all of the records associated with this query, we could split it into two pieces and make two calls to search_pv():

query_1a <- with_qfuns(
  and(
    text_any(patent_abstract = 'tool animal'),
    lte(patent_date = "2010-01-01")
  )
)

query_1b <- with_qfuns(
  and(
    text_any(patent_abstract = 'tool animal'),
    gt(patent_date = "2010-01-01")
  )
)

How do I access the data frames inside of the subentity list columns?

Let's consider the following data, in which assignees are the primary entity while applications and "government interest statements" are the subentities:

# Create field list
asgn_flds <- c("assignee_id", "assignee_organization")
subent_flds <- get_fields("assignees", c("applications", "gov_interests"))
fields <- c(asgn_flds, subent_flds)

# Pull data
res <- search_pv(
  query = qry_funs$contains(inventor_last_name = "smith"), 
  endpoint = "assignees", 
  fields = fields
)
res$data

res$data has assignee-level columns that are vectors (e.g., res$data$assignees$assignee_id) and subentity-level columns that are lists (e.g., res$data$assignees$applications). You have two good ways to get at data frames nested inside the subentity lists:

1. Use tidyr::unnest. (This is probably the easier choice of the two).

library(tidyr)

# Get assignee/application data:
res$data$assignees %>% 
  unnest(applications) %>%
  head()

# Get assignee/gov_interest data:
res$data$assignees %>% 
  unnest(gov_interests) %>%
  head()

1. Use patentsview::unnest_pv_data. unnest_pv_data() creates a series of data frames (one for each entity/subentity level) that are like tables in a relational database. You provide it with the data that search_pv() returned to you, as well as a field that can act as a unique identifier for the primary entities:

unnest_pv_data(data = res$data, pk = "assignee_id")

Now we are left with a series of flat data frames instead of having a single data frame with other data frames nested inside of it. These flat data frames can be joined together as needed via the primary key (assignee_id).

[^1]: You can use get_endpoints() to list the endpoint names as the API expects them to appear (e.g., assignees, cpc_subsections, inventors, locations, nber_subcategories, patents, and uspc_mainclasses). [^2]: This webpage includes some details that are not relevant to the query argument in search_pv, such as the field list and sort parameter. [^3]: You can unnest the data frames that are stored in the list columns using unnest_pv_data(). See the FAQs for details.

crew102/patentsview documentation built on May 14, 2019, 11:33 a.m.