PX-WEB API Interface for R

This R package provides tools to access PX-WEB API. Your contributions and bug reports and other feedback are welcome!

More information on the PX-Web/PC-Axis API can be found here.

Introduction

PXWEB is an API structure developed by Statistics Sweden together with other national statistical institutions (NSI) to disseminate public statistics in a structured way. This enables downloading and usage of data from statistical agencies without using a web browser direct over HTTP/HTTPS.

The pxweb R package connects any PXWEB API to R and hence facilitate the access, use and referencing of data from PXWEB APIs.

Available data sources and tools

A number of organizations use PXWEB to distribute hierarchical data. You can browse the available data sets at:

About PXWEB APIs

The data in PXWEB APIs consists of a metadata part and a data part. Metadata is structured in a hierarchical node tree, where each node contains information about subnodes that are below it in the tree or, if the nodes are at the bottom of the tree structure, the data referenced by the node as well as what dimensions are available for the data at that subnode.

Installation

To install the latest stable release version from CRAN, just use:

install.packages("pxweb")

To install the latest stable release version from GitHub, just use:

library("devtools")
devtools::install_github("ropengov/pxweb")

Test the installation by loading the library:

library(pxweb)

A tutorial is included with the package with:

vignette(topic="pxweb")

Installation issues

We also recommend setting the UTF-8 encoding since each individual API may have local specificl letters:

Sys.setlocale(locale="UTF-8") 

Accessing PXWEB from R

There are two ways of using the pxweb R package to access data, either interactively of using the core functions. To access data, two parts are needed, an URL to the data table in the API and a query specifying what data is of interest.

Interactive use

The simplest way of using pxweb is to use it interactively and navigate the API to the data of interest and then set up the data query of interest.

# Navigate through all pxweb api:s in the R package API catalogue
d <- pxweb_interactive()

# Get data from SCB (Statistics Sweden)
d <- pxweb_interactive("api.scb.se")

# Fetching data from statfi (Statistics Finland)
d <- pxweb_interactive("pxnet2.stat.fi")

# Fetching data from StatBank (Statistics Norway)
d <- pxweb_interactive("data.ssb.no")

# To see all available PXWEB APIs use
pxweb_apis <- pxweb_api_catalogue()

In the example above we use the interactive functionality from the PXWEB API root, but we could use any path to the API.

# Start with a specific path.
d <- pxweb_interactive("http://api.scb.se/OV0104/v1/doris/en/ssd/BE/BE0101/BE0101A")

This also means that we can navigate any PXWEB API, irrespectively of if they are a part of the R package API catalog or not. Just supply an URL to somewhere in the API and then navigate the API from there.

Due to new CRAN policies, it is not possible to use an R function to edit the api catalogue of the R package, but editing the can be done easily from R using file.edit().

file.edit(pxweb_api_catalogue_path())

Although, if the pxweb is installed again, it will overwrite the old api catalogue. So the easiest way is to do add a PXWEB API to the global catalogue. To do this, just do a pull request at the pxweb GitHub page here.

Direct use

Under the hood, the pxweb package uses the pxweb_get() function to access data from the PXWEB API. It also keeps track of the time limits of the API and split up to big queries into optimal downloadable chunks. If we use pxweb_get() without a query, the function either returns a PXWEB LEVELS object or a PXWEB METADATA object, depending if the URL points to a table in the API or not. Here is an example of a PXWEB LEVELS object.

# Get PXWEB levels
px_levels <- pxweb_get("http://api.scb.se/OV0104/v1/doris/en/ssd/BE/BE0101/BE0101A/")
px_levels

And if we use pxweb_get() for a table, a PXWEB METADATA object is returned.

# Get PXWEB metadata about a table
px_meta <- pxweb_get("http://api.scb.se/OV0104/v1/doris/en/ssd/BE/BE0101/BE0101A/BefolkningNy")
px_meta

Creating data queries

To download data we need both the URL to the table and a query specifying what parts of the table are of interest. An URL to a table is an URL that will return a metadata object if not a query is supplied. Creating a query can be done in three main ways. The first and simplest approach is to use pxweb_interactive() to explore the table URL and create a query interactively.

d <- pxweb_interactive("http://api.scb.se/OV0104/v1/doris/en/ssd/BE/BE0101/BE0101A/BefolkningNy")

The interactive function will return the query and the url, even if the data is not downloaded.

# save(d, file = "d_example.rda")
load("d_example.rda")
d$url
d$query

We can also turn the query to a json query that can be used outside R.

pxweb_query_as_json(d$query, pretty = TRUE)

The second approach is to specify the query either as an R list or a JSON object. Some Statistical Agencies, such as Statistics Sweden, supply queries directly as a JSON object on their web pages. These queries can be used directly. Below is another example of a JSON query for the table above. For details on how to set up a JSON query, see the PXWEB API documentation.

{
  "query": [
    {
      "code": "Civilstand",
      "selection": {
        "filter": "item",
        "values": ["OG", "G", "ÄNKL", "SK"]
      }
    },
    {
      "code": "Kon",
      "selection": {
        "filter": "item",
        "values": ["1", "2"]
      }
    },
    {
      "code": "ContentsCode",
      "selection": {
        "filter": "item",
        "values": ["BE0101N1"]
      }
    },
    {
      "code": "Tid",
      "selection": {
        "filter": "item",
        "values": ["2015", "2016", "2017"]
      }
    }
  ],
  "response": {
    "format": "json"
  }
} 

To use this JSON query we just store the JSON query as a file and supply the path to the file to the pxweb_query() function.

pxq <- pxweb_query("path/to/the/json/query.json")

Finally, we can create a PXWEB query from an R list where each list element is a variable and selected observation.

pxweb_query_list <- 
  list("Civilstand"=c("*"), # Use "*" to select all
       "Kon"=c("1","2"),
       "ContentsCode"=c("BE0101N1"),
       "Tid"=c("2015","2016","2017"))
pxq <- pxweb_query(pxweb_query_list)
pxq

The query can be validated against the metadata object to asses that the query can be used. This is done automatically when the data is fetched with pxweb_get(), but can also be done manually.

pxweb_validate_query_with_metadata(pxq, px_meta)

Downloading data

When we have the URL to a data table and a query we can simply download the data with pxweb_get(). The function returns a pxweb_data object that contains the downloaded data.

pxd <- pxweb_get("http://api.scb.se/OV0104/v1/doris/en/ssd/BE/BE0101/BE0101A/BefolkningNy",
                 pxq)
pxd

If we instead want a JSON-stat object, we just change the response format to JSON-stat and we will get a JSON-stat object returned. Only JSON and JSON-stat formats are implemented in the PXWEB API.

pxq$response$format <- "json-stat"
pxjstat <- pxweb_get("http://api.scb.se/OV0104/v1/doris/en/ssd/BE/BE0101/BE0101A/BefolkningNy",
                     pxq)
pxjstat

If the queries are large (contain more values than the PXWEB API maximum allowed values), the query is chunked into optimal chunks and is then downloaded sequentially. PXWEB data objects are then combined to one large PXWEB data object, while JSON-stat objects are returned as a list of JSON-stat objects.

For more advanced connections to the API, the pxweb_advanced_get() gives the flexibility to access the underlying HTTP calls using httr as well as logging the HTTP calls for debugging.

The downloaded PXWEB data objects can then be converted to either data.frames or to a character matrix. The character matrix contains the "raw" data while the data.frame returns a data.frame for analysis in a tidy format. This means that missing values (such as ".." are converted to NA) in a data.frame. Using the arguments variable.value.type and column.name.type we can also choose if we want the code or the text column names and value types.

pxdf <- as.data.frame(pxd, column.name.type = "text", variable.value.type = "text")
head(pxdf)
pxdf <- as.data.frame(pxd, column.name.type = "code", variable.value.type = "code")
head(pxdf)

In a similar way, we can access the raw data as a character matrix with as.matrix.

pxmat <- as.matrix(pxd, column.name.type = "code", variable.value.type = "code")
head(pxmat)

Access data footnotes/comments

In addition to the data, the PXWEB DATA object may also contain comments for the data. This can be accessed using pxweb_data_comments() function.

pxdc <- pxweb_data_comments(pxd)
pxdc

In this case, we did not have any comments. If we have comments we can turn the comments into a data.frame with one comment per row.

as.data.frame(pxdc)

Citation

Finally, if we use the data, we can easily create a citation for a pxweb_data object using the pxweb_cite() function. For full reproducibility, please also cite the package.

pxweb_cite(pxd)

Known issues and troubleshooting

See TROUBLESHOOTING.md for a list of current known issues.

Licensing

This work can be freely used, modified and distributed under the open license specified in the DESCRIPTION file.

Session info

This vignette was created with

sessionInfo()


Try the pxweb package in your browser

Any scripts or data that you put into this service are public.

pxweb documentation built on Oct. 10, 2021, 1:06 a.m.