IPUMS Data and R
In ipumsr: An R Interface for Downloading, Reading, and Handling IPUMS Data

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
library(ipumsr)

This text provides an overview of how to find, request, download, and read IPUMS data into R. For a general introduction to IPUMS and ipumsr, see the ipumsr home page.

Obtaining IPUMS data

IPUMS data are free, but do require registration. New users can register with a particular IPUMS project by clicking the Register link at the top right of the project website.

Users obtain IPUMS data by creating and submitting an extract request. This specifies which data to include in the resulting extract (or data extract). IPUMS servers process each submitted extract request, and when complete, users can download the extract containing the requested data.

Extracts typically contain both data and metadata files. Data files typically come as fixed-width (.dat) files or comma-delimited (.csv) files. Metadata files contain information about the data file and its contents, including variable descriptions and parsing instructions for fixed-width data files. IPUMS microdata projects provide metadata in DDI (.xml) files. Aggregate data projects provide metadata in either .txt or .csv formats.

Users can submit extract requests and download extracts via either the IPUMS website or the IPUMS API, or via ipumsr functions that interface with the IPUMS API. The API currently supports access to the extract system only for certain IPUMS projects, which also determines the functionality that ipumsr can support.

Obtaining data via an IPUMS website

To create a new extract request via IPUMS websites, navigate to the extract interface for the IPUMS project of interest by clicking Select Data in the heading of the project website. The project extract interface allows you to explore what's available, find documentation about data concepts and sources, and then specify the data you'd like to download. The data selection parameters will differ across projects; see each project's documentation for more details on the available options. If you've never created an extract for the project you're interested in, a good way to learn the basics is to watch a project-specific video on creating extracts hosted on the IPUMS Tutorials page.

Downloading from microdata projects

Once your extract is ready, click the green Download button to download the data file. Then, right-click the DDI link in the Codebook column, and select Save Link As... (see below).

Note that some browsers may display different text, but there should be an option to download the DDI file as .xml. For instance, on Safari, select Download Linked File As.... For ipumsr to read the metadata, it is necessary to save the file in .xml format, not .html format.

{width="1000"}

Downloading from aggregate data projects

Aggregate data projects include data and metadata together in a single .zip archive file. To download them, simply click on the green Tables button (for tabular data) and/or GIS Files button (for spatial boundary or location data) in the Download Data column.

Obtaining data via the IPUMS API

Users can also create and submit extract requests within R by using ipumsr functions that interface with the IPUMS API. The IPUMS API currently supports access to the extract system for the following collections:

IPUMS USA
IPUMS CPS
IPUMS International
IPUMS NHGIS

The IPUMS API and ipumsr also support access to IPUMS NHGIS metadata, so users can query NHGIS metadata in R to explore what data are available and specify NHGIS data requests. At this time, creating requests for microdata generally requires using the corresponding project websites to find samples and variables of interest and obtain their identifiers for use in R extract definitions.

Once you have identified the data you would like to request, the workflow for requesting and downloading data via API is straightforward. First, define the parameters of your extract. The available extract definition options will differ by IPUMS data collection. See the microdata API request and NHGIS API request vignettes for more details on defining an extract. (The NHGIS vignette also discusses how to access NHGIS metadata.)

cps_extract_request <- define_extract_cps(
  description = "2018-2019 CPS Data",
  samples = c("cps2018_05s", "cps2019_05s"),
  variables = c("SEX", "AGE", "YEAR")
)

nhgis_extract_request <- define_extract_nhgis(
  description = "NHGIS Data via IPUMS API",
  datasets = ds_spec(
    "1990_STF1",
    data_tables = c("NP1", "NP2", "NP3"),
    geog_levels = "state"
  )
)

Next, submit your extract definition. After waiting for it to complete, you can download the files directly to your local machine without ever having to leave R:

submitted_extract <- submit_extract(extract_request)
downloadable_extract <- wait_for_extract(submitted_extract)
data_files <- download_extract(downloadable_extract)

You can also get the specifications of your previous extract requests, even if they weren't made with the API:

past_extracts <- get_extract_history("nhgis")

See the introduction to the IPUMS API for R users for more details about how to use ipumsr to interact with the IPUMS API.

Reading IPUMS data

Once you have downloaded an extract, you can load the data into R with the family of read_*() functions in ipumsr. These functions expand on those provided in readr in two ways:

ipumsr anticipates standard IPUMS file structures, limiting the need for users to manually extract and organize their downloaded files before reading.
ipumsr uses an extract's metadata files to automatically attach contextual information to the data. This allows users to easily identify variable names, variable descriptions, and labeled data values (from haven), which are common in IPUMS files.

For microdata files, use the read_ipums_micro_*() family:

cps_file <- ipums_example("cps_00157.xml")
cps_data <- read_ipums_micro(cps_file)

head(cps_data)

For NHGIS files, use read_nhgis():

nhgis_file <- ipums_example("nhgis0972_csv.zip")
nhgis_data <- read_nhgis(nhgis_file)

head(nhgis_data)

ipumsr also supports the reading of IPUMS shapefiles (spatial boundary and location files) into the sf format provided by the sf package:

shp_file <- ipums_example("nhgis0972_shape_small.zip")
nhgis_shp <- read_ipums_sf(shp_file)

head(nhgis_shp)

ipumsr is primarily designed to read data produced by the IPUMS extract system. However, IPUMS does distribute other files, often available via direct download. In many cases, these can be loaded with ipumsr. Otherwise, these files can likely be handled by existing data reading packages like readr (for delimited files) or haven (for Stata, SPSS, or SAS files).

See the vignette on reading IPUMS data for more information.

Exploring file metadata

Load a file's metadata with read_ipums_ddi() (for microdata projects) and read_nhgis_codebook() (for NHGIS). These provide file- and variable-level metadata for a given data source, which can be used to interpret the data contents.

cps_meta <- read_ipums_ddi(cps_file)
nhgis_meta <- read_nhgis_codebook(nhgis_file)

Summarize the variable metadata for a dataset using ipums_var_info():

ipums_var_info(cps_meta)

You can also get contextual details for specific variables:

ipums_var_desc(cps_data$INCTOT)

ipums_val_labels(cps_data$STATEFIP)

ipumsr also provides a family of lbl_*() functions to assist in accessing and manipulating the value-level metadata included in IPUMS data. This allows for value labels to be incorporated into the data processing pipeline. For instance:

# Remove labels for values that do not appear in the data
cps_data$STATEFIP <- lbl_clean(cps_data$STATEFIP)

ipums_val_labels(cps_data$STATEFIP)

# Combine North and South Dakota into a single value/label pair
cps_data$STATEFIP <- lbl_relabel(
  cps_data$STATEFIP,
  lbl("38_46", "Dakotas") ~ grepl("Dakota", .lbl)
)

ipums_val_labels(cps_data$STATEFIP)