In antaldaniel/retroharmonize: Ex Post Survey Data Harmonization

## https://github.com/tidyverse/rvest/blob/master/vignettes/selectorgadget.Rmd
requireNamespace("png", quietly = TRUE)
embed_png <- function(path, dpi = NULL) {
  meta <- attr(png::readPNG(path, native = TRUE, info = TRUE), "info")
  if (!is.null(dpi)) meta$dpi <- rep(dpi, 2)
  knitr::asis_output(paste0(
    "<img src='", path, "'",
    " width=", round(meta$dim[1] / (meta$dpi[1] / 96)),
    " height=", round(meta$dim[2] / (meta$dpi[2] / 96)),
    " />"
  ))
}

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
library(retroharmonize)
library(dplyr)
library(tidyr)
load(file = system.file(
  "afrob", "afrob_vignette.rda", 
  package = "retroharmonize"))

The goal of this case study is to explore the variation in trust in various state institutions among African societies, as well as changes in trust over time.

To do this, we use data from Afrobarometer, a cross-national survey project measuring attitudes and opinions about democracy, the economy and society in African countries based on general population samples. Currently, seven rounds of the Afrobarometer are available, covering the period between 1999 and 2019.

retroharmonize is not affiliated with Afrobarometer. To fully reproduce this example, you must acquire the data files from them, which is free of charge. Afrobarometer data is protected by copyright. Authors of any published work based on Afrobarometer data or papers are required to acknowledge the source, including, where applicable, citations to data sets posted on this website. Please acknowledge the copyright holders in all publications resulting from its use by means of bibliographic citation in this form:

In this vignette, we harmonize data from rounds Afrobarometer Data Round 5 (all countries), Afrobarometer Data Round 6 (all countries) and Afrobarometer Data Round 7 (all countries). Some elements of the vignette are not “live”, because we want to avoid re-publishing the original microdata files from Afrobarometer, but you can access the data directly from the www.afrobarometer.org website.

File names (to identify file versions):
Round 5: merged-round-5-data-34-countries-2011-2013-last-update-july-2015.sav
Round 6: merged_r6_data_2016_36countries2.sav
Round 7: r7_merged_data_34ctry.release.sav

For reproducibility, we are storing only a small subsample from the files and the metadata. We assume that you extracted and copied all .sav files into a single folder that we will call in this vignette the afrobarometer_dir. Define your afrobarometer_dir with file.path() in your own system.

Importing Afrobarometer Files

We start by reading in the three rounds of the Afrobarometer.

library(retroharmonize)
library(dplyr)
library(tidyr)

### use here your own directory
ab <- dir ( afrobarometer_dir, pattern = "sav$" )
afrobarometer_rounds <- file.path(afrobarometer_dir, ab)

ab_waves <- read_surveys(afrobarometer_rounds, .f='read_spss')

Let's give a bit more meaningful identifiers than the file names:

attr(ab_waves[[1]], "id") <- "Afrobarometer_R5"
attr(ab_waves[[2]], "id") <- "Afrobarometer_R6"
attr(ab_waves[[3]], "id") <- "Afrobarometer_R7"

We can review if the main descriptive metadata is correctly present with document_surveys().

documented_ab_waves <- document_surveys(ab_waves)

print(documented_ab_waves)

Create a metadata table, or a data map, that contains information about the variable names and labels, and where each row correspnds to one variable in the survey data file. We do this with the function metadata_survey_create(), which extracts the metadata from the survery data files, normalizes variable labels, identifies ranges of substantive responses and missing value codes. Its wrapper, metadata_create(), loops the function on a list of surveys, or on a set of files containing surveys.

ab_metadata <- metadata_create ( survey_list = ab_waves )

Working with metadata

From the metadata file we select only those rows that correspond to the variables that we're interested in: the rowid being the unique case identifier, DATEINTR with the interview date, COUNTRY containing information about the country where the interview was conducted, REGION with the region (sub-national unit) where the interview was conducted, and withinwt being the weighting factor.

Further, we select all variables that have the word "trust" in the normalized variable label.

For these variables, we create normalized variable names (var_name) and labels (var_label).

library(dplyr)
to_harmonize <- ab_metadata %>%
  filter ( var_name_orig %in% 
             c("rowid", "DATEINTR", "COUNTRY", "REGION", "withinwt") |
             grepl("trust ", label_orig ) ) %>%
  mutate ( var_label = var_label_normalize(label_orig)) %>%
  mutate ( var_label = case_when ( 
    grepl("^unique identifier", var_label) ~ "unique_id", 
    TRUE ~ var_label)) %>%
  mutate ( var_name = val_label_normalize(var_label))

To further reduce the size of the analysis, we only choose a few variables based on the normalized variable names (var_name).

to_harmonize <- to_harmonize %>%
  filter ( 
    grepl ( "president|parliament|religious|traditional|unique_id|weight|country|date_of_int", var_name)
    )

The resulting table with information about the variables selected for harmonization looks as follows:

head(to_harmonize %>%
       select ( all_of(c("id", "var_name", "var_label"))), 10)

The merge_surveys() function harmonizes the variable names, the variable labels and survey identifiers and returns a list of surveys (of class survey().) The parameter var_harmonization must be a list or a data frame that contains at least the original file name (filename), original variable names (var_name_orig), the new variable names (var_name) and their labels (var_label), so that the program knows which variables to take from what files and how to call and label them after transformation.

merged_ab <- merge_surveys ( survey_list = ab_waves, 
                             var_harmonization = to_harmonize )

# country will be a character variable, and doesn't need a label
merged_ab <- lapply ( merged_ab, 
         FUN = function(x) x  %>%
           mutate( country = as_character(country)))

Review the most important metadata with document_surveys():

documenteded_merged_ab <- document_surveys(merged_ab)

print(documenteded_merged_ab)

Harmonization

In the Afrobarometer trust is measured with four-point ordinal rating scales. Such data are best analyzed with ordinal models, which do not assume that the points are equidistant. However, to get a quick idea of how the data look like, we will assign numbers 0-3, with 0 corresponding to the least trust, and 3 corresponding to the most trust, and for the time-being analyze the data as if they were metric.

To review the harmonization on a single survey use pull_survey(). Here we select Afrobarometer Round 6.

R6 <- pull_survey ( merged_ab, id = "Afrobarometer_R6" )

attributes(R6$trust_president[1:20])

The document_survey_item() function shows the metadata of a single variable.

document_survey_item(R6$trust_president)

Afrobarometer's SPSS files do not mark the missing values, so we have to be careful. The set of labels that are identified as missing:

collect_na_labels( to_harmonize )

The set of valid category labels and missing value labels are as follows:

collect_val_labels (to_harmonize %>%
                      filter ( grepl( "trust", var_name) ))

We create a harmonization function from the harmonize_values() prototype function. In fact, this is just a re-setting the default values of the original function. It makes future reference in pipelines easier, or it can be used for a question block only, in this case to variables with starts_with("trust").

harmonize_ab_trust <- function(x) {
  label_list <- list(
    from = c("^not", "^just", "^somewhat",
             "^a", "^don", "^ref", "^miss", "^not", "^inap"), 
    to = c("not_at_all", "little", "somewhat", 
           "a_lot", "do_not_know", "declined", "inap", "inap", 
           "inap"), 
    numeric_values = c(0,1,2,3, 99997, 99998, 99999,99999, 99999)
  )

  harmonize_values(
    x, 
    harmonize_labels = label_list, 
    na_values = c("do_not_know"=99997,
                  "declined"=99998,
                  "inap"=99999)
  )
}

We apply this function to the trust variables. The harmonize_surveys() function binds all variables that are present in all surveys.

harmonized_ab_waves <- harmonize_surveys ( 
  survey_list = merged_ab, 
  .f = harmonize_ab_trust )

Let's look at the attributes of harmonized_ab_waves.

h_ab_structure <- attributes(harmonized_ab_waves)

h_ab_structure$row.names <- NULL # We have over 100K row names
h_ab_structure

Let's add the year of the interview:

harmonized_ab_waves <- harmonized_ab_waves %>%
  mutate ( year = as.integer(substr(as.character(
    date_of_interview),1,4)))

To keep our example manageable, we subset the datasets to include only five countries.

harmonized_ab_waves <- harmonized_ab_waves %>%
  filter ( country %in% c("Niger", "Nigeria", "Algeria", 
                          "South Africa", "Madagascar"))

Analyzing the harmonized data

The harmonized data can be exported and analyzed in another statistical program. The labelled survey data is stored in labelled_spss_survey() vectors, which is a complex class that retains metadata for reproducibility. Most statistical R packages do not know it. The data should be presented either as numeric data with as_numeric() or as categorical with as_factor(). (See more why you should not fall back on the more generic as.factor() or as.numeric() methods in The labelled_spss_survey class vignette.)

Please note that the numeric form of these trust variables is not directly comparable with the numeric averages of the Eurobarometer trust variables, because the middle of the range is at r mean(0:3) and not r mean(0:1).

harmonized_ab_waves %>%
  mutate_at ( vars(starts_with("trust")), 
              ~as_numeric(.)*within_country_weighting_factor) %>%
  select ( -all_of("within_country_weighting_factor") ) %>%
  group_by ( country, year ) %>%
  summarize_if ( is.numeric, mean, na.rm=TRUE )

And the factor representation, without weighting:

library(tidyr)  ## tidyr::pivot_longer()
harmonized_ab_waves %>%
  select ( -all_of("within_country_weighting_factor") ) %>%
  mutate_if ( is.labelled_spss_survey, as_factor) %>%
  pivot_longer ( starts_with("trust"), 
                        names_to  = "institution", 
                        values_to = "category") %>%
  mutate ( institution = gsub("^trust_", "", institution) ) %>%
  group_by ( country, year, institution, category ) %>%
  summarize ( n = n() )

We subsetted the datasets to meet the vignette size limitations. If you are following our code without reducing the number of countries, you get the following results: