In Big-Life-Lab/cchsflow: Transforming and Harmonizing CCHS Variables

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)

Introduction

The variable_details.csv worksheet contain details for the variables in variables.csv. Information from variable_details.csv worksheet is used by the rec_with_table() function to transform variables identified in variable_details$variableStart to the newly transformed variable in variable_details$variable.

library(readr)
library(DT)
library(knitr)
library(kableExtra)
library(cchsflow)
datatable(variable_details, options = list(pageLength = 5))
cat("In the `variable_details.csv` worksheet there are", nrow(variable_details), "rows and", ncol(variable_details), "columns", "\n\n")

Structure of variable_details.csv

Rows

Each row in variable_details.csv holds the recode rules for transforming a single category for a variable in variables.csv. An exception to this rule are the "don't know", "refusal", and "not stated" categories, which are combined as a single missing category. For each unique variable, an else row is used to assign values not identified in other rows and are outside identified ranges. We recommend not combining variables across the CCHS if variable has an important change between CCHS cycles variable_details$notes is used to identify issues that may be relevant when transforming the variable or category.

If a categorical variable has 4 distinct categories, along with a "not applicable" category and the 3 missing categories, there will be 7 rows:

4 for each distinct category
1 for the not applicable category
1 for the missing categories
1 else row.

Naming convention for not applicable and missing values

rec_with_table() uses the tagged_na() function from the haven package to tag not applicable responses as NA(a), and missing values (don't know, refusal, not stated) as NA(b). As you will see later, not applicable values are transformed to NA::a, and missing values are transformed to NA::b. See here for more information about tagged_na().

Columns

The following are the columns that are listed in variable_details.csv. Many of these columns need to be specified in order for rec_with_table() to be functional. We will use the sex variable to illustrate how each column is specified:

variable: the name of the final transformed variable. In variable_details.csv, we have designated the variable names used in CCHS cycles from 2007 to 2018 as the final transformed variable name.

kable(variable_details[variable_details$variable == "DHH_SEX", 1], col.names = 'variable')

dummyVariable: the dummy variable for each category in a transformed categorical variable. This is only applicable for categorical variables; for continuous variables it is set as N/A. The name of a dummy variable consists of the final variable name, the number of categories in the variable, and the category level for each category. Note that this column is not necessary for rec_with_table().

kable(variable_details[variable_details$variable == "DHH_SEX", c(1:2)])

typeEnd: the variable type of the final transformed variable. In this column, a transformed variable that is categorical will be specified as cat; while a transformed variable that is continuous will be specified as cont.

kable(variable_details[variable_details$variable == "DHH_SEX", c(1:3)])

databaseStart: the CCHS surveys that contain the variable of interest, separated by commas. If the PUMF dataset from the CCHS is used, the survey names will have a _p suffix. If the Shared dataset from the CCHS is used, the survey names will have a _s suffix.

kable(variable_details[variable_details$variable == "DHH_SEX", c(1:4)])

variableStart: the original names of the variables as they are listed in each respective CCHS cycle, separated by commas. If the variable name in a particular CCHS survey is different from the transformed variable name, write out the CCHS survey identifier, add two colons, and write out the original variable name for that cycle. If the variable name in a particular CCHS survey is the same as the transformed variable name, the variable name is written out surrounded by square brackets. Note: this only needs to be written out once.

kable(variable_details[variable_details$variable == "DHH_SEX", c(1:5)])

The categorical sex variable in the 2001 CCHS survey is DHHA_SEX. If the final variable name for categorical age in the variable column is DHH_SEX, you would write the following in this column: cchs2001::DHH_SEX
The categorical age variable in the CCHS surveys from 2007 to 2018 is DHH_SEX. Since it is the same as the final variable name, you would write in this column [DHH_SEX] once. The variable name that is denoted within the square brackets is the default variable name.
typeStart: the variable type as indicated in the CCHS surveys. As indicated in the toType column, categorical variables are denoted as cat and continuous variables are denoted as cont.

kable(variable_details[variable_details$variable == "DHH_SEX", c(1:6)])

recEnd: the value you would like to recode each category value to. For continuous variables that are not transformed in type, you would write in this column copy so that the function copies the values without any transformations. For the not applicable category, write NA::a. For missing & else categories, write NA::b

kable(variable_details[variable_details$variable == "DHH_SEX", c(1:7)])

For categorical variables that are not changing variable types (i.e. cat to cat), it is ideal to retain the same values as indicated in each CCHS survey. But for transformed categorical variables that have changed in type (i.e cat to cont), you will have to develop values that make the most sense to your analysis. In variable_details.csv, variables that have gone from cat to cont have used midpoints of each category.
numValidCat: the number of categories for a variable. This only applies to variables in which the toType is cat. For continuous variables, numValidCat = N/A. Not applicable, missing, and else categories are not included in the category count. Note that this column is not necessary for rec_with_table().

kable(variable_details[variable_details$variable == "DHH_SEX", c(1:8)])

catLabel: short form label describing the category of a particular variable.

kable(variable_details[variable_details$variable == "DHH_SEX", c(1:9)])

catLabelLong: more detailed label describing the category of a particular variable. This label should be identical to what is shown in the CCHS data documentation, unless you are creating derived variables and would like to create your own label for it.

kable(variable_details[variable_details$variable == "DHH_SEX", c(1:10)])

units: the units of a particular variable. If there are no units for the variable, write N/A. Note, the function will not work if there different units between the rows of the same variable (i.e. height using both m and ft).

kable(variable_details[variable_details$variable == "DHH_SEX", c(1:11)])

recStart: the range of values for a particular category in a variable as indicated in the CCHS. See CCHS data documentation for each survey cycle and use the smallest and large values as your range to capture all values between the survey years.

The rules for each category of a new variable are a string in recFrom and value in recTo. These recode pairs are the same syntax as interval notation in which a closed range of values are specified using square brackets. See here for more information on interval notation. Recode pairs are obtained from the RecFrom and RecTo columns value range is indicated by a comma, e.g. recFrom= [1,4]; recTo = 1 (recodes all values from 1 to 4 into 1} value range for double vectors (with fractional part), all values within the specified range are recoded; e.g. recFrom = [1,2.5]; recTo = 1 recodes 1 to 2.5 into 1, but 2.55 would not be recoded (since it's not included in the specified range). NA is used for missing values (don't know, refusal, not stated) else is used all other values, which have not been specified yet, are indicated by else, e.g. recFrom = "else"; recTo = NA (recode all other values (not specified in other rows) to "NA")} copy the else token can be combined with copy, indicating that all remaining, not yet recoded values should stay the same (are copied from the original value), e.g. recFrom = "else"; recTo = "copy"

kable(variable_details[variable_details$variable == "DHH_SEX", c(1:12)])

catStartLabel: label describing each category. This label should be identical to what is shown in the CCHS data documentation. For the missing row, each missing category is described along with their coded values. You can import labels from the CCHS DDI files using bllflow helper functions. See bllflow documentation.

kable(variable_details[variable_details$variable == "DHH_SEX", c(1:13)])

variableStartShortLabel: short form label describing the variable.

kable(variable_details[variable_details$variable == "DHH_SEX", c(1:14)])

variableStartLabel: more detailed label describing the variable. This label should be identical to what is shown in the CCHS data documentation.

kable(variable_details[variable_details$variable == "DHH_SEX", c(1:15)])

notes: any relevant notes to inform the user running the recode-with-table function. Things to include here would be changes in wording between CCHS surveys, missing/changes in categories, and changes in variable type between CCHS surveys.

kable(variable_details[variable_details$variable == "DHH_SEX", c(1:16)])

Derived Variables

The same naming convention applies to derived variables with the exception of two columns:

In variableStart, instead of database names being listed, DerivedVar:: is written followed with the list of CCHS variables used inside square brackets.
DerivedVar::[var1, var2, var3]
In recEnd, write Func:: followed with the name of the custom function used to create the derived variable.
Func::derivedFunction

A derived variable looks like this in variable_details.csv

sample_variable_details <- read.csv(file.path(getwd(), '../inst/extdata/sample_variable_details.csv'))
kable(sample_variable_details[sample_variable_details$variable == "derivedVariable",], options = list(dom='t'))