In bwiernik/progdata: Tutorials for USF Programming with Data

library(learnr)
library(dplyr)
library(tidyr)
library(gradethis)
library(stringr)
gradethis_setup()
knitr::opts_chunk$set(echo = FALSE)

Getting Started

Today’s concept is tidy data and the tidyr package.

In fact tidyr Version 1.0.0 just came out recently with some great new additions that we’ll be looking at. We’ll focus on:

• Reshaping data by pivoting with tidyr::pivot_longer() and tidyr::pivot_wider().
• Making tibbles using tibble::tibble() and tidyr::expand_grid().

Resources

For concepts of tidy data:

• Jenny Bryan’s intro to tidy data is short and sweet.
• the repo this links to has some useful exercises too, but uses the older spread() and gather() functions.
• tidyr vignette on tidy data.
• Hadley’s paper on tidy data provides a thorough investigation.

For pivoting with tidyr, check out the pivot vignette

I also recommend reading the new additions that come with the new tidyr Version 1.0.0 in this tidyverse article.

Tidy Data

A data set is tidy if:

• Each row is an observation;
• Each column is a variable;
• Each cell is a value.

This means that each value belongs to exactly one variable and one observation.

Why bother? Because doing computations with untidy data can be a nightmare. Computations become simple with tidy data.

This also means that tidy data is relative, as it depends on how you define your observational unit and variables.

As an example, consider this example derived from the datasets::HairEyeColor dataset, containing the number of people having a certain hair and eye colour.

If one observation is identified by a hair-eye color combination, then the tidy dataset is:

haireye <- datasets::HairEyeColor %>%
  as_tibble() %>%
  janitor::clean_names() %>%
  select(-c(sex)) %>%
  distinct(hair, eye, .keep_all = TRUE)

haireye %>%
  knitr::kable()

haireye %>%
  tidyr::uncount(n) %>%
  head()

Untidy Examples

The following are examples of untidy data. They’re untidy for either of the cases considered above, but for discussion, let’s take a hair-eye colour combination to be one observational unit.

Note that untidy does not always mean “bad”, especially when the data set is too wide.

Untidy Example 1: The following table is untidy because there are multiple observations per row. Its too wide.

haireye_untidy <- haireye %>%
  mutate(eye = str_c(eye, "_eyed")) %>%
  pivot_wider(id_cols = hair, names_from = eye, values_from = n)

haireye_untidy

Let me ask you, if you wanted to calculate the total number of people with each hair color how would you do it?

It seems like a simple question but because the data are in the format it is, it can be a challenge.

Untidy Example 2: This is untidy because each observational unit is spread across multiple columns. It’s too long. In fact, we needed to add an identifier for each observation, otherwise we would have lost which row belongs to which observation!

haireye %>% 
  mutate(obs = 1:n()) %>% 
  pivot_longer(cols = hair:eye, names_to = "body_part", values_to = "colour") %>% 
  select(-n, n) %>% 
  DT::datatable(rownames = FALSE)

Untidy Example 3: Just when you thought a data set couldn't get any longer! Now, each variable has its own row: hair colour, eye colour, and n. This demonstrates that there's no such thing as "long" and "wide" format, since these terms are relative.

haireye %>% 
  mutate(obs = 1:n(),
         n   = as.character(n)) %>% 
  pivot_longer(cols = c(hair, eye, n), names_to = "variable", values_to = "value") %>% 
  DT::datatable(rownames = FALSE)

This is the sort of format that is common pulling data from the web or other "Big Data" sources.

Pivoting Tools

The task of making tidy data is about making data either longer, by stacking two or more rows, or wider, by putting one or more columns alongside each other based on groups. This is called pivoting (or, reshaping).

Sometimes, tidy data is incorrectly referred to as data in long format as opposed to wide format, where “length” refers to the number of rows, and “width” the number of columns. But Example 3 of untidy data (above) is in fact too long and needs to be made wider! However, usually the task of tidying data involves lengthening, and usually the task of widening is useful for turning data into something more friendly for human eyes.

The (new!) easiest and most powerful way to widen or lengthen data are with the functions tidyr::pivot_wider() and tidyr::pivot_longer().

History: R has seen many attempts at reshaping, all that’s progressively gotten better. First came the reshape package. Then the reshape2 package. Both were finicky. And they used function names that I could never remember: melt() and cast(). Then, the tidyr::spread() and tidyr::gather() functions provided a simple interface (and are still part of the tidyr package!), but used awkward terminology and weren’t as flexible as they ought to be.

Univariate Pivoting

Let’s start with pivoting in the simplest case where only one variable is “out of place”. We’ll use the hair and eye colour example from before, using the untidy data version from Example 1:

haireye_untidy <- haireye %>%
  mutate(eye = str_c(eye, "_eyed")) %>%
  pivot_wider(id_cols = hair, names_from = eye, values_from = n)

haireye_untidy

The eye colour variable is spread out across columns. To fix this, we need to convert the eye colour columns to two columns:

• one column to hold the eye colour (column names),
• one column to hold the values.

Doing this, we obtain:

haireye_untidy %>% 
  pivot_longer(contains("eyed"), names_to = "eye", values_to = "n")

For the reverse operation, we take the column eye and make each unique entry a new column, and the values of those columns take on n.

pivot_longer()

pivot_longer() takes a data frame, and returns a data frame. The arguments after the data argument that we’ll need are:

• cols for the column names that we want to turn into a single column.
• names_to: the old column names are going to a new column. What should this new column be named? (optional, but highly recommended)
• values_to: the values underneath the old columns are going to a new column. What should this new column be named? (optional, but highly recommended)

Possibly the trickiest bit is in identifying the column names. We could list all of them, but it’s not robust to changes:

haireye_untidy %>%
  pivot_longer(
    cols = c(Blue_eyed, Brown_eyed, Green_eyed, Hazel_eyed),
    names_to = "eye",
    values_to = "n"
  )

We could identify a range. This is more robust, but still not very robust.

haireye_untidy %>%
  pivot_longer(
    cols = Blue_eyed:Hazel_eyed,
    names_to = "eye",
    values_to = "n"
  )

Better is to use helper functions from the tidyselect package. In this case, we know the columns contain the text “eyed”, so let’s use tidyselect::contains():

haireye_untidy %>%
  pivot_longer(
    cols = contains("eyed"),
    names_to = "eye",
    values_to = "n"
  )

Yet another way is to indicate everything except the hair column:

haireye_untidy %>%
  pivot_longer(
    cols = -hair,
    names_to = "eye",
    values_to = "n"
  )

pivot_wider()

Like pivot_longer(), pivot_wider() takes a data frame and returns a data frame. The arguments after the data argument that we’ll need are:

• id_cols: The columns you would like to keep. If widening to make data tidy, then this is an identifier for an observation.
• names_from: the new column names are coming from an old column. Which column is this?
• values_from: the values under the new columns are coming from an old column. Which column is this?

haireye %>%
  pivot_wider(
    id_cols = hair,
    names_from = eye,
    values_from = n
  )

You try!

Consider the Lord of the Rings data:

lotr  <- readr::read_csv("https://raw.githubusercontent.com/jennybc/lotr-tidy/master/data/lotr_tidy.csv")
lotr

1. We want to make it so we can graph the amount of words for each gender within races from each movie

question("Would you say this data is in tidy format?",
         answer("Yes", correct = TRUE),
         answer("No"),
         random_answer_order = TRUE,
         allow_retry = TRUE
)

1. Widen the data so that we see the words spoken by each race, by putting race as its own column.

lotr_wide <- lotr %>%
  pivot_wider(
    FILL_THIS_IN = c(-Race, -Words),
    FILL_THIS_IN = Race,
    FILL_THIS_IN = Words
  )

lotr_wide <- lotr %>%
  pivot_wider(
    id_cols = c(-Race, -Words),
    names_from = Race,
    values_from = Words
  )

grade_code()

1. Re-lengthen the wide LOTR data from Question 2 above.

lotr_wide <- lotr %>%
  pivot_wider(
    id_cols = c(-Race, -Words),
    names_from = Race,
    values_from = Words
  )

lotr_wide %>%
  pivot_longer(
    FILL_THIS_IN = FILL_THIS_IN,
    names_to = FILL_THIS_IN,
    values_to = FILL_THIS_IN
  )

lotr_wide %>%
  pivot_longer(
    cols = c(-Film, -Gender),
    names_to = "Race",
    values_to = "Words"
  )

grade_code()

Multivariate Pivoting

Now let’s consider the case when more than one variable are “out of place” – perhaps there are multiple variables per row, and/or multiple observations per row.

For example, consider the (lightly modified) iris data set that we’ll call iris2:

iris2 <- iris %>%
  mutate(id = 1:n()) %>% 
  rename(species = Species) %>% 
  pivot_longer(c(-species, -id), 
               names_to  = "variable", 
               values_to = "measurement") %>% 
  mutate(variable = variable %>% 
           str_replace("\\.", "_") %>% 
           tolower()) %>% 
  pivot_wider(c(id, species), 
              names_from  = variable, 
              values_from = measurement)

iris2_longest <- iris2 %>% 
  pivot_longer(cols      = c(-species, -id), 
               names_to  = c("part", "dimension"),
               names_sep = "_",
               values_to = "measurement")

iris2_longer <- iris2 %>% 
  pivot_longer(cols      = c(-id, -species), 
               names_to  = c("part", ".value"), 
               names_sep = "_")
iris2 %>% 
  head() %>% 
  knitr::kable() 

Although we probably wouldn’t, we could view this as having two variables bundled into the column names:

“Plant part”, either sepal or petal. “Dimension”, either length or width. The resulting tidy data frame would then be:

iris2_longest

More realistic is the situation where there are multiple observations per row:

An observation of (length, width) of the sepal. An observation of (length, width) of the petal. The resulting tidy data frame has a length that’s in between the above two:

iris2_longer

pivot_longer()

To obtain the case where two (or more) variables are contained in column names, here’s how we specify the arguments of pivot_longer():

• cols: As usual.
• names_sep: What is separating the variables in the column names?
• names_to: The old columns are going to be put into new columns, after being separated. What should those columns be named?
• values_to: As usual.

Here is the code:

iris2 %>% 
  pivot_longer(cols      = c(-species, -id), 
               names_to  = c("part", "dimension"),
               names_sep = "_",
               values_to = "measurement")

To obtain the case where multiple observations are contained in one row, here’s how to specify the arguments of pivot_longer():

cols: As usual. names_sep: As above. names_to: As above, except this time, one part of the old column names are going to stay as columns (in this case, “length” and “width”). Indicate ".value" instead of a new column name. values_to: Not needed! You’ve already indicated that using the ".value" placeholder.

iris2 %>% 
  pivot_longer(cols      = c(-id, -species), 
               names_to  = c("part", ".value"), 
               names_sep = "_")

pivot_wider()

If two or more columns contain parts of a variable name (i.e., each unique combination of these columns gives rise to a new variable), here’s how we can use pivot_wider():

• id_cols: as usual.
• names_from: the new variable names are coming from old columns. Which old columns?
• names_sep: What character should you separate the entries of the old columns by?
• values_from: as usual.

Here is the code to go from the longest form to the original:

iris2_longest %>% 
  pivot_wider(id_cols     = c(id, species),
              names_from  = c(part, dimension), 
              names_sep   = "_", 
              values_from = measurement)

If variables are spread out amongst rows and columns (for example, “sepal width” has “sepal” in a column, and “width” as a column name), here’s how we can use pivot_wider():

• id_cols: as usual
• names_from: Which column contains the part of the variable?
• names_sep: As before, what character should you separate the entries of the old columns by?
• values_from: Which column names contain the other part of the variable?

Here is the code to go from the “semi-long” form to the original:

iris2_longer %>% 
  pivot_wider(id_cols     = c(id, species), 
              names_from  = part, 
              names_sep   = "_",
              values_from = c(length, width))

You Try!

fam_data <- read.csv("https://raw.githubusercontent.com/wesley4546/teach-r-temp/master/data/family_data.csv?token=ANSE7O4LLSHB3RH7IFICDFK7ZVXLC")
fam_long_data <- read.csv("https://raw.githubusercontent.com/wesley4546/teach-r-temp/master/data/family_long_data.csv?token=ANSE7O5METBJEZG6MGXP4DS7ZVXM6")

fam_data %>%
  pivot_longer(
    cols = FILL_THIS_IN,
    names_to = FILL_THIS_IN,
    names_sep = FILL_THIS_IN,
    values_to = FILL_THIS_IN
  )

fam_data %>% 
  pivot_longer(cols = (-family_id),
               names_to = c("wave","type","member"),
               names_sep = "_",
               values_to = "score")

grade_code()

Making tibbles

In base R, we can make data frames using the data.frame() function. The tidyverse version is tibble::tibble(), which also has backwards referencing to variables you make on the fly. It’s also stricter by not allowing recycling unless the vector is of length 1:

Good:

tibble(x = 1:6,
       y = min(x))

Bad:

tibble(x = 1:6,
       y = 1:2)

Truly manual construction of tibbles is easy with tibble::tribble():

tribble(
  ~Day, ~Breakfast,
  1, "Apple",
  2, "Yogurt",
  3, "Yogurt"
)

Check out the datapasta package for ways to reproducibly copy-paste data from spreadsheets into R. datapasta uses tribble().

List columns are easy with tibbles!

(list_col <- tibble(n = 1:2,
                    y = list(iris, mtcars)))

Often obtained with nest() and unnest():

(iris_nest <- iris %>% 
   group_by(Species) %>% 
   nest())

iris_nest %>% 
  unnest(data)

expand_grid() to obtain all combinations:

expand_grid(x = 1:2, y = 1:2, z = 1:2)

In conjunction with nesting():

expand_grid(nesting(x = 1:2, y = 1:2), z = 1:2)

Implicit NA’s

Sometimes there’s “hidden” missing data in a tibble. Here’s an example from the documentation of tidyr::expand():

(df <- tibble(
  year   = c(2010, 2010, 2010, 2010, 2012, 2012, 2012),
  qtr    = c(   1,    2,    3,    4,    1,    2,    3),
  return = rnorm(7)
))

Here, there are values for qtr 1, 2, 3, 4 in year 2010 and qtr 1, 2, 3 in year 2012, but no qtr 4 in year 2012. The value of return in year 2012, qtr 4 is “implicilty” NA.

We can use functions to fill out this data frame and make these “implicit” missing values explicit.

We can consider all existing combinations by invoking the column names in expand() or complete() (which either drops or keeps all other columns):

df %>% 
  expand(year, qtr)

df %>% 
  complete(year, qtr)

We can consider new combinations by specifying an expectation of possible values:

df %>% 
  expand(year = 2010:2012, qtr)

df %>% 
  complete(year = 2010:2012, qtr)

Want to link two or more columns when looking for combinations? Use nesting().

You try!

1. Create a tibble that has the following columns:

2. A label column with "Sample A" in its entries.

3. 100 random observations drawn from the N(0,1) distribution in the column x
4. "N" means the normal distribution. "(0, 1)" means mean = 0, sd = 1.
5. Use rnorm()
6. y calculated as the x values + N(0,1) error.

n <- 100
FILL_THIS_IN(label = FILL_THIS_IN,
             FILL_THIS_IN = rnorm(n),
             FILL_THIS_IN)

bwiernik/progdata documentation built on Feb. 1, 2021, 2:33 a.m.