In emeyers/SDS230: Tools for the class Data Exploration and Analysis

$\$

# install.packages("gapminder")

SDS230::download_data("amazon.rda")

SDS230::download_image("qq-plot.png")

devtools::install_github("dill/emoGG")

# install.packages("latex2exp")

library(latex2exp)

library(dplyr)


#options(scipen=999)


knitr::opts_chunk$set(echo = TRUE)

# hide all plot output - useful for printing the code
# knitr::opts_chunk$set(fig.show='hide')

set.seed(123)

$\$

Overview

• ggplot bonus features
• visual hypothesis tests
• functions
• qqplots

$\$

Part 1: Review of ggplot

ggplot2 is a R package created by Hadley Wickham that implements Leland Wilkinson's concept of a grammar of graphics. In the grammar of graphics (and ggplot) data graphics are composed of:

• a frame
• glyphs
• scales and guides
• facets
• layers
• themes

$\$

Part 1.1: Color scales using scale_color_brewer

Let's use the gapminder data to construct a visualization using this grammar that has:

1. A basic frame constructed from called the ggplot() function

2. A geom of points where:

3. gdpPercap is mapped to the x position
4. lifeExp is mapped to the y position

5. continent is mapped to color

6. Transform the scales so that

7. The x values are on a $log_10$ scale
8. The color scale using a different qualitative color brewer pallet

library(ggplot2)
library(gapminder)


ggplot(data = gapminder) +                                                 # create the basic frame
    geom_point(aes(x = gdpPercap, y = lifeExp, color = continent))  +        # add a geom with 3 aesthetic mappings 
  scale_x_continuous(trans = "log10") +                                          # change the scales
  scale_color_brewer(type = "qua", palette = 2)                     

$\$

Part 1.2: ggplot themes

Let's continue to use the gapminder data to construct a visualization using this grammar that has:

1. The gapminder data filtered to only have data from France, Haiti and the United States

2. A geom of line where:

3. year is mapped to the x position

4. lifeExp is mapped to the y position

5. Faceting based on the country

6. A second glyph layer with a red vertical line at the year 1969

7. The Wall Street Journal theme from the ggthemes package

library(dplyr)
library(ggthemes)


gapminder |>
  filter(country %in% c("France", "Haiti", "United States")) |>
ggplot(aes(year, lifeExp)) + 
  geom_line() + 
    facet_wrap(~country)  +        # add a geom with 3 aesthetic mappings 
  geom_vline(xintercept = 1969, col = "red") + 
  theme_wsj()

$\$

Part 1.3: geom_text()

We can also create: - Text as a glyph using geom_text() - Annotations using annotate(geom_type, geom_properties) - Manually modify the theme using theme() with different arguments

Let's create a plot where:

1. It only has data from 2007

2. A text glyph with the following properties:

3. gdpPercap is mapped to the x position
4. lifeExp is mapped to the y position
5. continent is mapped to color

6. country is mapped to the text label

7. Let's use a log 10 scale

8. Let's turn off the legend by modifying the theme

9. Let's add a text annotation with the following attributes

10. x = 600
11. y = 80
12. label = "this plot is messy :("

gapminder |>
  filter(year == 2007) |>
  ggplot(aes(x = gdpPercap, y = lifeExp, label = country, color = continent)) + 
  geom_text() + 
  scale_x_continuous(trans = "log10") + 
  theme(legend.position="none") + 
  annotate("text", x = 600, y = 80, label = "this plot is messy :(") 

$\$

Part 2: Additional ggplot themes and features

$\$

Part 2.1: Plotly

The plotly package can be used to create interactive visualizations.

Let's recreate an interactive version of the gapminder data from 2007 using a geom_point() and:

• gdpPercap is mapped to the x position
• lifeExp is mapped to the y position
• continent is mapped to color
• country is mapped to the name

We will save our plot to an object g and then we can use ggplotly(g) to create an interactive visualization.

library(gapminder)
library(plotly)


g <- gapminder |> 
  filter(year == 2007) |>   
  ggplot(aes(x = gdpPercap, y = lifeExp, col = continent, name = country)) +      
  geom_point()   

ggplotly(g)

$\$

Part 2.2: emojis

Everyone loves emojis!

Let's use the motor trends car data to plot miles per gallon (mpg) as a function of weight using the proper glyph.

# devtools::install_github("dill/emoGG")
library(emoGG)
ggplot(mtcars, aes(wt, mpg)) + 
   geom_emoji(emoji="1f697")



# another option...
#install.packages("emojifont")
# 
# library(emojifont)
# 
# load.emojifont('OpenSansEmoji.ttf')
# 
# 
# ggplot(mtcars, aes(x = wt, y = mpg, label = emoji("car"), col = factor(cyl))) +       
#   geom_text(family="OpenSansEmoji", size=6)  +    
#   xlab("Weight") +   
#   ylab("Miles per Gallon") + 
#   theme_classic()

$\$

Part 2.3: animations

We can crate animated gifs too...

#install.packages('png')
#install.packages('gifski')
#install.packages('gganimate')


# library(gganimate)
# 
# 
# ggplot(gapminder, aes(gdpPercap, lifeExp, size = pop, colour = country)) +
#   geom_point(alpha = 0.7, show.legend = FALSE) +
#   scale_colour_manual(values = country_colors) +
#   scale_size(range = c(2, 12)) +
#   scale_x_log10() +
#   facet_wrap(~continent) +
#   # Here comes the gganimate specific bits
#   labs(title = 'Year: {frame_time}', x = 'GDP per capita', y = 'life expectancy') +
#   transition_time(year) +
#   ease_aes('linear')

$\$

Part 3: Visual hypothesis tests

In "visual hypothesis tests", we assess whether we can identify which plot contains relationships in the data, from plots that show scrambled versions of the data. For more information about these tests, and an R package that implements these tests, see:

• https://cran.r-project.org/web/packages/nullabor/vignettes/nullabor.html
• https://vita.had.co.nz/papers/inference-infovis.pdf

$\$

Part 3.1: An example problem

Let's run a visual hypothesis test to see if there is a correlation between the number of pages in a book and the price. We can start by stating the null and alternative hypothesis:

In words:

Null hypothesis: there is no relationship between the number of pages in a book and the price.

Alternative hypothesis: books that have more pages have higher prices

In symbols:

$H_0: \rho = 0$

$H_0: \rho > 0$

Let's now run the visual hypothesis test. Where we have:

• The "observed visualization" (step 2) is the plot of the actual data.

• The "null visualizations" (step 3) are visualizations of data where the relationship between weight and mpg has been shuffled.

• The "p-value/decision" (step 4-5) is whether we can identify the real plot from the shuffled plots.

$\$

Part 3.2: Running the analysis

Let's use the nullabor package to run the analysis. We will use the null_permute() function to randomly shuffle the List.Price variable and we will use the lineup() to create a data frame that has the real data and 19 other randomly shuffled data sets. We can then plot the real and shuffled data sets using ggplot() to see if we can identify the real one. We can also use the decrypt() function to reveal the answer about which data set is the real one.

# install.packages("nullabor")

library(nullabor)
library(ggplot2)

load("amazon.rda")

d <- lineup(null_permute("List.Price"), amazon)

ggplot(data=d, aes(x=NumPages, y=List.Price)) + 
  geom_point() + 
  facet_wrap(~ .sample)

The nullabor package can be used to examine other types of relationships, including examining differences in word useage (word clouds) and assessing whether there are trends on a map.

$\$

Part 4: Writing functions in R and conditional statements

Part 4.1: Writing functions

We can write functions in R using the function() function!

Let's write a function called cube_it() that takes a value x and returns x cubed.

# the square root function
sqrt(49)



# a function that takes a value to the third power
cube_it <- function(x) {

  x^3

}


cube_it(2)

$\$

Part 4.2: Addtional function arguments

We can specify additional arguments which can also have default values.

Let's write a function called power_it which takes a value x and a value pow and returns x to the pow power. Let's also have the default value of the pow argument be 3.

# a function that takes a value to the third power
power_it <- function(x, pow = 3) {

  x^pow

}



power_it(2)

power_it(2, 8)

$\$

Part 4.3: Writing our own permutation test function for correlation

Let's write our permutation test for testing whether there is a significant correlation in data.

We can then test it to see if there is a correlation between the price of a book and the number of pages in a book.

# write a cor_permtest function that tests where there is a significant correlation
# between the values in two vectors
cor_permtest <- function(vec1, vec2, n_null = 10000) {


  # calculate the observed statistic
  obs_stat <- cor(vec1, vec2)


  # create the null distribution 
  null_dist <- NULL
  for (i in 1:n_null) {
    curr_null_stat <- cor(vec1, sample(vec2))
    null_dist[i] <- curr_null_stat
  }


  # Calculate the two-tailed p-value
  pval_left <- mean(null_dist <= -1 * abs(obs_stat))
  pval_right <- mean(null_dist >= abs(obs_stat))

  pval <- pval_left + pval_right


  # return the p-value
  pval


}




# test the function to see if there is a correlation between the number of pages in a book
# and the price of a book 
load("amazon.rda")


cor_permtest(amazon$NumPages, amazon$List.Price)

Try this at home: write a function that can run a permutation test comparing two means from two different data samples (e.g., a function that can run a permutation test to assess whether cognitive abilities different depending on whether participants took a Gingko supplement or a placebo).

$\$

Part 5: Examining how data is distributed using quantile-quantile plots

We can check if empirical data seems to come from a particular distribution using quantile-quantile plots (qqplots). qqplots plot the sorted data values in your sample as a function of the theoretical quantile values (at evenly spaced probability areas).

Below is an illustration of quantile values for 10 data points that come from a normal distribution. If we have 10 data points in our sample, then to create a qqplot comparing our data to a standard normal distribution we would plot our data as a function of these theoretical quantile values. If the plot falls along a diagonal line, this indicates our data comes from the standard normal distribution

Also see this explanation

$\$

Part 5.1

Let's create a qqplot to assess whether the thickness of books on Amazon are normally distributed.

# get the heights of all baseball players
load("amazon.rda")
thickness <- amazon$Thick



# view a histogram of book thicknesses
hist(thickness, breaks = 20)


# create an sequence of values between 0 and 1 at even spaces
prob_area_vals <- seq(0, 1, length.out = length(thickness))


# get the quantiles from these values
quantile_vals <- qnorm(prob_area_vals)


# create the qqplot
plot(quantile_vals, sort(thickness),
     xlab = "Normal quantiles",
     ylab = "Data quantiles",
     main = "Quantile-quantile plot")

$\$

Part 5.2

We can also use the qqnorm() function to do this more easily when comparing data to the normal distribution. Or, if we want a better visualization, we can use the qqPlot() function in the car package.

qqnorm(thickness)

#install.packages("car")
car::qqPlot(thickness)

This data is pretty normal as can see in the plots above. Let's look at some highly skewed data.

# data that is skewed to the right
exp_data <- rexp(1000)

hist(exp_data, breaks = 50, 
     main = "Data from an exponential distribution", 
     xlab = "Data values")

qqnorm(exp_data)



# data that is skewed to the left
exp_data <- -1 * rexp(1000)

hist(exp_data, breaks = 50, 
     main = "Left skewed data", 
     xlab = "Data values")

qqnorm(exp_data)

emeyers/SDS230 documentation built on Jan. 18, 2024, 1:01 a.m.