In LukasWallrich/GoldCoreQuants: Does not matter.

Importing and describing data

Importing data into R

Generally, R is a poor choice for data entry, so you will usually import data from spreadsheet software or other sources. That also helps to keep data and analyses apart, and thereby increases transparency.

The most common ways to import data into R are to

read tabular data saved as text (such as the common .csv files) with the readr package
read Excel files with the readxl package
read data files from SPSS or SAS with haven.

In these packages, the functions to read data are always read_x(), with x replaced by the type of file you want to open. You always need to specify the name of the datafile, with its path (unless it is in the same folder as your .Rmd file), as the first argument of the read_x() function. For the path, it can be helpful to start with the dot (.), which means: start from the current folder. For example, I often have a data folder within the folder where the .Rmd file is, so that the path would then be "./data/file.csv". Always remember to assign the result of the function (i.e. the data) to a variable using <-, otherwise the data is just printed and not saved in R.

The functions that load data from tables need to establish the variable class for each variable. They can either guess (which you will then need to check), or you can provide the classes in the col_types-argument. Check ?readr::read_csv for details.

#readr is loaded with the tidyverse, but can be loaded separately 
#as well if you don't need the other packages
pacman::p_load(readr)
gapminder <- read_csv("./data/gapminder.csv", col_types = "finnnnnff")

pacman::p_load(readxl)
gapminder <- read_xlsx("./data/gapminder.xlsx") 
# Here col_types are not defined so that R will guess and print 
# out the results - make sure to check whether the classes are 
# what you would expect

pacman::p_load(haven)
gapminder <- read_sav("./data/gapminder.sav") 
#SPSS files contain information on the class of each variable, 
#so that col_types do not need to be defined

There are shortcuts to quickly get some data into R, which can be helpful if you just want to try something out - all of them only make sense in the Console, not in a .Rmd script. Using the edit() function, you can open up a simple spreadsheet to edit and input data, using read.table(file="clipboard", sep="\t"), you can get data you copied in Excel. As with all ways of importing data, remember to assign the result to a variable.

#If you want to edit an existing dataset
gapminder_corrected <- edit(gapminder)

#If you want to type or copy in a new dataset
yourdata <- edit(data.frame())
#The data.frame() function creates an empty dataframe for you to then edit

#If you want to read data from the clipboard
pasted <- read.table(file="clipboard", sep="\t")

#If that data contains variable names in the first row:
pasted <- read.table(file="clipboard", sep="\t", header = TRUE)

#Note that reading data from the clipboard does not always work - 
#if it doesn't, try pasting your data into the window opened 
#with the edit() function

View data

When you start with a new dataset, there are some helpful functions to get a first look at the data: glimpse()gives a good overview of the variables contained in the dataset, while head() and tail() print the first and last lines. In the Console (but not really within scripts such as .Rmd files), you can also use View() to open up the whole dataset. Finally, you can have a look at data in the Environment pane in RStudio.

pacman::p_load(dslabs) #Load the gapminder teaching dataset
pacman::p_load(dplyr) #Load the dplyr package that offers glimpse()
glimpse(gapminder)

head(gapminder, n=5) #n defines number of rows shown

tail(gapminder, n=5)

Manipulating data and using dplyr

Once you have imported your data into R, you might need to filter it, sort it, add some new variables, and calculate summary statistics. The dplyr package (part of the tidyverse) is designed to make all these steps easier, so we use it extensively during this course. dplyr is based on two main ideas:

Develop a chain of data manipulation steps that get you towards the desired data, with the steps connected by the %>% operator ("pipe"). That removes the need for nested functions or for saving lots of intermediate results.
Use functions names after natural language verbs to develop code that can be easily understood.

%>% takes the argument on the left and places it as the first argument into the function on the right. For example:

x <- c(1,2,3, NA)

#The following two commands are equivalent
mean(x, na.rm = TRUE)
x %>% mean(na.rm = TRUE)

The most important dplyr functions are:

select(): select specific variables
filter(): filter rows based on condition
arrange(): sort data ascendingly (arrange(desc()) to reverse)
mutate(): create/change variables
summarise(): calculate summary statistics
group_by(): separate data into groups, usually to summarise by group

All functions use a dataframe as their first argument, usually from %>%. After that, variables in the dataframe are accessed just with their name (no $).

If we want to calculate the average income per capita in 2010 on each continent from the gapminder dataset, we need most of these functions - if you read the %>%-operator as 'then', and mentally add 'take' at the very start, you should be able to follow along quite naturally.

One thing to note: you can split R code into multiple lines as long as each line is incomplete. Since the lines here end with %>%, R includes the next line into the same command. If that operator was moved to the start of the next line, the code would no longer work.

pacman::p_load(dplyr)
gapminder %>% 
  filter(year == 2010) %>%
    mutate(gdpPerCap = gdp/population) %>%
      filter(!is.na(gdpPerCap)) %>% 
      #This filter() removes countries where either gdp or population is missing
        group_by(continent) %>% 
        #group_by() allows for summary statistics to be calculated 
        #for each continent separately
          summarise(AvgGdpPerCap_Nations = mean(gdpPerCap), 
                    AvgGdpPerCap_People = sum(gdp)/sum(population)) %>%
              mutate_if(is.numeric, round, 0) %>% 
              #This rounds all numeric variables to 0 decimal places 
              #(beyond expectations for this course)    
                arrange(desc(AvgGdpPerCap_Nations))

Just as an aside: consider the differences between the two ways of calculating averages per continent. One focuses on the average of country values, and thus yields the average national income per capita. The other takes population sizes into account and thus yields the average personal income per capita. For most continents, the difference in results is huge. Both approaches are used in the media - so it's always worth looking a little closer at summary statistics.

For an introduction to dplyr, you can watch this video:

r video_code("QtQE-b5iMUQ")

Summarise data (descriptive statistics)

Next you might want to use summary functions - either in a dplyr summarise() function or on their own. For most of them, you can use the na.rm = TRUE argument to tell R to ignore missing values (only do that when you have reason to assume that missing values can safely be ignored).

gapminder2010 <- gapminder %>% filter(year==2010)

#Global population
sum(gapminder2010$population, na.rm = TRUE)

#Number of countries included
nrow(gapminder2010)
#To get the number inside dplyr's summarise function, it would have to be
gapminder2010 %>% summarise(n_rows = n())

#Average (mean) number of children per woman
mean(gapminder2010$fertility, na.rm = TRUE)

#Average (median) number of children per woman
median(gapminder2010$fertility, na.rm = TRUE)

#Highest and lowest fertility rates
max(gapminder2010$fertility)
min(gapminder2010$fertility)

#Countries with highest and lowest fertility.
#There are many ways to look this up - here is a simple dplyr pipe 
## Note that | means 'or' in the context of logical comparisons 
## and == is needed to test for equality because = just assigns a value.

gapminder2010 %>% 
  filter(fertility == max(fertility) | fertility == min(fertility)) %>% 
    select(country, fertility)