data-raw/03_extract-from-excel-gdpPercap.md

r format(Sys.Date()) Cleaning history

suppressPackageStartupMessages(library(dplyr))
library(tidyr)
library(ggplot2)
library(readr)

Extract the GDP per capita data from the Excel file downloaded 2009-04-30 from gapminder.org.

2015-12-29 NOTE: I am punting here for the moment and basing intake on manually exported delimited text. For population and life expectancy, I have re-implemented the Excel extraction programmatically using gdata::read.xls() (2014) and then readxl (2015), which I plan to try here as well. But I dread doing GDP per capita because the spreadsheet is transposed relative to the other two. In Excel, I ruthlessly deleted tons of columns at front and up to 1950, then saved as tab-delimited text file gdpPercap.txt.

gdp_xls <- read_tsv("xls-manual-extract/gdpPercap.txt")
gdp_xls %>% str(list.len = 4)
## Classes 'tbl_df', 'tbl' and 'data.frame':    259 obs. of  59 variables:
##  $ Area: chr  "Abkhazia" "Afghanistan" "Akrotiri and Dhekelia" "Albania" ...
##  $ 1950: num  NA 757 NA 1532 2429 ...
##  $ 1951: num  NA 767 NA 1598 2398 ...
##  $ 1952: num  NA 779 NA 1601 2449 ...
##   [list output truncated]
gdp_xls %>% head()
## Source: local data frame [6 x 59]
## 
##                    Area      1950      1951      1952      1953      1954
##                   (chr)     (dbl)     (dbl)     (dbl)     (dbl)     (dbl)
## 1              Abkhazia        NA        NA        NA        NA        NA
## 2           Afghanistan  757.3188  766.7522  779.4453  812.8563  815.3595
## 3 Akrotiri and Dhekelia        NA        NA        NA        NA        NA
## 4               Albania 1532.3539 1598.4927 1601.0561 1665.7947 1714.6888
## 5               Algeria 2429.2137 2397.5311 2449.0082 2436.3400 2557.8185
## 6        American Samoa 4465.1447        NA        NA        NA        NA
## Variables not shown: 1955 (dbl), 1956 (dbl), 1957 (dbl), 1958 (dbl), 1959
##   (dbl), 1960 (dbl), 1961 (dbl), 1962 (dbl), 1963 (dbl), 1964 (dbl), 1965
##   (dbl), 1966 (dbl), 1967 (dbl), 1968 (dbl), 1969 (dbl), 1970 (dbl), 1971
##   (dbl), 1972 (dbl), 1973 (dbl), 1974 (dbl), 1975 (dbl), 1976 (dbl), 1977
##   (dbl), 1978 (dbl), 1979 (dbl), 1980 (dbl), 1981 (dbl), 1982 (dbl), 1983
##   (dbl), 1984 (dbl), 1985 (dbl), 1986 (dbl), 1987 (dbl), 1988 (dbl), 1989
##   (dbl), 1990 (dbl), 1991 (dbl), 1992 (dbl), 1993 (dbl), 1994 (dbl), 1995
##   (dbl), 1996 (dbl), 1997 (dbl), 1998 (dbl), 1999 (dbl), 2000 (dbl), 2001
##   (dbl), 2002 (dbl), 2003 (dbl), 2004 (dbl), 2005 (dbl), 2006 (dbl), 2007
##   (dbl).

Sadly, this file is transposed relative to population and life expectancy. Each row is a country and the columns give the GDP data for different years. WHY?!?

Reshape the data by gathering all the year variables.

gdp_tidy <- gdp_xls %>%
  gather(key = "Xyear", value = "gdpPercap", -Area)
gdp_tidy %>% str()
## Classes 'tbl_df', 'tbl' and 'data.frame':    15022 obs. of  3 variables:
##  $ Area     : chr  "Abkhazia" "Afghanistan" "Akrotiri and Dhekelia" "Albania" ...
##  $ Xyear    : Factor w/ 58 levels "1950","1951",..: 1 1 1 1 1 1 1 1 1 1 ...
##  $ gdpPercap: num  NA 757 NA 1532 2429 ...

Rename Area --> country and fix the years.

gdp_tidy <- gdp_tidy %>%
  rename(country = Area) %>%
  mutate(Xyear = levels(Xyear)[as.numeric(Xyear)],
         year = gsub("X", "", Xyear) %>% as.integer(),
         Xyear = NULL)
gdp_tidy %>% str()
## Classes 'tbl_df', 'tbl' and 'data.frame':    15022 obs. of  3 variables:
##  $ country  : chr  "Abkhazia" "Afghanistan" "Akrotiri and Dhekelia" "Albania" ...
##  $ gdpPercap: num  NA 757 NA 1532 2429 ...
##  $ year     : int  1950 1950 1950 1950 1950 1950 1950 1950 1950 1950 ...

Filter rows where gdpPercap is NA.

gdp_tidy <- gdp_tidy %>%
  filter(!is.na(gdpPercap))
gdp_tidy %>% str()
## Classes 'tbl_df', 'tbl' and 'data.frame':    10911 obs. of  3 variables:
##  $ country  : chr  "Afghanistan" "Albania" "Algeria" "American Samoa" ...
##  $ gdpPercap: num  757 1532 2429 4465 3363 ...
##  $ year     : int  1950 1950 1950 1950 1950 1950 1950 1950 1950 1950 ...

Look into the coverage by year.

summary(gdp_tidy$year)
##    Min. 1st Qu.  Median    Mean 3rd Qu.    Max. 
##    1950    1967    1981    1981    1995    2007
(p <- ggplot(gdp_tidy, aes(x = year)) + geom_histogram(binwidth = 1))

\

Unlike population and life expectancy, there's no obvious reason to filter on year at this point.

Is country ok as is?

n_distinct(gdp_tidy$country)
## [1] 229
unique(gdp_tidy$country)
##   [1] "Afghanistan"                      "Albania"                         
##   [3] "Algeria"                          "American Samoa"                  
##   [5] "Angola"                           "Argentina"                       
##   [7] "Armenia"                          "Australia"                       
##   [9] "Austria"                          "Azerbaijan"                      
##  [11] "Bahamas"                          "Bahrain"                         
##  [13] "Bangladesh"                       "Barbados"                        
##  [15] "Belarus"                          "Belgium"                         
##  [17] "Belize"                           "Benin"                           
##  [19] "Bermuda"                          "Bhutan"                          
##  [21] "Bolivia"                          "Bosnia and Herzegovina"          
##  [23] "Botswana"                         "Brazil"                          
##  [25] "Brunei"                           "Bulgaria"                        
##  [27] "Burkina Faso"                     "Burundi"                         
##  [29] "Cambodia"                         "Cameroon"                        
##  [31] "Canada"                           "Cape Verde"                      
##  [33] "Central African Rep."             "Chad"                            
##  [35] "Chile"                            "China"                           
##  [37] "Colombia"                         "Comoros"                         
##  [39] "Congo, Dem. Rep."                 "Congo, Rep."                     
##  [41] "Costa Rica"                       "Cote d'Ivoire"                   
##  [43] "Croatia"                          "Cuba"                            
##  [45] "Cyprus"                           "Czech Rep."                      
##  [47] "Denmark"                          "Djibouti"                        
##  [49] "Dominica"                         "Dominican Rep."                  
##  [51] "Ecuador"                          "Egypt"                           
##  [53] "El Salvador"                      "Equatorial Guinea"               
##  [55] "Eritrea"                          "Estonia"                         
##  [57] "Ethiopia"                         "Finland"                         
##  [59] "France"                           "French Guiana"                   
##  [61] "French Polynesia"                 "Gabon"                           
##  [63] "Gambia"                           "Germany"                         
##  [65] "Ghana"                            "Greece"                          
##  [67] "Grenada"                          "Guadeloupe"                      
##  [69] "Guatemala"                        "Guinea"                          
##  [71] "Guinea-Bissau"                    "Guyana"                          
##  [73] "Haiti"                            "Honduras"                        
##  [75] "Hong Kong, China"                 "Hungary"                         
##  [77] "Iceland"                          "India"                           
##  [79] "Indonesia"                        "Iran"                            
##  [81] "Iraq"                             "Ireland"                         
##  [83] "Isle of Man"                      "Israel"                          
##  [85] "Italy"                            "Jamaica"                         
##  [87] "Japan"                            "Jordan"                          
##  [89] "Kazakhstan"                       "Kenya"                           
##  [91] "Korea, Dem. Rep."                 "Korea, Rep."                     
##  [93] "Kuwait"                           "Kyrgyzstan"                      
##  [95] "Laos"                             "Lebanon"                         
##  [97] "Lesotho"                          "Liberia"                         
##  [99] "Libya"                            "Lithuania"                       
## [101] "Luxembourg"                       "Macao, China"                    
## [103] "Macedonia, FYR"                   "Madagascar"                      
## [105] "Malawi"                           "Malaysia"                        
## [107] "Maldives"                         "Mali"                            
## [109] "Malta"                            "Martinique"                      
## [111] "Mauritania"                       "Mauritius"                       
## [113] "Mexico"                           "Moldova"                         
## [115] "Mongolia"                         "Montenegro"                      
## [117] "Morocco"                          "Mozambique"                      
## [119] "Myanmar"                          "Namibia"                         
## [121] "Nepal"                            "Netherlands"                     
## [123] "New Caledonia"                    "New Zealand"                     
## [125] "Nicaragua"                        "Niger"                           
## [127] "Nigeria"                          "Northern Mariana Islands"        
## [129] "Norway"                           "Oman"                            
## [131] "Pakistan"                         "Panama"                          
## [133] "Papua New Guinea"                 "Paraguay"                        
## [135] "Peru"                             "Philippines"                     
## [137] "Poland"                           "Portugal"                        
## [139] "Puerto Rico"                      "Reunion"                         
## [141] "Romania"                          "Russia"                          
## [143] "Rwanda"                           "Saint Lucia"                     
## [145] "Saint Vincent and the Grenadines" "Sao Tome and Principe"           
## [147] "Saudi Arabia"                     "Senegal"                         
## [149] "Serbia"                           "Seychelles"                      
## [151] "Sierra Leone"                     "Singapore"                       
## [153] "Slovak Republic"                  "Slovenia"                        
## [155] "Somalia"                          "South Africa"                    
## [157] "Spain"                            "Sri Lanka"                       
## [159] "Sudan"                            "Suriname"                        
## [161] "Swaziland"                        "Sweden"                          
## [163] "Switzerland"                      "Syria"                           
## [165] "Taiwan"                           "Tajikistan"                      
## [167] "Tanzania"                         "Thailand"                        
## [169] "Timor-Leste"                      "Togo"                            
## [171] "Trinidad and Tobago"              "Tunisia"                         
## [173] "Turkey"                           "Turkmenistan"                    
## [175] "Uganda"                           "Ukraine"                         
## [177] "United Kingdom"                   "United States"                   
## [179] "Uruguay"                          "Uzbekistan"                      
## [181] "Wallis et Futuna"                 "Venezuela"                       
## [183] "West Bank and Gaza"               "Vietnam"                         
## [185] "Yemen, Rep."                      "Zambia"                          
## [187] "Zimbabwe"                         "United Arab Emirates"            
## [189] "Fiji"                             "Georgia"                         
## [191] "Latvia"                           "Solomon Islands"                 
## [193] "Andorra"                          "Anguilla"                        
## [195] "Antigua and Barbuda"              "Aruba"                           
## [197] "British Virgin Islands"           "Cayman Islands"                  
## [199] "Cook Islands"                     "Greenland"                       
## [201] "Kiribati"                         "Liechtenstein"                   
## [203] "Micronesia, Fed. Sts."            "Monaco"                          
## [205] "Montserrat"                       "Nauru"                           
## [207] "Netherlands Antilles"             "Palau"                           
## [209] "Qatar"                            "Saint Kitts and Nevis"           
## [211] "San Marino"                       "Turks and Caicos Islands"        
## [213] "Tuvalu"                           "Guernsey"                        
## [215] "Marshall Islands"                 "Samoa"                           
## [217] "Tonga"                            "Vanuatu"                         
## [219] "Saint Helena"                     "Faeroe Islands"                  
## [221] "Falkland Islands (Malvinas)"      "Gibraltar"                       
## [223] "Jersey"                           "Kosovo"                          
## [225] "Mayotte"                          "Niue"                            
## [227] "Northern Cyprus"                  "Saint-Pierre-et-Miquelon"        
## [229] "Tokelau"

No obvious train wrecks. Save for now.

write_tsv(gdp_tidy, "03_gdpPercap.tsv")

devtools::session_info()
## Session info --------------------------------------------------------------
##  setting  value                       
##  version  R version 3.2.3 (2015-12-10)
##  system   x86_64, darwin13.4.0        
##  ui       X11                         
##  language (EN)                        
##  collate  en_CA.UTF-8                 
##  tz       America/Vancouver           
##  date     2015-12-29
## Packages ------------------------------------------------------------------
##  package    * version    date       source                          
##  assertthat   0.1        2013-12-06 CRAN (R 3.2.0)                  
##  colorspace   1.2-6      2015-03-11 CRAN (R 3.2.0)                  
##  DBI          0.3.1      2014-09-24 CRAN (R 3.2.0)                  
##  devtools     1.9.1.9000 2015-12-18 Github (hadley/devtools@9aaa3af)
##  digest       0.6.8      2014-12-31 CRAN (R 3.2.0)                  
##  dplyr      * 0.4.3.9000 2015-11-24 Github (hadley/dplyr@4f2d7f8)   
##  evaluate     0.8        2015-09-18 CRAN (R 3.2.0)                  
##  formatR      1.2.1      2015-09-18 CRAN (R 3.2.0)                  
##  ggplot2    * 2.0.0      2015-12-18 CRAN (R 3.2.3)                  
##  gtable       0.1.2      2012-12-05 CRAN (R 3.2.0)                  
##  htmltools    0.2.6      2014-09-08 CRAN (R 3.2.0)                  
##  knitr        1.11.16    2015-11-23 Github (yihui/knitr@6e8ce0c)    
##  labeling     0.3        2014-08-23 CRAN (R 3.2.0)                  
##  lazyeval     0.1.10     2015-01-02 CRAN (R 3.2.0)                  
##  magrittr     1.5        2014-11-22 CRAN (R 3.2.0)                  
##  memoise      0.2.1      2014-04-22 CRAN (R 3.2.0)                  
##  munsell      0.4.2      2013-07-11 CRAN (R 3.2.0)                  
##  plyr         1.8.3      2015-06-12 CRAN (R 3.2.0)                  
##  R6           2.1.1      2015-08-19 CRAN (R 3.2.0)                  
##  Rcpp         0.12.2     2015-11-15 CRAN (R 3.2.2)                  
##  readr      * 0.2.2      2015-10-22 CRAN (R 3.2.0)                  
##  rmarkdown    0.9        2015-12-22 CRAN (R 3.2.3)                  
##  scales       0.3.0      2015-08-25 CRAN (R 3.2.0)                  
##  stringi      1.0-1      2015-10-22 CRAN (R 3.2.0)                  
##  stringr      1.0.0      2015-04-30 CRAN (R 3.2.0)                  
##  tidyr      * 0.3.1.9000 2015-12-29 Github (hadley/tidyr@d534fc7)   
##  yaml         2.1.13     2014-06-12 CRAN (R 3.2.0)

title: "03_extract-from-excel-gdpPercap.R" author: "jenny" date: "Tue Dec 29 21:51:53 2015"



YTLogos/gapminder documentation built on May 20, 2019, 1:47 p.m.