This document is relating development steps towards converting JSON data from the UN comtrade database to R data frames.
I have been inspired by several online reading, from package authors and by questions on stackoverflow: Converting a list of data frames into one data frame in R. After unsatisfactory trials with the rjson and RJSONIO packages. The most crucial step was discovering the jsonlite package. jsonlite transforms a json file directly into a list of data frames.
library(knitr) opts_knit$set(root.dir="../..") # file paths are relative to the root of the project directory library(tradeflows) library(RJSONIO) library(jsonlite) # Why are the plyr and dplyr packages not loaded whith treadeflows? # This didn't work before because you need to properly document the function in roxygen # placing @export before the function creation instruction # then create a NAMESPACE file with devtools::document() # Will make the required package autoload. library(plyr) library(dplyr) # Load plyr before dplyr to avoid this message: # ------------------------------------------------------------------------------- # You have loaded plyr after dplyr - this is likely to cause problems. # If you need functions from both plyr and dplyr, please load plyr first, then dplyr: # library(plyr); library(dplyr) # -------------------------------------------------------------------------------
warning("If data-raw/UNCOMTRADE_exp.JSON file missing, look at the unevaluated peace of code above to download the JSON file.") json <- RJSONIO::fromJSON("data-raw/UNCOMTRADE_exp.JSON", nullValue = NA) stopifnot(json$validation$status$name == "Ok") # Check status
# head() keeps only the first few lines swd <- as.data.frame(do.call("rbind", head(json$dataset))) # When I use do.call, the # structure of the data frame looks bizarre, more like a list than a dataframe. str(swd) # Structure of this binded list created by do.call str(do.call("rbind", head(json$dataset)))
# Use a loop to create data frame first then bind them together swd2 <- data.frame() system.time( for (obs in json$dataset){ swd2 <- rbind(swd2,data.frame(obs)) }) str(swd2)
This gives a nice table structure
system.time(swd3 <- ldply(json$dataset, function(l) data.frame(l))) str(swd3)
The structure is still not nice.
system.time(swd4 <- data.frame(t(sapply(head(json$dataset), data.frame)))) str(swd4)
Gives a nice data.frame structure
system.time(json2 <- fromJSON("data-raw/UNCOMTRADE_exp.JSON")) stopifnot(json2$validation$status$name=="Ok") swd5 <- json2$dataset str(swd5)
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.