get_eurostat_data: Download, extract and filter Eurostat data
In restatapi: Search and Retrieve Data from Eurostat Database
View source: R/get_eurostat_data.R
get_eurostat_data R Documentation
Download, extract and filter Eurostat data
Description
Download full or partial data set from Eurostat database.
Usage
get_eurostat_data(
id,
filters = NULL,
exact_match = TRUE,
date_filter = NULL,
label = FALSE,
select_freq = NULL,
cache = TRUE,
update_cache = FALSE,
cache_dir = NULL,
compress_file = TRUE,
stringsAsFactors = TRUE,
keep_flags = FALSE,
cflags = FALSE,
check_toc = FALSE,
local_filter = TRUE,
force_local_filter = FALSE,
mode = "xml",
verbose = FALSE,
...
)
Arguments
id
A code name for the dataset of interest.
See search_eurostat_toc for details how to get an id.
filters
a string, a character vector or named list containing words to filter by the different concepts or geographical location.
If filter applied only part of the dataset is downloaded through the API. The words can be
any word, Eurostat variable code, and value which are in the DSD search_eurostat_dsd.
If a named list is used, then the name of the list elements should be the concepts from the DSD and the provided values will be used to filter the dataset for the given concept.
The default is NULL, in this case the whole dataset is returned via the bulk download. To filter by time see date_filter below.
If after filtering still the dataset has more observations than the limit per query via the API, then the raw download is used to retrieve the whole dataset and apply the filter on the local computer. This option can be disabled with the local_filter=FALSE parameter.
exact_match
a boolean with the default value TRUE, if the strings provided in filters shall be matched exactly as it is or as a pattern.
date_filter
a vector which can be numeric or character containing dates to filter the dataset. If date is defined as character string it should follow the format yyyy[-mm][-dd], where the month and the day part is optional.
If date filter applied only part of the dataset is downloaded through the API.
The default is NULL, in this case the whole dataset is returned via the bulk download.
If after filtering still the dataset has more observations than the limit per query via the API, then the raw download is used to retrieve the data and apply the filter on the local computer.
This option can be disabled with the local_filter=FALSE parameter.
label
a boolean with the default FALSE. If it is TRUE then the code values are replaced by the name from the Data Structure Definition (DSD) get_eurostat_dsd.
For example instead of "D1110A", "Raw cows' milk from farmtype" is used or "HU32" is replaced by "Észak-Alföld".
select_freq
a character symbol for a time frequency when a dataset has multiple time
frequencies. Possible values are:
A = annual, S = semi-annual, H = half-year, Q = quarterly, M = monthly, W = weekly, D = daily.
The default is NULL as most datasets have just one time
frequency and in case there are multiple frequencies, then only the most common frequency kept.
If all the frequencies needed the get_eurostat_raw can be used.
cache
a logical whether to do caching. Default is TRUE. Affects
only queries without filtering. If filters or date_filter is used then there is no caching.
update_cache
a logical with a default value FALSE, whether to update the data in the cache. Can be set also with
options(restatapi_update=TRUE)
cache_dir
a path to a cache directory. The NULL (default) uses the memory as cache.
If the folder cache_dir directory does not exist it saves in the 'restatapi' directory
under the temporary directory from tempdir(). Directory can also be set with
option(restatapi_cache_dir=...).
compress_file
a logical whether to compress the
RDS-file in caching. Default is TRUE.
stringsAsFactors
if TRUE (the default) the non-numeric columns are
converted to factors. If the value FALSE they are returned as characters.
keep_flags
a logical whether the observation status (flags) - e.g. "confidential",
"provisional", etc. - should be kept in a separate column or if they
can be removed. Default is FALSE. For flag values see:
https://ec.europa.eu/eurostat/web/main/data/database/information.
cflags
a logical whether the missing observations with flag 'c' - "confidential"
should be kept or not. Default is FALSE, in this case these observations dropped from the dataset. If this parameter
TRUE then the flags are kept and the parameter provided in keep_flags is not taken into account.
check_toc
a boolean whether to check the provided id in the Table of Contents (TOC) or not. The default value
FALSE, in this case the base URL for the download link is retrieved from the configuration file.
If the value is TRUE then the TOC is downloaded and the id is checked in it. If it found then the download link
is retrieved form the TOC.
local_filter
a boolean whether do the filtering on the local computer or not in case after filtering still the dataset has more observations
than the limit per query via the API would allow to download. The default is TRUE, in this case if the response footer contains information
that the result cannot be downloaded becuase it is too large, then the whole raw dataset is downloaded and filtered on the local computer.
force_local_filter
a boolean with the default value FALSE. In case, if there are existing filter conditions, then it will do the filtering on the local
computer and not requesting through the REST API. It can be useful, if the values are not numeric as these are provided as NaN (Not a Number) through the REST API,
but it is fully listed in the raw dataset.
mode
defines the format of the dataset response from the API. It can be
csv for SDMX-CSV or xml for the SDMX-ML version.
verbose
A boolean with default FALSE, so detailed messages (for debugging) will not printed.
Can be set also with options(restatapi_verbose=TRUE)
...
further arguments to the for search_eurostat_dsd function, e.g.: ignore.case or name.
The ignore.case has the default value FALSE, then the strings provided in filters are matched as is,
otherwise the case of the letters is ignored. If the name=FALSE then the pattern(s) provided in the filters
argument is only searched in the code column of the DSD, and the names of the codes will not be searched.
Details
Data sets are downloaded from the Eurostat Web Services
SDMX API if there is a filter otherwise the
the Eurostat bulk download facility is used.
If only the table id is given, the whole table is downloaded from the
bulk download facility. If also filters or date_filter is defined then the SDMX REST API is
used. In case after filtering the dataset has more rows than the limitation of the SDMX REST API (1 million values at one time) then the bulk download is used to retrieve the whole dataset .
By default all datasets cached as they are often rather large.
The datasets cached in memory (default) or can be stored in a temporary directory if cache_dir or option(restatpi_cache_dir) is defined.
The cache can be emptied with clean_restatapi_cache.
The id, is a value from the code column of the table of contents (get_eurostat_toc), and can be searched
for with the search_eurostat_toc function. The id value can be retrieved from the Eurostat database
as well. The Eurostat database gives codes in the Data Navigation Tree after every dataset in parenthesis.
Filtering can be done by the codes as described in the API documentation providing in the correct order and connecting with "." and "+".
If we do not know the codes we can filter based on words or by the mix of the two putting in a vector like c("AT$","Belgium","persons","Total").
Be careful that the filter is case sensitive, if you do not know the code or label exactly you can use the option ignore.case=TRUE and exact_match=FALSE,
but in this case the results may include unwanted elements as well. In the filters parameter regular expressions can be used as well.
We do not have to worry about the correct order of the filter, it will be put in the correct place based on the DSD.
The date_filter shall be a string in the format yyyy[-mm][-dd]. The month and the day part is optional, but if we use the years and we have monthly frequency then all the data for the given year is retrieved.
The string can be extended by adding the "<" or ">" to the beginning or to the end of the string. In this case the date filter is treated as range, and the date is used as a starting or end date. The data will include the observation of the start/end date.
A single date range can be defined as well by concatenating two dates with the ":", e.g. "2016-08:2017-03-15". As seen in the example the dates can have different length: one defined only at year/month level, the other by day level.
If a date range is defined with ":", it is not possible to use the "<" or ">" characters in the date filter.
If there are multiple dates which is not a continuous range, it can be put in vector in any order like c("2016-08",2013:2015,"2017-07-01"). In this case, as well, it is not possible to use the "<" or ">" characters.
Value
a data.table with the following columns:
freq A column for the frequency of the data in case there are multiple frequencies, for
single frequency this columns is dropped from the data table
dimension names One column for each dimension in the data
time A column for the time dimension
values A column for numerical values
flags A column for flags if the keep_flags=TRUE or cflags=TRUE otherwise this column
is not included in the data table
The data.table does not include all missing values. The missing values are dropped if the value and flag are missing
on a particular time.
In case the provided filters can be found in the DSD, then it is used to query the API or applied locally. If the applied filters with combination of date_filter
and select_freq has no observation in the data set then the fucntion returns the data.table with 0 row.
In case none of the provided filters, date_filter or select_freq can be parsed or found in the DSD then the whole dataset downloaded through the bulk download with a warning message.
In case the id is not exist then the function returns the value NULL.
See Also
search_eurostat_toc, search_eurostat_dsd, get_eurostat_bulk
Examples
load_cfg()
eu<-get("cc",envir=restatapi::.restatapi_env)
options(timeout=2)
dt<-get_eurostat_data("NAMA_10_GDP")
dt<-get_eurostat_data("htec_cis3",update_cache=TRUE,check_toc=TRUE)
dt<-get_eurostat_data("agr_r_milkpr",cache_dir="/tmp",cflags=TRUE)
options(restatapi_update=FALSE)
options(restatapi_cache_dir=file.path(tempdir(),"restatapi"))
dt<-get_eurostat_data("avia_gonc",select_freq="A",cache=FALSE)
dt<-get_eurostat_data("agr_r_milkpr",date_filter=2008,keep_flags=TRUE)
dt<-get_eurostat_data("avia_par_me",
filters="BE$",
exact_match=FALSE,
date_filter=c(2016,"2017-03","2017-07-01"),
select_freq="Q",
label=TRUE,
name=FALSE)
dt<-get_eurostat_data("agr_r_milkpr",
filters=c("BE$","Hungary"),
date_filter="2007-06<",
keep_flags=TRUE)
dt<-get_eurostat_data("nama_10_a10_e",
filters=c("Annual","EU28","Belgium","AT","Total","EMP_DC","person"),
date_filter=c("2008",2002,2013:2018))
dt<-get_eurostat_data("vit_t3",
filters=c("EU28",eu$EA15,"HU$"),
date_filter=c("2015",2007))
dt<-get_eurostat_data("avia_par_me",
filters="Q...ME_LYPG_HU_LHBP+ME_LYTV_UA_UKKK",
date_filter=c("2016-08","2017-07-01"),
select_freq="M")
dt<-get_eurostat_data("htec_cis3",
filters="lu",
ignore.case=TRUE)
dt<-get_eurostat_data("bop_its6_det",
filters=list(bop_item="SC",
currency="MIO_EUR",
partner="EXT_EU28",
geo=c("EU28","HU"),
stk_flow="BAL"),
date_filter="2010:2012",
select_freq="A",
label=TRUE,
name=FALSE)
clean_restatapi_cache("/tmp",verbose=TRUE)
options(timeout=60)
restatapi documentation built on Sept. 19, 2023, 5:08 p.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
View source: R/get_eurostat_data.R
| get_eurostat_data | R Documentation |
Download, extract and filter Eurostat data
Description
Download full or partial data set from Eurostat database.
Usage
get_eurostat_data(
id,
filters = NULL,
exact_match = TRUE,
date_filter = NULL,
label = FALSE,
select_freq = NULL,
cache = TRUE,
update_cache = FALSE,
cache_dir = NULL,
compress_file = TRUE,
stringsAsFactors = TRUE,
keep_flags = FALSE,
cflags = FALSE,
check_toc = FALSE,
local_filter = TRUE,
force_local_filter = FALSE,
mode = "xml",
verbose = FALSE,
...
)
Arguments
id |
A code name for the dataset of interest.
See |
filters |
a string, a character vector or named list containing words to filter by the different concepts or geographical location.
If filter applied only part of the dataset is downloaded through the API. The words can be
any word, Eurostat variable code, and value which are in the DSD |
exact_match |
a boolean with the default value |
date_filter |
a vector which can be numeric or character containing dates to filter the dataset. If date is defined as character string it should follow the format yyyy[-mm][-dd], where the month and the day part is optional.
If date filter applied only part of the dataset is downloaded through the API.
The default is |
label |
a boolean with the default |
select_freq |
a character symbol for a time frequency when a dataset has multiple time
frequencies. Possible values are:
A = annual, S = semi-annual, H = half-year, Q = quarterly, M = monthly, W = weekly, D = daily.
The default is |
cache |
a logical whether to do caching. Default is |
update_cache |
a logical with a default value |
cache_dir |
a path to a cache directory. The |
compress_file |
a logical whether to compress the
RDS-file in caching. Default is |
stringsAsFactors |
if |
keep_flags |
a logical whether the observation status (flags) - e.g. "confidential",
"provisional", etc. - should be kept in a separate column or if they
can be removed. Default is |
cflags |
a logical whether the missing observations with flag 'c' - "confidential"
should be kept or not. Default is |
check_toc |
a boolean whether to check the provided |
local_filter |
a boolean whether do the filtering on the local computer or not in case after filtering still the dataset has more observations
than the limit per query via the API would allow to download. The default is |
force_local_filter |
a boolean with the default value |
mode |
defines the format of the dataset response from the API. It can be
|
verbose |
A boolean with default |
... |
further arguments to the for |
Details
Data sets are downloaded from the Eurostat Web Services
SDMX API if there is a filter otherwise the
the Eurostat bulk download facility is used.
If only the table id is given, the whole table is downloaded from the
bulk download facility. If also filters or date_filter is defined then the SDMX REST API is
used. In case after filtering the dataset has more rows than the limitation of the SDMX REST API (1 million values at one time) then the bulk download is used to retrieve the whole dataset .
By default all datasets cached as they are often rather large.
The datasets cached in memory (default) or can be stored in a temporary directory if cache_dir or option(restatpi_cache_dir) is defined.
The cache can be emptied with clean_restatapi_cache.
The id, is a value from the code column of the table of contents (get_eurostat_toc), and can be searched
for with the search_eurostat_toc function. The id value can be retrieved from the Eurostat database
as well. The Eurostat database gives codes in the Data Navigation Tree after every dataset in parenthesis.
Filtering can be done by the codes as described in the API documentation providing in the correct order and connecting with "." and "+".
If we do not know the codes we can filter based on words or by the mix of the two putting in a vector like c("AT$","Belgium","persons","Total").
Be careful that the filter is case sensitive, if you do not know the code or label exactly you can use the option ignore.case=TRUE and exact_match=FALSE,
but in this case the results may include unwanted elements as well. In the filters parameter regular expressions can be used as well.
We do not have to worry about the correct order of the filter, it will be put in the correct place based on the DSD.
The date_filter shall be a string in the format yyyy[-mm][-dd]. The month and the day part is optional, but if we use the years and we have monthly frequency then all the data for the given year is retrieved.
The string can be extended by adding the "<" or ">" to the beginning or to the end of the string. In this case the date filter is treated as range, and the date is used as a starting or end date. The data will include the observation of the start/end date.
A single date range can be defined as well by concatenating two dates with the ":", e.g. "2016-08:2017-03-15". As seen in the example the dates can have different length: one defined only at year/month level, the other by day level.
If a date range is defined with ":", it is not possible to use the "<" or ">" characters in the date filter.
If there are multiple dates which is not a continuous range, it can be put in vector in any order like c("2016-08",2013:2015,"2017-07-01"). In this case, as well, it is not possible to use the "<" or ">" characters.
Value
a data.table with the following columns:
freq | A column for the frequency of the data in case there are multiple frequencies, for single frequency this columns is dropped from the data table |
| dimension names | One column for each dimension in the data |
time | A column for the time dimension |
values | A column for numerical values |
flags | A column for flags if the keep_flags=TRUE or cflags=TRUE otherwise this column
is not included in the data table
|
The data.table does not include all missing values. The missing values are dropped if the value and flag are missing on a particular time.
In case the provided filters can be found in the DSD, then it is used to query the API or applied locally. If the applied filters with combination of date_filter
and select_freq has no observation in the data set then the fucntion returns the data.table with 0 row.
In case none of the provided filters, date_filter or select_freq can be parsed or found in the DSD then the whole dataset downloaded through the bulk download with a warning message.
In case the id is not exist then the function returns the value NULL.
See Also
search_eurostat_toc, search_eurostat_dsd, get_eurostat_bulk
Examples
load_cfg()
eu<-get("cc",envir=restatapi::.restatapi_env)
options(timeout=2)
dt<-get_eurostat_data("NAMA_10_GDP")
dt<-get_eurostat_data("htec_cis3",update_cache=TRUE,check_toc=TRUE)
dt<-get_eurostat_data("agr_r_milkpr",cache_dir="/tmp",cflags=TRUE)
options(restatapi_update=FALSE)
options(restatapi_cache_dir=file.path(tempdir(),"restatapi"))
dt<-get_eurostat_data("avia_gonc",select_freq="A",cache=FALSE)
dt<-get_eurostat_data("agr_r_milkpr",date_filter=2008,keep_flags=TRUE)
dt<-get_eurostat_data("avia_par_me",
filters="BE$",
exact_match=FALSE,
date_filter=c(2016,"2017-03","2017-07-01"),
select_freq="Q",
label=TRUE,
name=FALSE)
dt<-get_eurostat_data("agr_r_milkpr",
filters=c("BE$","Hungary"),
date_filter="2007-06<",
keep_flags=TRUE)
dt<-get_eurostat_data("nama_10_a10_e",
filters=c("Annual","EU28","Belgium","AT","Total","EMP_DC","person"),
date_filter=c("2008",2002,2013:2018))
dt<-get_eurostat_data("vit_t3",
filters=c("EU28",eu$EA15,"HU$"),
date_filter=c("2015",2007))
dt<-get_eurostat_data("avia_par_me",
filters="Q...ME_LYPG_HU_LHBP+ME_LYTV_UA_UKKK",
date_filter=c("2016-08","2017-07-01"),
select_freq="M")
dt<-get_eurostat_data("htec_cis3",
filters="lu",
ignore.case=TRUE)
dt<-get_eurostat_data("bop_its6_det",
filters=list(bop_item="SC",
currency="MIO_EUR",
partner="EXT_EU28",
geo=c("EU28","HU"),
stk_flow="BAL"),
date_filter="2010:2012",
select_freq="A",
label=TRUE,
name=FALSE)
clean_restatapi_cache("/tmp",verbose=TRUE)
options(timeout=60)
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.