knitr::opts_chunk$set( collapse = TRUE, comment = "#>", eval = FALSE )
The datacaged package simplifies access to CAGED microdata (Cadastro Geral de Empregados e Desempregados) directly from HuggingFace, loading data into a local DuckDB database for efficient analysis.
It supports three series:
| Period | Series | Database table |
|----------------------|-----------------|-----------------------------------------|
| Jan/2020 – present | Novo CAGED | caged_mov, caged_for, caged_exc |
| Jan/2007 – Dec/2019 | Legacy CAGED | caged_antigo |
| 2002 – Dec/2019 | CAGED Adjustments| caged_ajustes |
# Via remotes remotes::install_github("gecomt/datacaged")
The datacaged package is compatible with Windows, macOS and Linux with no extra configuration for Novo CAGED (2020+).
| Recurso | Windows | macOS | Linux | |---|---|---|---| | Download (HTTPS) | OK | OK | OK | | Downloads paralelos | OK | OK | OK | | Novo CAGED (2020+) | OK | OK | OK | | Legacy CAGED (pre-2020, PPMd) | (!) requires 7-Zip | (!) requires 7-Zip | (!) requires 7-Zip |
Legacy CAGED uses PPMd compression. If you need this data, install 7-Zip:
brew install 7-zip sudo apt install 7zip ou sudo dnf install 7zipNovo CAGED (2020+) uses LZMA, natively supported by the archive package on all platforms.
By default, the package downloads 3 files simultaneously (MOV, FOR and EXC for each month), resulting in approximately 3× faster downloads compared to sequential mode.
# Control the number of workers caged_download(years = 2023, months = 1:3, workers = 3) # padrão # Set globally for the entire session options(datacaged.workers = 4) # Sequential mode (useful for unstable connections) caged_download(years = 2023, months = 1, workers = 1)
The caged_load() function does everything in a single command: downloads .7z files from HuggingFace, extracts, normalises and writes to DuckDB.
library(datacaged) # Download Novo CAGED Jan–Dec/2023 # Novo CAGED: national file, `states` does not filter caged_load( years = 2023, months = seq_len(12L), db_path = "caged.duckdb" )
Progress is displayed in the terminal with a progress bar and final summary.
After populating the database, connect and query with dplyr or plain SQL:
library(dplyr) con <- caged_connect("caged.duckdb") # Monthly employment balance in 2023 saldo_mensal <- tbl(con, "caged_mov") |> group_by(competenciamov) |> summarise(saldo = sum(saldomovimentacao, na.rm = TRUE)) |> arrange(competenciamov) |> collect() saldo_mensal
# Or with direct SQL DBI::dbGetQuery(con, " SELECT competenciamov, uf, SUM(saldomovimentacao) AS saldo, AVG(salario) AS salario_medio, COUNT(*) AS movimentacoes FROM caged_mov WHERE uf = 35 -- Sao Paulo GROUP BY competenciamov, uf ORDER BY competenciamov ")
Always close the connection when done:
DBI::dbDisconnect(con, shutdown = TRUE)
For more control, use the functions individually:
# Download and save to local cache (~/.local/share/R/datacaged por padrão) manifest <- caged_download( years = 2023, months = c(1L, 2L, 3L), destdir = "~/meus_dados/caged_cache" ) # manifest is a data.frame with the status of each file dplyr::count(manifest, status)
# One file at a time df <- caged_parse("~/meus_dados/caged_cache/caged_mov/2023/CAGEDMOV202301.7z") glimpse(df) # Several at once arquivos <- list.files( "~/meus_dados/caged_cache/NOVO_CAGED/2023", pattern = "CAGEDMOV", full.names = TRUE ) df_todos <- caged_parse_batch(arquivos)
caged_to_duckdb(df_todos, db_path = "caged.duckdb")
caged_info("caged.duckdb") #> ── datacaged - 'caged.duckdb' ───────────────────────────────── #> File size: 142.3 MB #> #> ── Tables ── #> #> • "caged_mov" #> Records : 3,665,155 #> Periods : 202301 - 202312 #> • "caged_for" #> Records : 91,098 #> Periods : 202301 - 202312 #> • "caged_exc" #> Records : 7,900 #> Periods : 202301 - 202312
# Baixa Legacy CAGED para Nordeste (2015–2019) nordeste <- c("MA", "PI", "CE", "RN", "PB", "PE", "AL", "SE", "BA") caged_load( years = 2015:2019, db_path = "caged_historico.duckdb" ) con <- caged_connect("caged_historico.duckdb") # Evolução anual do saldo formal no Nordeste tbl(con, "caged_antigo") |> mutate(ano = as.integer(substr(as.character(competencia), 1, 4))) |> group_by(ano, uf) |> summarise(saldo = sum(saldomovimentacao, na.rm = TRUE)) |> collect() |> tidyr::pivot_wider(names_from = uf, values_from = saldo) DBI::dbDisconnect(con, shutdown = TRUE)
CAGED Adjustments contain retroactive corrections to legacy CAGED records (up to 2019).
Use caged_adjustments_load() to download and write to the caged_ajustes table.
# Baixar ajustes de 2019 caged_adjustments_load(years = 2019, months = seq_len(12L), db_path = "caged.duckdb") # Listar o que está disponível no HuggingFace caged_hf_files(type = "ajustes") # Comparar saldo original vs ajustado con <- caged_connect("caged.duckdb") antigo <- dplyr::tbl(con, "caged_antigo") |> dplyr::group_by(competencia) |> dplyr::summarise(saldo_original = sum(saldomovimentacao, na.rm = TRUE)) ajustes <- dplyr::tbl(con, "caged_ajustes") |> dplyr::group_by(competencia) |> dplyr::summarise(saldo_ajuste = sum(saldomovimentacao, na.rm = TRUE)) dplyr::full_join(antigo, ajustes, by = "competencia") |> dplyr::mutate(saldo_final = saldo_original + saldo_ajuste) |> dplyr::collect() DBI::dbDisconnect(con, shutdown = TRUE)
select() before collect() to bring only the necessary columns.caged_load() skips competencies already in the database by default. Use overwrite_competencies = TRUE to overwrite..7z cache: downloaded files are stored in ~/.local/share/R/datacaged. They can be reused without a new HuggingFace connection.# Verificar se o HuggingFace está online antes de baixar caged_status() # Listar competências disponíveis no HuggingFace caged_hf_files() # Novo CAGED (últimos 12 meses) caged_hf_files(type = "antigo") # Legacy CAGED caged_hf_files(type = "ajustes") # CAGED Adjustments # Atualização incremental — baixa apenas o que ainda não está no banco caged_update(db_path = "caged.duckdb") caged_update(db_path = "caged.duckdb", series = c("novo", "antigo")) # Exportar tabelas para Parquet (nativo DuckDB, muito rápido) caged_to_parquet("caged.duckdb", output_dir = "~/exports") caged_to_parquet("caged.duckdb", output_dir = "~/exports", tables = "caged_mov", partition_by = "uf")
| Column | Description |
|---|---|
| competenciamov | Competency in Novo CAGED, YYYYMM format (ex: 202301) |
| competencia | Competency in legacy CAGED and Adjustments, format AAAAMM |
| uf | IBGE state code (ex: 35 = SP) |
| municipio | IBGE municipality code |
| saldomovimentacao | +1 hire, -1 dismissal |
| salario | Contracted wage in BRL |
| sexo | 1 male, 3 female |
| idade | Age in years |
| escolaridade | Education level code (1–9) |
| racacor | Race/colour code (1–5) |
| tipomovimentacao | Reason for movement code |
| secao | CNAE 2.0 section (Novo CAGED) |
| fonte_tipo | MOV, FOR, EXC, ANTIGO or AJUSTES |
Any scripts or data that you put into this service are public.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.