Large files, raw microdata and provenance"

In datasus: An Interface to Brazilian Public Health Data

knitr::opts_chunk$set(echo = TRUE, collapse = TRUE, comment = "#>")
library(datasus)

Overview

Record-level health data can be much larger than TABNET aggregates. The package provides two complementary workflows:

• DBC/DBF microdata from the traditional DATASUS file servers;
• CSV and JSON resources published through OpenDataSUS.

Both workflows support discovery, local caching, selected columns, standardized schemas and provenance.

Traditional DBC/DBF microdata

The raw microdata catalog covers SIM, SINASC and SIH/SUS:

microdados_catalogo()

microdados_arquivos(
  sistema = "sih",
  ano = 2024,
  mes = 1,
  uf = "AC"
)

Select only the columns required by the analysis and start with a small number of records:

admissions <- sih_microdados(
  ano = 2024,
  mes = 1,
  uf = "AC",
  colunas = c(
    "MUNIC_RES", "DT_INTER", "DIAG_PRINC", "VAL_TOT"
  ),
  n_max = 1000,
  normalizar = TRUE
)

deaths <- sim_microdados(
  ano = 2023,
  uf = "RR",
  colunas = c("CODMUNRES", "DTOBITO", "CAUSABAS"),
  n_max = 1000,
  normalizar = TRUE
)

births <- sinasc_microdados(
  ano = 2023,
  uf = "RR",
  colunas = c("CODMUNRES", "DTNASC", "SEXO", "PESO"),
  n_max = 1000,
  normalizar = TRUE
)

DBC files are decoded directly and read through the same public interface as DBF files.

Inspect schemas before reading

head(datasus_dicionario("sih"), 10)

After a read, validate the critical analysis fields:

datasus_validar_esquema(
  admissions,
  sistema = "sih",
  campos = c(
    "codigo_municipio_residencia",
    "data_internacao",
    "diagnostico_principal_cid10",
    "valor_total"
  ),
  estrito = TRUE
)

Selected OpenDataSUS columns

For ordinary files, opendatasus_ler() can select columns while parsing:

resources <- opendatasus_recursos("arboviroses-dengue")
csv_id <- resources$id[resources$formato == "CSV"][1]

sample <- opendatasus_ler(
  "arboviroses-dengue",
  recurso = csv_id,
  colunas = c("DT_NOTIFIC", "SG_UF", "ID_MUNICIP"),
  n_max = 1000
)

Convenience wrappers should be preferred when available because they encode the dataset's partition rules and curated schema.

Process CSV files in bounded memory

opendatasus_processar() calls a function for each block instead of retaining the entire dataset:

resources <- opendatasus_recursos(
  "notificacoes-de-sindrome-gripal-leve-2020"
)
ms_id <- resources$id[
  resources$formato == "CSV" & grepl("^Dados MS", resources$nome)
][1]

processed <- opendatasus_processar(
  "notificacoes-de-sindrome-gripal-leve-2020",
  recurso = ms_id,
  ano = NULL,
  colunas = c("municipioIBGE", "resultadoTeste"),
  tamanho_bloco = 50000,
  sistema = "sindrome_gripal",
  FUN = function(dados, posicao, arquivo) {
    data.frame(
      arquivo = arquivo,
      bloco_inicial = posicao,
      registros = nrow(dados),
      positivos = sum(
        dados$resultado_teste == "Positivo",
        na.rm = TRUE
      )
    )
  }
)

processed$linhas
processed$blocos
processed$resultados

The callback receives the standardized block when sistema is supplied. Physical parts are processed in catalog order and are listed in processed$arquivos.

Cache and provenance

Downloads are written atomically. Cached files are reused unless atualizar = TRUE is requested:

first <- ocupacao_hospitalar(
  ano = 2022,
  n_max = 1000,
  cache = TRUE
)

source <- datasus_proveniencia(first)
str(source)

The provenance record identifies the official URL, resource, portal update time, local path, download time and MD5 checksum. Multipart reads retain a record for each physical file.

Practical strategy

For large resources:

1. inspect resource metadata and partitions;
2. select the smallest useful year, month and state;
3. provide colunas before increasing n_max;
4. use opendatasus_processar() for reductions that do not need all rows in memory;
5. validate critical fields with datasus_validar_esquema();
6. retain provenance with the analytical output.

n_max limits parsed rows, but a remote file may still need to be downloaded in full before parsing. Cache reuse prevents that transfer from being repeated.

datasus documentation built on Oct. 7, 2026, 1:07 a.m.