Man pages for vimc/dettl
Data extract, transform, test and load

add_fk_dataAdd tables with foreign key constraints for testing
add_missing_function_fieldsAdd any missing required fields to the config using defaults...
build_log_dataBuild the data to be written to the import log
convert_bit64_columnsConvert any bit64 type columns to integers.
DataImportData Import
db_connectConnect to the database configured via yaml.
db_get_log_tableGet the log table name from configuration for a particular db
dettlCreate an import object using functions defined at specified...
dettl_auto_loadAutomatic load function.
dettl_configGet db config from path.
dettl_config_read_yamlRead yaml file representing a db config.
dettl_db_argsGet the DB args from config.
dettl_db_create_log_tableInitialise the database by creating log table if it doesn't...
dettl_locate_configLocate the directory containing the config file.
dettl_newCreate new directory and templated code for new dettl...
dettl_run_extractRun extract stage of an import
dettl_run_loadRun up to and including the load stage of an import
dettl_run_transformRun up to and including the transform stage of an import
dettl_saveSave data
expand_wildcardsExpand any wildcards in source file paths.
get_fk_constraintsGet FK constraints
get_postgres_fkGet FK constraints for postgres connection
get_sqlite_fkGet FK constraints for SQLite connection
git_branchGet the current git branch from path.
git_emailGet the git user.email from git dir
git_hashGet the full hash of the current git HEAD
git_repo_is_cleanCheck if git repo is clean
git_root_directoryLocate the git project root directory
git_userGet the git user.name from git dir
insert_into_returningInsert values into table returning the primary key
load_sourcesRead file fields of the dettl config yaml file.
parse_constraintsParse foreign key constraint data frame into list...
parse_sql_dateParse date into POSIXct UTC from SQL db.
postgres_insert_empty_rowInsert an empty row and returned desired columns.
prepare_example_dbPrepare example db
prepare_test_importPrepare example import inside a git repo
read_configGet dettl config from path.
read_function_fieldsRead file fields of the dettl config yaml file.
run_extractRun extract step.
run_loadRun the load step ensuring tests pass before db changes are...
run_transformRun transform step.
setup_dettlSetup dettl import in a tempdir.
sql_dialectGet the SQL dialect used for a connection.
sqlite_enable_fkEnable foreign key constraints for SQLite connections
sqlite_execute_queryExecute sql insert query returning added columns desired...
sqlite_insert_empty_rowInsert an empty row and returned desired columns.
update_child_tablesUpdate child tables using inserted foreign keys
verify_dataVerify the transformed data adhered to the DB schema.
verify_first_runVerify that this is the first time the import has been run.
verify_tableVerify the data adheres to the DB schema.
write_logWrite the log data to the database.
vimc/dettl documentation built on Aug. 30, 2019, 11:45 a.m.