Import: Generic import object.

In vimc/dettl: Data Extract, Transform, Test and Load

Generic import object.

Description

Generic import object.

Generic import object.

Details

This shouldn't be initialised directly - this object manages transactions and the db connection and should be extended to implement actual imports.

Public fields

path

Path to directory containing import object

Methods

Public methods

• Import$get_connection()

• Import$begin_transaction()

• Import$rollback_transaction()

• Import$commit_transaction()

• Import$get_log_table()

• Import$new()

• Import$reload()

• Import$read_config()

• Import$extract()

• Import$transform()

• Import$pre_modify_checks()

• Import$load()

• Import$run_import()

• Import$format()

• Import$help()

---

Method get_connection()

Tidy up open connections, rolling back any active transactions

Get the database connection being used by the import. Used for testing.

Usage

Import$get_connection()

Returns

The DBI connection

---

Method begin_transaction()

Start a transaction

Usage

Import$begin_transaction()

---

Method rollback_transaction()

Rollback a transaction

Usage

Import$rollback_transaction()

---

Method commit_transaction()

Commit a transaction

Usage

Import$commit_transaction()

---

Method get_log_table()

Get the name of the log table for a particular import. This is the name of the log table configured in the dettl config

Usage

Import$get_log_table()

Returns

Log table name

---

Method new()

Create Import object - shouldn't be called directly. Use dettl to create the object

Usage

Import$new(path, db_name)

Arguments

path

Path to directory containing import object

db_name

Database from dettl config to create import object for

Returns

A new 'Import' object

---

Method reload()

Reload the objects sources to refresh source code or repair a broken Postgres connection.

Usage

Import$reload()

---

Method read_config()

Abstract impl should be overridden by subclass

Usage

Import$read_config()

---

Method extract()

Run the extract stage of the data import - does nothing for generic override in subclass if required.

Usage

Import$extract()

---

Method transform()

Run the extract stage of the data import - does nothing for generic override in subclass if required.

Usage

Import$transform()

---

Method pre_modify_checks()

Run checks that db can be modified, this checks that: * If require_branch set in cfg that import is for that branch * If confirm TRUE asks users to confirm action will modify db * If git is dirty, checks that user has explicitly said that is okay

Usage

Import$pre_modify_checks(dry_run, allow_dirty_git)

Arguments

dry_run

Whether to run in dry run mode. If TRUE then any database changes will be rolled back. Defaults to FALSE.

allow_dirty_git

If TRUE then skips check that the import is up to date with remote git repo. FALSE by default.

---

Method load()

Run the import step ensuring tests pass before db changes are committed.

Runs the import on the DB within a transaction. Then run a set of tests on the DB and rollback the changes if any should fail.

Usage

Import$load(comment = NULL, dry_run = FALSE, allow_dirty_git = FALSE)

Arguments

comment

An optional comment to add to the import log table for this run.

dry_run

Whether to run in dry run mode. If TRUE then any database changes will be rolled back. Defaults to FALSE.

allow_dirty_git

If TRUE then skips check that the import is up to date with remote git repo. FALSE by default.

---

Method run_import()

For generic import this is just running the load step, expect this gets overridden in subclasses. .

Usage

Import$run_import(
  comment = NULL,
  dry_run = FALSE,
  allow_dirty_git = FALSE,
  stage = c("extract", "transorm")
)

Arguments

comment

An optional comment to add to the import log table for this run.

dry_run

Whether to run in dry run mode. If TRUE then any database changes will be rolled back. Defaults to FALSE.

allow_dirty_git

If TRUE then skips check that the import is up to date with remote git repo. FALSE by default.

stage

The stage or stages of the import to be run.

---

Method format()

Custom formatter for pretty printing object summary.

Usage

Import$format(brief = FALSE)

Arguments

brief

If TRUE then print a brief summary.

---

Method help()

Print help page for the object

Usage

Import$help()

vimc/dettl documentation built on Oct. 6, 2022, 2:13 p.m.