inst/doc/loading.md

In jangorecki/anchormodeling: Anchor Modeling

Design notes for am$load method:

• [ ] loading model unit tests:
• [ ] one flat data
• [ ] two flat data
• [ ] multiple flat data
• [x] ETL template:
• [x] isolate IM - loading anchors/knots ID only
  • [x] lookup anchor PK from local IM table to incoming data
  • [x] generate new PK for new rows
  • [x] write new rows to IM
  • [x] include just im$use()
• [x] isolate single anchor loading: model processing of single anchor load as atomic operation
  • [x] as easy as lapply - easy parallel in future
• [ ] loading process
• [x] batch model vs data validation
• [ ] AM$load
  • [ ] loading of entities
  • [x] anchor
  • [x] attribute
    • [x] static
    • [x] knotted
    • [x] historized
    • [x] knotted historized
  • [x] knot
    • [x] static of attributes
    • [x] historoized of attributes
    • [x] static of ties
    • [x] historized of ties
    • [x] shared knots of attributes
    • [x] dynamic shared knots
    • [x] shared knots of ties
    • [x] dynamic shared knots
    • [x] shared knots of attribute and tie
  • [ ] tie
    • [x] static
    • [x] knotted
    • [x] historized
    • [x] knotted historized
    • [ ] self ties
    • [ ] '2+ anchor' ties
  • [x] unique names length(names(data))==uniqueN(names(data))
  • [x] model-mapping validation
  • [x] multiple exists/valid checks
    • [x] any nested element with "" / NULL / hist name must exists in incoming data
  • [x] surrogate key gen function based on nk
  • [x] check NK (provide by anchor and src cols)
  • [x] exists in incoming data
  • [x] exists in currently using mapping
  • [x] prepare sequence of loading
  • [x] knots
  • [x] anchors
    • [x] attributes
  • [x] ties
  • [x] child classes related validation
  • [x] call $load.AMobj just with a subset of cols
  • [x] rename src cols to tbl cols
  • [x] add metadata id
  • [x] add metadata src + auto (using deparse-substitute on dataset variable name)
  • [x] add metadata user + auto
• [x] AMobj$load
  • [x] validate nrow
  • [x] data types validation - except first load
  • [x] PK and new data value violation for static attributes
  • [x] restatement + idempotency
  • [x] note: does not detect restatements within loading batch, only batch vs history
  • [x] append log
• [x] AMobj$insert
  • [x] rbindlist
  • [x] resetkey
• [ ] rollback based on metadata ID delete from all
• [ ] new method for delete data by meta id from all tables
• [ ] create indexes after insert

jangorecki/anchormodeling documentation built on May 18, 2019, 12:24 p.m.