In greatgray/scorecard: GGCM Model Performance Scorecard

The 'scorecard' package supports automated update and production of GGCM model performance scorecards. This vignette describes the essential elements to run the scorecard workflow manually, and can be used to configure the data files for automated analysis.

Scorecard File

The scorecard file is a layout specification for the final product, used to both update and report on the scorecard. It is a YAML-format file, usually called scorecard.yml, passed into the update function. The table layout includes four sections: activated, candidate, deactivated, and retired. Each section includes zero or more model specifications to be evaluated within that category.

version: 0.1
vintage: "2017-01-14"

table:
 activated: 
 - id: "3ROC 13"
   partner: "MRB"
   config: "model_roc13.yml"
   live: "2016-08-01"
 - id: "RSO SM20"
   partner: "STN"
   config: "model_rso20.yaml"
   live: "2016-09-14"
 candidate: 
 - id: "3ROC 04"
   partner: "MRB"
   config: "model_roc04.yml"
 - id: "3ROC 10"
   partner: "MRB"
   config: "model_roc10.yml"
 deactivated:
 - id: "3ROC 02"
   partner: "MRB"
   config: "model_roc02.yml"
 retired: 
 - id: NA

Each category has some effect on the update work performed for each model specification within the category:

1. activated models require the id, partner, config and live values, and the update includes buy-hold performance for the model basket, active performance computed from Bivio transactions, out-of-sample performance for the model, and scorecard performance metrics rankings together with the candidate models.
2. candidate models require the id, partner, config and live values, and the update includes buy-hold performance for the model basket, out-of-sample performance for the model, and scorecard performance metrics rankings together with the activated models. These do not include actual performance updates because they refer to models that are not being traded within the portfolio.
3. deactivated models require the id, partner and config values, and the update includes buy-hold performance for the model basket, and out-of-sample performance for the model. These do not include scorecard performance metrics rankings or actual transaction performance.
4. retired models require an id but no other specification values. These perform no performance update or ranking.

Model Configuration Files

The model specification record in the scorecard file includes a config attribute. The value of the attribute is the name of a file containing the model configuration in YAML format. A model configuration is unique to the model update function (backtested trading algorithm), so various configuration files may have different attributes. A typical file may appear as follows:

# 3ROC model config
model: "3ROC13"
version: "0.1"
config:
  periods: [1,2,3]
  weights: [0.4,0.3,0.3]
  rebalance: months
  trailstop: 0.0
  stretch: 1
  weightmode: monthly.average
  topn: 3
  basket: [XLY, XLP, XLE, XLK, XLU, XLF, AGG, TLT, GLD]
  function: model_update_rm_3roc
backtest:
  start: 2004-12-01
  stop: 2016-06-16 # out-of-sample start
  initeq: 1000000
  transaction: -4.95
  benchmark: SPY
  cagr: 17.8
  mdd: 29.5
  sortino: 0.0 # unknown
  calmar: 0.0 # unknown

The function attribute of the config section refers to an R function name implementing the trading algorithm and some pre-packaged reporting. The update function will call the R function name during the update stage in order to compute out-of-sample analysis. The update function will pass the model configuration to the function in order to provide details for generic algorithm functions.

Workflow

The manual workflow requires just two steps: update and produce. These share a workspace archive file written by the update function. If a specific workspace is not provided to the produce function, then it uses the latest dated workspace file name found in the scorecard directory.

The following R code implements an update given the transaction history and scorecard files:

output_dir <- '/users/johndoe/models'
scorecard_file <- '/users/johndoe/scorecard.yml'
transaction_file <- '/users/johndoe/transaction.csv'

# check the integrity of the scorecard definition
# additional detail will be in the rv list
rv <- scorecard_precheck(scorecard_file)
if ( ! rv$valid ) {
  stop(rv$message)
}

# check the integrity of the transaction record
# additional detail will be in the rv list
rv <- scorecard_check(scorecard_file, transaction_file)
if ( ! rv$valid ) {
  stop(rv$message)
}

# call the update function
# results will be written to <output_dir>/<today>_scorecard_workspace.RData
scorecard_update(output_dir,scorecard_file,transaction_file)

# call the produce function
# this one produces four PNG files for various scorecards
# workspace read from <output_dir>/<today>_scorecard_workspace.RData
# results will be written to <output_dir>/<today>_sc_*.png
scorecard_produce(output_dir)

greatgray/scorecard documentation built on May 17, 2019, 8:34 a.m.