modelPerformance: Evaluate the performance of a model
In dnegrey/miscTools: R package with miscellaneous tools

Description Usage Arguments Value See Also Examples

modelPerformance is a wrapper function that runs various model evaluation techniques and returns the results in a convenient list object. Parameters prefaced with a function dot (i.e. gainsChartDT.) are only applicable to the use of that function.

modelPerformance(y, yhat, y2 = NULL, v = NULL, numBins = 10,
  gainsChartDT.scoreFormatType = "int", gainsChartDT.scoreFormatDigits = 5,
  gainsChartDT.contSumFormatType = "dlr",
  gainsChartDT.contSumFormatDigits = 0,
  gainsChartDT.contAvgFormatType = "dlr",
  gainsChartDT.contAvgFormatDigits = 0, gainsChartDT.KScolor = "#FF3005",
  gainsChartBarGraph.cb = "#009DDC", gainsChartBarGraph.cv = "#77BF30",
  gainsChartBarGraph.xLabel = "Model Quantile",
  gainsChartBarGraph.yLabel = "Mean Outcome",
  gainsChartBarGraph.yType = ifelse(is.null(y2), "pct", "dlr"),
  gainsChartBarGraph.yDigits = ifelse(is.null(y2), 1, 0),
  gainsChartPerfCurve.cb = "#009DDC", gainsChartPerfCurve.cv = "#77BF30",
  gainsChartPerfCurve.cr = "#666666",
  gainsChartPerfCurve.xLabel = "Cumulative Total Percent",
  gainsChartPerfCurve.yLabel = "Cumulative Outcome Percent",
  variableImportanceGraph.x = NULL, variableImportanceGraph.y = NULL,
  variableImportanceGraph.sumYcheck = TRUE,
  variableImportanceGraph.barColor = "rgba(0, 55, 82, 0.6)",
  variableImportanceGraph.lineColor = "#003752",
  variableImportanceGraph.lineWidth = 2)

`y`	logical, integer or numeric vector (dependent variable)
`yhat`	numeric vector (predicted values of `y`)
`y2`	logical, integer or numeric vector (other dependent variable for combined models)
`v`	logical, integer or numeric vector of binary values (distinguishes validate (TRUE) vs. build (FALSE) observations)
`numBins`	integer value >= 2; number of desired bins
`gainsChartDT.scoreFormatType`	character string; format type for score field(s); valid values are "int", "dlr" and "pct"
`gainsChartDT.scoreFormatDigits`	non-negative integer value; number of decimal places for score field(s)
`gainsChartDT.contSumFormatType`	character string; format type for continuous sum field(s); valid values are "int", "dlr" and "pct"
`gainsChartDT.contSumFormatDigits`	non-negative integer value; number of decimal places for continuous sum field(s)
`gainsChartDT.contAvgFormatType`	character string; format type for continuous average (mean) field(s); valid values are "int", "dlr" and "pct"
`gainsChartDT.contAvgFormatDigits`	non-negative integer value; number of decimal places for continuous average (mean) field(s)
`gainsChartDT.KScolor`	character string; text color for max KS value (valid color)
`gainsChartBarGraph.cb`	character string; fill color for build bars (valid color)
`gainsChartBarGraph.cv`	character string; fill color for validate bars (valid color)
`gainsChartBarGraph.xLabel`	character string; x-axis label
`gainsChartBarGraph.yLabel`	character string; y-axis label
`gainsChartBarGraph.yType`	character string; y-axis format type; valid values are `"int"`, `"dlr"` and `"pct"`
`gainsChartBarGraph.yDigits`	non-negative integer value indicating the number of decimal places to show when hovering over the bars
`gainsChartPerfCurve.cb`	character string; line color for build (valid color)
`gainsChartPerfCurve.cv`	character string; line color for validate (valid color)
`gainsChartPerfCurve.cr`	character string; random/reference line color (valid color)
`gainsChartPerfCurve.xLabel`	character string; x-axis label
`gainsChartPerfCurve.yLabel`	character string; y-axis label
`variableImportanceGraph.x`	character vector
`variableImportanceGraph.y`	numeric vector
`variableImportanceGraph.sumYcheck`	logical value; check that `sum(y) == 1`?
`variableImportanceGraph.barColor`	character string; fill color for bars (valid color)
`variableImportanceGraph.lineColor`	character string; line color (valid color)
`variableImportanceGraph.lineWidth`	non-negative integer value indicating line width (use 0 to omit)

A named list with class mt_modelPerformance containing the following objects. Note that when v is supplied, the data will be split into build and validate sets where each set will be evaluated. When v is not supplied, the data is assumed to just be the "build" set.

build: a list containing the following objects:
- gc: gainsChart() object based on build set
- gcDT: gainsChartDT() object based on build set
- gcBG: gainsChartBarGraph() object based on build set
- gcPC: gainsChartPerfCurve() object based on build set
validate: (only returned when v is supplied) same type of list as build but based on validate set
both: (only returned when v is supplied) a list containing the following objects:
- gcBG: gainsChartBarGraph() object based on build and validate sets
- gcPC: gainsChartPerfCurve() object based on build and validate sets
varImp: (only returned when variableImportanceGraph.* parms are supplied) variableImportanceGraph() object

gainsChart, gainsChartDT, gainsChartBarGraph, gainsChartPerfCurve, variableImportanceGraph

# pull in sample scored data frame
x <- modelSampleScored
head(x)
# binary target
z <- modelPerformance(
    y = x$TargetFlag,
    yhat = x$pTargetFlag,
    v = x$ValidateFlag
)
class(z)
names(z)
names(z$build)
names(z$validate)
names(z$both)
z$validate$gcDT
z$both$gcBG
z$both$gcPC
# continuous target
z <- modelPerformance(
    y = x[x$TargetFlag, ]$TargetValue,
    yhat = x[x$TargetFlag, ]$pTargetValue,
    v = x[x$TargetFlag, ]$ValidateFlag,
    gainsChartBarGraph.yType = "dlr",
    gainsChartBarGraph.yDigits = 0
)
class(z)
names(z)
names(z$build)
names(z$validate)
names(z$both)
z$validate$gcDT
z$both$gcBG
z$both$gcPC
# combined target (with arbitrary variable importance parms)
z <- modelPerformance(
    y = x$TargetFlag,
    yhat = x$pTargetFlag*x$pTargetValue,
    y2 = x$TargetValue,
    v = x$ValidateFlag,
    variableImportanceGraph.x = c(
        "Days Since Last Transaction",
        "Average Order Value",
        "Signed Up Online",
        "Number of Transactions",
        "Tenure",
        "Multi-Category Flag"
    ),
    variableImportanceGraph.y = c(.18, .4, .04, .25, .12, .01)
)
class(z)
names(z)
names(z$build)
names(z$validate)
names(z$both)
z$validate$gcDT
z$both$gcBG
z$both$gcPC
z$varImp