PredVal: Predictive Values of Each Base Learner in Each Data Set

In kforthman/caretStack: Stacks models from the caret package to create a more robust prediction

Description

To assess model performance in the training and testing sets, we need:

1. Predictived values of each base learner in each data set.

2. If a stack model is build on the top of individual base learners, then we also need the predictive values of the stack model in both sets.

3. Compute performance metrics.

Usage

PredVal(
  models,
  TestSet,
  resp.var,
  ref.lv = NULL,
  method = "none",
  metric = NULL,
  stack.wt = NULL,
  trControl = NULL,
  tuneLength = NULL
)

Arguments

ref.lv	reference level for categorical variables.
method	A string specifying which classification or regression model to use. Possible values are found using names(getModelInfo()). See http://topepo.github.io/caret/train-models-by-tag.html. A list of functions can also be passed for a custom model function. See http://topepo.github.io/caret/using-your-own-model-in-train.html for details.
metric	A string that specifies what summary metric will be used to select the optimal model. By default, possible values are "RMSE" and "Rsquared" for regression and "Accuracy" and "Kappa" for classification. If custom performance metrics are used (via the summaryFunction argument in trainControl, the value of metric should match one of the arguments. If it does not, a warning is issued and the first metric given by the summaryFunction is used. (NOTE: If given, this argument must be named.)
stack.wt	???
trControl	A list of values that define how this function acts. See trainControl and http://topepo.github.io/caret/using-your-own-model-in-train.html. (NOTE: If given, this argument must be named.)
tuneLength	An integer denoting the amount of granularity in the tuning parameter grid. By default, this argument is the number of levels for each tuning parameters that should be generated by train. If trainControl has the option search = "random", this is the maximum number of tuning parameter combinations that will be generated by the random search. (NOTE: If given, this argument must be named.)

Details

For consistency purpose (with stacking predictions), I use defaultSummary(pred)

In PredVal, you can

1. specify a stacking method

2. specify a weight for each ML algorithm

kforthman/caretStack documentation built on June 21, 2021, 8:38 a.m.