find_permuted_perf_metric: Get permuted performance metric difference for a single...

In SchlossLab/mikropml: User-Friendly R Package for Supervised Machine Learning Pipelines

Get permuted performance metric difference for a single feature (or group of features)

Description

Requires the future.apply package

Usage

find_permuted_perf_metric(
  test_data,
  trained_model,
  outcome_colname,
  perf_metric_function,
  perf_metric_name,
  class_probs,
  feat,
  test_perf_value,
  nperms = 100,
  alpha = 0.05,
  progbar = NULL
)

Arguments

test_data	Held out test data: dataframe of outcome and features.
trained_model	Trained model from caret::train().
outcome_colname	Column name as a string of the outcome variable (default NULL; the first column will be chosen automatically).
perf_metric_function	Function to calculate the performance metric to be used for cross-validation and test performance. Some functions are provided by caret (see caret::defaultSummary()). Defaults: binary classification = twoClassSummary, multi-class classification = multiClassSummary, regression = defaultSummary.
perf_metric_name	The column name from the output of the function provided to perf_metric_function that is to be used as the performance metric. Defaults: binary classification = "ROC", multi-class classification = "logLoss", regression = "RMSE".
class_probs	Whether to use class probabilities (TRUE for categorical outcomes, FALSE for numeric outcomes).
feat	feature or group of correlated features to permute.
test_perf_value	value of the true performance metric on the held-out test data.
nperms	number of permutations to perform (default: 100).
alpha	alpha level for the confidence interval (default: 0.05 to obtain a 95% confidence interval)
progbar	optional progress bar (default: NULL)

Value

vector of mean permuted performance and mean difference between test and permuted performance (test minus permuted performance)

Author(s)

Begüm Topçuoğlu, [e-mail]

Zena Lapp, [e-mail]

Kelly Sovacool, [e-mail]

SchlossLab/mikropml documentation built on Aug. 24, 2023, 9:51 p.m.