MLP_varIMP_NULL_model: MLP variable importance using NULL model

In xinghuq/DeepGenomeScan: DeepGenomeScan: A Deep Learning Approach for Whole Genome Scan (WGS) and Genome-wide Association Studies (GWAS)

MLP variable importance using NULL model

Description

MLP NULL variable importance. The NULL importance measured by the decrease in a model score (i.e., Mean Decrease Accuracy (MDA), Mean Decrease in RMSE) when a variable is set as NULL variable.

Usage

MLP_varIMP_NULL_model(optmodel, feature_names = NULL, train_y = NULL, train_x = NULL, smaller_is_better = NULL, type = c("difference", "ratio"), nsim = 1, sample_size = NULL, sample_frac = NULL, verbose = FALSE, progress = "none", parallel = FALSE, paropts = NULL, ...)

Arguments

optmodel	The optimal model
feature_names	The variable names, should be the same as the dim of X
train_y	The Y variable (dependent variable) used in regression
train_x	The independent variable dataset
smaller_is_better
type
nsim
sample_size
sample_frac
verbose
progress
parallel
paropts
...

Details

This function estimates the similar model agnostic feature importance but rather using time consuming permutation importance, it uses the Null model importance. Likewise, the best model is determined and the orignal variable metrics are used as the baseline. Then the NULL variable performance metrics are tested using the best model as the training set. This procedure directly set the target variable as NULL variable (Zero variance), thus the drop in the model score is indicative of how much the model depends on the variable.

Value

Return a list of scores, including MLP model decrease in accuracy, the permutation metrics, and the baseline metrics.

References

Ribeiro, Marco Tulio, Sameer Singh, and Carlos Guestrin. "Model-agnostic interpretability of machine learning." arXiv preprint arXiv:1606.05386 (2016). Fisher, Aaron, Cynthia Rudin, and Francesca Dominici. “Model Class Reliance: Variable importance measures for any machine learning model class, from the ‘Rashomon’ perspective.” http://arxiv.org/abs/1801.01489 (2018).

Examples

##---- Should be DIRECTLY executable !! ----
##-- ==>  Define data, use random,
##--	or do  help(data=index)  for the standard data sets.

## The function is currently defined as
function (optmodel, feature_names = NULL, train_y = NULL, train_x = NULL, 
    smaller_is_better = NULL, type = c("difference", "ratio"), 
    nsim = 1, sample_size = NULL, sample_frac = NULL, verbose = FALSE, 
    progress = "none", parallel = FALSE, paropts = NULL, ...) 
{
    require(caret)
    baseline <- as.data.frame(t(caret::postResample(pred = keras::predict_on_batch(optmodel, 
        train_x), obs = train_y)))
    type <- match.arg(type)
    `%compare%` <- if (type == "difference") {
        `-`
    }
    else {
        `/`
    }
    NULL_columns <- function(x, columns = NULL) {
        if (is.null(columns)) {
            stop("No columns specified for permutation.")
        }
        x[, columns] <- rep(1, nrow(x))
        x
    }
    sort_importance_scores <- function(x, decreasing) {
        x[order(x$Importance, decreasing = decreasing), ]
    }
    MLP_varIMP <- replicate(nsim, (plyr::llply(feature_names, 
        .progress = "none", .parallel = parallel, .paropts = paropts, 
        .fun = function(x) {
            if (verbose && !parallel) {
                message("Computing variable importance for ", 
                  x, "...")
            }
            if (!is.null(sample_size)) {
                ids <- sample(length(train_y), size = sample_size, 
                  replace = FALSE)
                train_x <- train_x[ids, ]
                train_y <- train_y[ids]
            }
            train_x_NULL <- NULL_columns(train_x, columns = x)
            permuted <- as.data.frame(t(caret::postResample(pred = keras::predict_on_batch(optmodel, 
                train_x_NULL), obs = train_y)))
        })))
    varimp1 = apply(MLP_varIMP, MARGIN = 1, function(x) do.call(cbind, 
        x))
    varimp2 = do.call(rbind, varimp1)
    nm <- colnames(varimp2)
    RMSE = varimp2[, grepl("^RMSE", nm)]
    Rsquared = varimp2[, grepl("^Rsquared", nm)]
    MAE = varimp2[, grepl("^MAE", nm)]
    RMSE = as.data.frame(rowMeans(RMSE, na.rm = TRUE))
    Rsquared = as.data.frame(rowMeans(Rsquared, na.rm = TRUE))
    MAE = as.data.frame(rowMeans(MAE, na.rm = TRUE))
    rownames(RMSE) = feature_names
    rownames(Rsquared) = feature_names
    rownames(MAE) = feature_names
    MLP_SNPsIMP = cbind(RMSE, Rsquared, MAE)
    rownames(MLP_SNPsIMP) = feature_names
    decrease_acc = lapply(1:3, function(i) baseline[, i] - MLP_SNPsIMP[, 
        i])
    MLP_Decrease_acc = do.call(cbind, decrease_acc)
    return(list(MLP_Decrease_acc = MLP_Decrease_acc, MLP_SNPsIMP = MLP_SNPsIMP, 
        baseline = baseline))
  }

xinghuq/DeepGenomeScan documentation built on Sept. 20, 2022, 8:46 a.m.