inspect_performance: Inspect performance from a fitted uplift model.

View source: R/inspect_performance.R

inspect_performanceR Documentation

Inspect performance from a fitted uplift model.

Description

inspect_performance returns various performance measures from a fitted uplift model, including predicted uplift versus observed response under each treatment, and the Qini coefficient. The results can be used to seamlessly produce Qini curves and calibration plots.

Usage

## S3 method for class 'formula'
inspect_performance(formula, data, subset,
  na.action = na.fail, method = "quantile", nBins = 10, qini = FALSE,
  qini.nBins = NULL, userBreaks = NULL, classLevel = NULL,
  treatLevel = NULL)

## S3 method for class 'inspect_performance'
print(x, ...)

## S3 method for class 'inspect_performance'
summary(object, ...)

Arguments

formula

A model formula of the form y ~ pred_1 + ....+ pred_n + trt(), where the left-hand side corresponds to the observed response, and the right-hand side corresponds to the predicted values from an arbitrary number of uplift models, and 'trt' is the special expression to mark the treatment term. If the treatment term is not a factor, it is converted to one.

data

A data frame in which to interpret the variables named in the formula.

subset

Expression indicating which subset of the rows of data should be included. All observations are included by default.

na.action

A missing-data filter function.

method

Possible values are "quantile" (default) if you want to create intervals from the predicted uplift values with approximately the same number of observations in each group, "bucket" if you want to divide the predicted response values into equally spaced intervals, or "user" to create intervals from user-specified breaks (see details below).

nBins

The number of bins to create.

qini

Return the Qini coefficient for each model?

qini.nBins

The number of cutoffs in the predicted values used for constructing the qini curves. Cutoffs are created based on quantiles of the distribution of the predicted values. By default, cutoffs are given by unique values of the predictions.

userBreaks

A user-specified numeric vector of breaks for the predicted uplift values from which to create bins. It is required when method = "user", and ignored otherwise.

classLevel

A character string for the class of interest. Only applicable when the response is a factor. Defaults to the last level of the factor.

treatLevel

A character string for the treatment level of interest. Defaults to the last level of the treatment factor.

x

A inspect_performance object.

...

Additional arguments for the S3 methods.

object

A inspect_performance object.

Details

The Qini curve (Radcliffe, 2007) is a two-dimensional depiction of model performance for uplift models. It represents a natural extension of the Gains curve (Blattberg et al., 2008, p. 319) for uplift models. It is constructed as follows: (i) observations are sorted by the uplift predictions in descending order, (ii) the cumulative number of responses are computed for each treatment and expressed as a percentage of the total number of observations within each treatment, (iii) the "net lift" is computed as the difference between the values obtained in (ii) for the reference treatment (treatLevel) and the alternative treatment. The Qini curve is a plot of the net lift versus the corresponding fraction of observations in the data.

The interpretation of the Qini curve is as follows: on the x-axis we show the fraction of subjects in the population in which the treatment is performed, and on the y-axis we show the difference in the success rate between the reference treatment and the alternative treatment.

A benchmark for a given uplift model can be represented by the strategy of randomly selecting subjects to perform the treatment. This is represented in the figure by the diagonal line. For example, if we perform the treatment on 30 percent of the population, we expect to obtain 30 percent of the net lift relative to performing the action on the entire population.

The Qini coefficient is a single estimate of model performance ranging from +100 (best possible model), to -100 (worst possible model). A value of zero represents a performance equivalent to that of a random model. The Qini coefficient is computed as

(AUQC_m - AUQC_r) / (AUQC_o - AUQC_r)

where AUQC_m, AUQC_r, and AUQC_o represent the area under the Qini curve for the fitted uplift model, the random model, and the optimal model, respectively.

If method = "quantile" (the default), 'n' bins (nBins) are created based on quantiles of the distribution of the predicted values (in an attempt to have the same number of observations in each bin). If method = "bucket", bins are created by dividing the predicted values into equally spaced intervals based on the difference between the minimum and maximum values. Unlike method = "quantile", the number of observations in each group is typically unequal. If method = "user", bins are created according to user-specified breaks (userBreaks).

In some cases, it may not be feasible to obtain the number of bins requested (nBins) (e.g., due to many ties in the predicted values). The function returns the effective number of bins created for each model (actualBins).

Value

An object of class "inspect_performance", which is a list with the following components:

  • data The data for underlying calibration plots

  • method The method used to create the bins

  • models The labels of the variables supplied in the right-hand side of the model formula

  • nBins The number of bins requested

  • actualnBins The effective number of bins created for each model

  • yFactor Is response a factor?

  • classLevel The class of interest

  • treatLevel The treatment level of interest

  • qiniCall Whether the qini coefficient was requested in the function call

  • qiniData The data for plotting the qini curves

  • qiniC The qini coefficient

  • treatInd The position of the treatment level of interest

  • call The original call to inspect_performance

Author(s)

Leo Guelman leo.guelman@gmail.com

References

Blattberg, R. C., Do, K. B., and Scott, N. A. (2008). "Database Marketing: Analyzing and Managing Customers". Springer Science+Business Media, New York, NY.

Radcliffe, N (2007). "Using control groups to target on predicted lift: Building and assessing uplift models." Direct Marketing Analytics Journal, An Annual Publication from the Direct Marketing Association Analytics Council: pp. 14–21.

See Also

ggplot.inspect_performance.

Examples


set.seed(324)
df_train <- sim_uplift(p = 30, response = "binary")
df_test <- sim_uplift(n = 10000, p = 30, response = "binary")
fit_t1 <- glm(as.formula(paste('y ~', paste('X', 1:30, sep = '', collapse = "+"))),
             family = "binomial", data = df_train, subset = T==1)
fit_t0 <- glm(as.formula(paste('y ~', paste('X', 1:30, sep = '', collapse = "+"))),
             family = "binomial", data = df_train, subset = T==-1)
uplift_score <- predict(fit_t1, df_test, type = "response") -
               predict(fit_t0, df_test, type = "response")
df_test$uplift_score <- uplift_score
res <- inspect_performance(y ~ uplift_score + trueUplift + trt(T),
                          data = df_test, qini = TRUE)
res
summary(res)

leoguelman/uplift2 documentation built on April 15, 2022, 4:34 a.m.