View source: R/inspect_performance.R
inspect_performance | R Documentation |
inspect_performance
returns various performance measures from a fitted
uplift model, including predicted uplift versus observed response under each
treatment, and the Qini coefficient. The results can be used to seamlessly
produce Qini curves and calibration plots.
## S3 method for class 'formula' inspect_performance(formula, data, subset, na.action = na.fail, method = "quantile", nBins = 10, qini = FALSE, qini.nBins = NULL, userBreaks = NULL, classLevel = NULL, treatLevel = NULL) ## S3 method for class 'inspect_performance' print(x, ...) ## S3 method for class 'inspect_performance' summary(object, ...)
formula |
A model formula of the form y ~ pred_1 + ....+ pred_n + trt(), where the left-hand side corresponds to the observed response, and the right-hand side corresponds to the predicted values from an arbitrary number of uplift models, and 'trt' is the special expression to mark the treatment term. If the treatment term is not a factor, it is converted to one. |
data |
A data frame in which to interpret the variables named in the formula. |
subset |
Expression indicating which subset of the rows of data should be included. All observations are included by default. |
na.action |
A missing-data filter function. |
method |
Possible values are |
nBins |
The number of bins to create. |
qini |
Return the Qini coefficient for each model? |
qini.nBins |
The number of cutoffs in the predicted values used for constructing the qini curves. Cutoffs are created based on quantiles of the distribution of the predicted values. By default, cutoffs are given by unique values of the predictions. |
userBreaks |
A user-specified numeric vector of breaks for the predicted
uplift values from which to create bins. It is required when |
classLevel |
A character string for the class of interest. Only applicable when the response is a factor. Defaults to the last level of the factor. |
treatLevel |
A character string for the treatment level of interest. Defaults to the last level of the treatment factor. |
x |
A |
... |
Additional arguments for the S3 methods. |
object |
A |
The Qini curve (Radcliffe, 2007) is a two-dimensional depiction of model
performance for uplift models. It represents a natural extension of the Gains
curve (Blattberg et al., 2008, p. 319) for uplift models. It is constructed as
follows: (i) observations are sorted by the uplift predictions in descending
order, (ii) the cumulative number of responses are computed for each
treatment and expressed as a percentage of the total number of observations
within each treatment, (iii) the "net lift" is computed as the difference
between the values obtained in (ii) for the reference treatment
(treatLevel
) and the alternative treatment. The Qini curve is a plot
of the net lift versus the corresponding fraction of observations in the data.
The interpretation of the Qini curve is as follows: on the x-axis we show the fraction of subjects in the population in which the treatment is performed, and on the y-axis we show the difference in the success rate between the reference treatment and the alternative treatment.
A benchmark for a given uplift model can be represented by the strategy of randomly selecting subjects to perform the treatment. This is represented in the figure by the diagonal line. For example, if we perform the treatment on 30 percent of the population, we expect to obtain 30 percent of the net lift relative to performing the action on the entire population.
The Qini coefficient is a single estimate of model performance ranging from +100 (best possible model), to -100 (worst possible model). A value of zero represents a performance equivalent to that of a random model. The Qini coefficient is computed as
(AUQC_m - AUQC_r) / (AUQC_o - AUQC_r)
where AUQC_m, AUQC_r, and AUQC_o represent the area under the Qini curve for the fitted uplift model, the random model, and the optimal model, respectively.
If method = "quantile"
(the default), 'n' bins (nBins
) are
created based on quantiles of the distribution of the predicted values (in an
attempt to have the same number of observations in each bin). If method
= "bucket"
, bins are created by dividing the predicted values into equally
spaced intervals based on the difference between the minimum and maximum
values. Unlike method = "quantile"
, the number of observations in each
group is typically unequal. If method = "user"
, bins are created
according to user-specified breaks (userBreaks
).
In some cases, it may not be feasible to obtain the number of bins requested
(nBins
) (e.g., due to many ties in the predicted values). The function
returns the effective number of bins created for each model
(actualBins
).
An object of class "inspect_performance"
, which is a list with
the following components:
data
The data for
underlying calibration plots
method
The method used to create
the bins
models
The labels of the variables supplied in the
right-hand side of the model formula
nBins
The number of bins
requested
actualnBins
The effective number of bins created for
each model
yFactor
Is response a factor?
classLevel
The class of interest
treatLevel
The
treatment level of interest
qiniCall
Whether the qini
coefficient was requested in the function call
qiniData
The
data for plotting the qini curves
qiniC
The qini coefficient
treatInd
The position of the treatment level of interest
call
The original call to inspect_performance
Leo Guelman leo.guelman@gmail.com
Blattberg, R. C., Do, K. B., and Scott, N. A. (2008). "Database Marketing: Analyzing and Managing Customers". Springer Science+Business Media, New York, NY.
Radcliffe, N (2007). "Using control groups to target on predicted lift: Building and assessing uplift models." Direct Marketing Analytics Journal, An Annual Publication from the Direct Marketing Association Analytics Council: pp. 14–21.
ggplot.inspect_performance
.
set.seed(324) df_train <- sim_uplift(p = 30, response = "binary") df_test <- sim_uplift(n = 10000, p = 30, response = "binary") fit_t1 <- glm(as.formula(paste('y ~', paste('X', 1:30, sep = '', collapse = "+"))), family = "binomial", data = df_train, subset = T==1) fit_t0 <- glm(as.formula(paste('y ~', paste('X', 1:30, sep = '', collapse = "+"))), family = "binomial", data = df_train, subset = T==-1) uplift_score <- predict(fit_t1, df_test, type = "response") - predict(fit_t0, df_test, type = "response") df_test$uplift_score <- uplift_score res <- inspect_performance(y ~ uplift_score + trueUplift + trt(T), data = df_test, qini = TRUE) res summary(res)
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.