Given probability scores predictedProb
as provided for example by a call to predict.CoreModel
and using one of available methods given by methods
the function calibrates predicted probabilities so that they
match the actual probabilities of a binary class 1 provided by correctClass
.
1 2 3 
correctClass 
A vector of correct class labels for a binary classification problem. 
predictedProb 
A vector of predicted class 1 (probability) scores of the same length as 
class1 
A class value (factor) or an index of the class value to be taken as a class to be calibrated. 
method 
One of 
weight 
If specified, should be of the same length as 
noBins 
The value of parameter depends on the parameter 
assumeProbabilities 
If 
Depending on the specified method
one of the following calibration methods is executed.
"isoReg"
isotonic regression calibration based on pairadjacent violators (PAV) algorithm.
"binning"
calibration into a prespecified number of bands given by noBins
parameter, trying to make bins of equal weight.
"binIsoReg"
first binning method is executed, following by a isotonic regression calibration.
"mdlMerge"
first intervals are merged by a MDL gain criterion into a prespecified number of intervals, following by the isotonic regression calibration.
If model="binning"
the parameter noBins
specifies the desired number of bins i.e., calibration bands;
if model="binIsoReg"
the parameter noBins
specifies the number of initial bins that are formed by binning before isotonic regression is applied;
if model="mdlMerge"
the parameter noBins
specifies the number of bins formed after first applying isotonic regression. The most similar bins are merged using MDL criterion.
A function returns a list with two vector components of the same length:
interval 
The boundaries of the intervals. Lower boundary 0 is not explicitly included but should be taken into account. 
calProb 
The calibrated probabilities for each corresponding interval. 
Marko RobnikSikonja
I. Kononenko, M. Kukar: Machine Learning and Data Mining: Introduction to Principles and Algorithms. Horwood, 2007
A. NiculescuMizil, R. Caruana: Predicting Good Probabilities With Supervised Learning. Proceedings of the 22nd International Conference on Machine Learning (ICML'05), 2005
CORElearn
,
predict.CoreModel
.
1 2 3 4 5 6 7 8 9 10 11 12 13 14  # generate data
train <classDataGen(noInst=200)
cal <classDataGen(noInst=200)
# build random forests model with certain parameters
modelRF < CoreModel(class~., train, model="rf", selectionEstimator="MDL",
minNodeWeightRF=5, rfNoTrees=100, maxThreads=1)
# prediction
pred < predict(modelRF, cal, rfPredictClass=FALSE)
destroyModels(modelRF) # clean up
# calibrate for a chosen class1 and method
class1<1
calibrate(cal$class, pred$prob[,class1], class1=1, method="binning", noBins=5)

Questions? Problems? Suggestions? Tweet to @rdrrHQ or email at ian@mutexlabs.com.
All documentation is copyright its authors; we didn't write any of that.