bivar: Bias-Variance Decomposition of the Misclassification Rate

In biVar: Bias-Variance Analysis of the Misclassification Rate

Description

Compute the bias-variance decomposition of the misclassification rate according to the approaches of James (2003) and Domingos (2000).

Usage

  bivar(y, grouping, ybayes, posterior, ybest = NULL)

Arguments

y	Predicted class labels on a test data set based on multiple training data sets. y is supposed to be a list where each element contains the predictions for one single test observation. #factor???
grouping	Vector of true class labels (a factor).
ybayes	(Optional.) Bayes prediction. Not used if posterior is specified as ybayes can be easily calculated from the posterior probabilities.
posterior	(Optional.) Matrix of posterior probabilities, either known or estimated. It is assumed that the columns are ordered according to the factor levels of grouping.
ybest	Prediction from best fitting model on the whole population. Used for calculation of model bias.

Details

If posterior is specified, ybayes is calculated from the posterior probabilities and the posteriors are used to estimate noise, error, systematic effect and variance effect. If ybayes is specified it is ignored if the posteriors are given. Otherwise the empirical distribution of ybayes is inferred and used to calculate the quantities of interest. If neither posterior nor ybayes are specified is assumed that the noise level is zero and the remaining quantities are calculated based on this assumption.

Value

A data.frame containing the following columns:

error	Estimated misclassification probability.
noise	(Only if ybayes or posterior was specified.) Noise.
bias	Bias.
variance	Variance.
unbiased.variance	Unbiased variance.
biased.variance	Biased variance.
net.variance	Pointwise net variance.
systematic.effect	Systematic effect.
variance.effect	Variance effect.
ymain	Main prediction.
ybayes	(Only if ybayes or posterior was specified.) The optimal prediction.
size	Numeric vector of the same length as the number of test observations. The number of predictions made for each test observation.

References

James, G. M. (2003). Variance and bias for general loss functions. Machine Learning, 51(2) 115–135.

Domingos, P. (2000). A unified bias-variance decomposition for zero-one and squared loss. In Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence, pages 564–569. AAAI Press / The MIT Press.

biVar documentation built on May 2, 2019, 6:29 p.m.