Construct and analyze confusion matrices

Share:

Description

Confusion matrices compare two classifications (usually one done automatically using a machine learning algorithm versus the true classication represented by a manual classification by a specialist... but one can also compare two automatic or two manual classifications against each other).

Usage

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
confusion(x, ...)
## Default S3 method:
confusion(x, y = NULL, vars = c("Actual", "Predicted"),
    labels = vars, merge.by = "Id", useNA = "ifany", prior, ...)
## S3 method for class 'mlearning'
confusion(x, y = response(x),
    labels = c("Actual", "Predicted"), useNA = "ifany", prior, ...)

## S3 method for class 'confusion'
print(x, sums = TRUE, error.col = sums, digits = 0,
    sort = "ward", ...)
## S3 method for class 'confusion'
summary(object, type = "all", sort.by = "Fscore",
    decreasing = TRUE, ...)
## S3 method for class 'summary.confusion'
print(x, ...)
## S3 method for class 'confusion'
plot(x, y = NULL, type = c("image", "barplot", "stars",
    "dendrogram"), stat1 = "Recall", stat2 = "Precision", names, ...)

confusionImage(x, y = NULL, labels = names(dimnames(x)), sort = "ward",
    numbers = TRUE, digits = 0, mar = c(3.1, 10.1, 3.1, 3.1), cex = 1, asp = 1,
    colfun, ncols = 41, col0 = FALSE, grid.col = "gray", ...)
confusionBarplot(x, y = NULL, col = c("PeachPuff2", "green3", "lemonChiffon2"),
    mar = c(1.1, 8.1, 4.1, 2.1), cex = 1, cex.axis = cex, cex.legend = cex,
    main = "F-score (precision versus recall)", numbers = TRUE, min.width = 17,
    ...)
confusionStars(x, y = NULL, stat1 = "Recall", stat2 = "Precision", names, main,
    col = c("green2", "blue2", "green4", "blue4"), ...)
confusionDendrogram(x, y = NULL, labels = rownames(x), sort = "ward",
    main = "Groups clustering", ...)

prior(object, ...)
## S3 method for class 'confusion'
prior(object, ...)
prior(object, ...) <- value
## S3 replacement method for class 'confusion'
prior(object, ...) <- value

Arguments

x

an object.

y

another object, from which to extract the second classification, or NULL if not used.

vars

the variables of interest in the first and second classification in the case the objects are lists or data frames. Otherwise, this argument is ignored and x and y must be factors with same length and same levels.

labels

labels to use for the two classifications. By default, it is the same as vars or the one in the confusion matrix.

merge.by

a character string with the name of variables to use to merge the two data frames, or NULL.

useNA

do we keep NAs as a separate category? The default "ifany" creates this category only if there are missing values. Other possibilities are "no", or "always".

prior

class frequencies to use for first classifier that is tabulated in the rows of the confusion matrix. For its value, see here under, the value argument.

sums

is the confusion matrix printed with rows and columns sums?

error.col

is a column with class error for first classifier added (equivalent to flase negative rate of FNR)?

digits

the number of digits after the decimal point to print in the confusion matrix. The default or zero leads to most compact presentation and is suitable for frequencies, but not for relative frequencies.

sort

are rows and columns of the confusion matrix sorted so that classes with larger confusion are closer together? Sorting is done using a hierachical clustering with hclust(). The clustering method is provided is the one provides ("ward", by default, but see the hclust() help for other options). If FALSE or NULL, no sorting is done.

object

a 'confusion' object.

sort.by

the statistics to use to sort the table (by default, Fmeasure, the F1 score for each class = 2 * recall * precision / (recall + precision)).

decreasing

do we sort in increasing or decreasing order?

type

the type of graph to plot (only "stars" if two confusion matrices are to be compared).

stat1

first statistic to compare in the stars plot.

stat2

second statistic to compare in the stars plot.

...

further arguments passed to the function. In particular for plot(), it can be all arguments for the corresponding plot.

numbers

are actual numbers indicated in the confusion matrix image?

mar

graph margins.

cex

text magnification factor.

cex.axis

idem for axes. If NULL, the axis is not drawn.

cex.legend

idem for legend text. If NULL, no legend is added.

asp

graph aspect ration. There is little reasons to cvhange the default value of 1.

col

color(s) to use fir the graph.

colfun

a function that calculates a series of colors, like e.g., cm.colors() and that accepts one argument being the number of colors to be generated.

ncols

the number of colors to generate. It should preferrably be 2 * number of levels + 1, where levels is the number of frequencies you want to evidence in the plot. Default to 41.

col0

should null values be colored or not (no, by default)?

grid.col

color to use for grid lines, or NULL for not drawing grid lines.

names

names of the two classifiers to compare.

main

main title of the graph.

min.width

minimum bar width required to add numbers.

value

a single positive numeric to set all class frequencies to this value (use 1 for relative frequencies and 100 for relative freqs in percent), or a vector of positive numbers of the same length as the levels in the object. If the vector is named, names must match levels. Alternatively, providing NULL or an object of null length resets row class frequencies into their initial values.

Value

A confusion matrix in a 'confusion' object. prior() returns the current class frequencies associated with first classification tabulated, i.e., for rows in the confusion matrix.

Author(s)

Philippe Grosjean <Philippe.Grosjean@umons.ac.be> and Kevin Denis <Kevin.Denis@umons.ac.be>

See Also

mlearning, hclust, cm.colors

Examples

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
data("Glass", package = "mlbench")
## Use a little bit more informative labels for Type
Glass$Type <- as.factor(paste("Glass", Glass$Type))

## Use learning vector quantization to classify the glass types
## (using default parameters)
summary(glassLvq <- mlLvq(Type ~ ., data = Glass))

## Calculate cross-validated confusion matrix and plot it in different ways
(glassConf <- confusion(cvpredict(glassLvq), Glass$Type))
## Raw confusion matrix: no sort and no margins
print(glassConf, sums = FALSE, sort = FALSE)
## Graphs
plot(glassConf) # Image by default
plot(glassConf, sort = FALSE) # No sorting
plot(glassConf, type = "barplot")
plot(glassConf, type = "stars")
plot(glassConf, type = "dendrogram")

summary(glassConf)
summary(glassConf, type = "Fscore")

## Build another classifier and make a comparison
summary(glassNaiveBayes <- mlNaiveBayes(Type ~ ., data = Glass))
(glassConf2 <- confusion(cvpredict(glassNaiveBayes), Glass$Type))

## Comparison plot for two classifiers
plot(glassConf, glassConf2)

## When the probabilities in each class do not match the proportions in the
## training set, all these calculations are useless. Having an idea of
## the real proportions (so-called, priors), one should first reweight the
## confusion matrix before calculating statistics, for instance:
prior1 <- c(10, 10, 10, 100, 100, 100) # Glass types 1-3 are rare
prior(glassConf) <- prior1
glassConf
summary(glassConf, type = c("Fscore", "Recall", "Precision"))
plot(glassConf)

## This is very different than if glass types 1-3 are abundants!
prior2 <- c(100, 100, 100, 10, 10, 10) # Glass types 1-3 are abundants
prior(glassConf) <- prior2
glassConf
summary(glassConf, type = c("Fscore", "Recall", "Precision"))
plot(glassConf)

## Weight can also be used to construct a matrix of relative frequencies
## In this case, all rows sum to one
prior(glassConf) <- 1
print(glassConf, digits = 2)
## However, it is easier to work with relative frequencies in percent
## and one gets a more compact presentation
prior(glassConf) <- 100
glassConf

## To reset row class frequencies to original propotions, just assign NULL
prior(glassConf) <- NULL
glassConf
prior(glassConf)