Feature selection: crabs
In fdm2id: Data Mining and R Programming for Beginners

knitr::opts_chunk$set (collapse = TRUE, comment = "#>", fig.width = 6, fig.height = 4.2,
                       fig.align = "center")
optional = c ("MASS", "e1071")
available = all (sapply (optional, requireNamespace, quietly = TRUE))
knitr::opts_chunk$set (eval = available)

cat ("**Note.** This vignette needs the following packages, some of which are missing:",
     paste (optional, collapse = ", "), "-- the code is shown but not run.\n")

One of the case studies of the Fouille de données (M2 SID) course, for which fdm2id was written. Filters against a wrapper, on five measurements that are almost perfectly correlated and two different targets.

The other case studies are listed by vignette (package = "fdm2id"); they use the same handful of functions on other data, and can be read in any order.

library (fdm2id)

The data

The crabs dataset of the MASS package, on Leptograpsus variegatus crabs collected at Fremantle, Australia. Two colour forms, orange and blue, and both sexes. Five measurements were taken on each individual (in mm): the size of the frontal lobe (FL), the rear width (RW), the length of the carapace (CL), its width (CW) and the depth of the body (BD).

There are two targets rather than one: the species (sp, O or B) and the sex (sex, F or M), with 50 males and 50 females of each species.

data (crabs, package = "MASS")
summary (crabs)

plotdata (crabs [, 4:8], crabs [, 1], type = "pairs")

plotdata (crabs [, 4:8], crabs [, 2], type = "pairs")

The five measurements are almost perfectly correlated with each other -- a crab is simply bigger or smaller -- so most of what any one of them says, the others say too. That is what makes the selection the interesting question here rather than the classifier.

round (cor (crabs [, 4:8]), 3)

Question 1. Selecting for the species

Using a ranking algorithm and Fisher's index as the univariate criterion, which multivariate criterion -- the F statistic, mRMR, or a wrapper -- gives the best predictions of the species with a naive Bayes classifier?

# The two filters are deterministic; the wrapper is not -- it judges each subset by fitting a
# naive Bayes classifier under a bootstrap, so without 'seed' it can stop at a different
# subset from one run to the next. That is the criterion's own variance, not the data's.
s.fstat1 = selectfeatures (crabs [, 4:8], crabs [, 1], algorithm = "ranking",
                           unieval = "fisher", multieval = "fstat", seed = 0)
s.mrmr1 = selectfeatures (crabs [, 4:8], crabs [, 1], algorithm = "ranking",
                          unieval = "fisher", multieval = "mrmr", seed = 0)
s.wrap1 = selectfeatures (crabs [, 4:8], crabs [, 1], algorithm = "ranking",
                          unieval = "fisher", multieval = "wrapper", wrapmethod = NB,
                          seed = 0)
s.fstat1
s.mrmr1
s.wrap1

performance (NB, crabs [, 4:8], crabs [, 1], nruns = 100, seed = 0)
performance (NB, crabs [, 4:8] [, s.fstat1$selection], crabs [, 1], nruns = 100, seed = 0)
performance (NB, crabs [, 4:8] [, s.mrmr1$selection], crabs [, 1], nruns = 100, seed = 0)
performance (NB, crabs [, 4:8] [, s.wrap1$selection], crabs [, 1], nruns = 100, seed = 0)

Answer. All three criteria stop at the same subset and therefore give the same performance -- which is better than using all five variables. Adding correlated measurements to a naive Bayes classifier, whose whole assumption is that they are independent, costs accuracy.

Question 2. Selecting for the sex

The same question, for the sex of the crabs.

s.fstat2 = selectfeatures (crabs [, 4:8], crabs [, 2], algorithm = "ranking",
                           unieval = "fisher", multieval = "fstat", seed = 0)
s.mrmr2 = selectfeatures (crabs [, 4:8], crabs [, 2], algorithm = "ranking",
                          unieval = "fisher", multieval = "mrmr", seed = 0)
s.wrap2 = selectfeatures (crabs [, 4:8], crabs [, 2], algorithm = "ranking",
                          unieval = "fisher", multieval = "wrapper", wrapmethod = NB,
                          seed = 0)
s.fstat2
s.mrmr2
s.wrap2

performance (NB, crabs [, 4:8], crabs [, 2], nruns = 100, seed = 0)
performance (NB, crabs [, 4:8] [, s.fstat2$selection], crabs [, 2], nruns = 100, seed = 0)
performance (NB, crabs [, 4:8] [, s.mrmr2$selection], crabs [, 2], nruns = 100, seed = 0)
performance (NB, crabs [, 4:8] [, s.wrap2$selection], crabs [, 2], nruns = 100, seed = 0)

Answer. Only the wrapper improves on the full set of variables. The two filter criteria stop at a single variable and lose accuracy -- which is the trade-off of the family: a filter judges a subset without ever fitting the model it is selecting for, and here that judgement is wrong.

Question 3. Which variables, in the end?

colnames (crabs) [4:8] [s.fstat1$selection]
colnames (crabs) [4:8] [s.wrap2$selection]

Answer. The frontal lobe alone is what distinguishes the two species; distinguishing the sexes takes three measurements -- the rear width, the length of the carapace and the depth of the body. Two different questions asked of the same five measurements, and two different answers.