A Short Introduction to the FSinR Package

In FSinR: Feature Selection

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)

FSinR Package

The package FSinR contains functions for performing the feature selection task. More specifically, it contains a large number of filter and wrapper functions widely used in the literature that can be integrated into search methods, although they can also be executed individually. The FSinR package uses the functions for training classification and regression models available in the R caret package to generate wrapper measurements. This gives the package a great background of methods and functionalities. In addition, the package has been implemented in such a way that its use is as easy and intuitive as possible. This is why the calls to all search methods and all filter and wrapper functions follow the same structure.

The way to install the package from the CRAN repository is as follows:

install.packages("FSinR")

As mentioned above, the package contains numerous filter and wrapper methods that can be executed as evaluation measures within a search algorithm in order to find a subset of features. This subset of features is used to generate models that represent the data set in a better manner. Therefore, the best way to use the filter and wrapper methods is through the search functions (although they can also be run independently). The search functions present in the package are the following:

• Sequential forward selection (sfs)
• Sequential floating forward selection (sffs)
• Sequential backward selection (sbs)
• Sequential floating backward selection (sfbs)
• Breadth first search (bfs)
• Deep first search (dfs)
• Genetic algorithm (ga)
• Whale optimization algorithm (woa)
• Ant colony optimization (aco)
• Simulated annealing (sa)
• Taboo search (ts)
• Hill-Climbing (hc)
• Las Vegas wrapper (lvw)

These search functions are the main functions on which the package works. The structure of all of them is the same and they contain the same following parameters:

• data: a matrix or data.frame with the dataset
• class: the name of the dependent variable
• featureSetEval: the filter or wrapper measurement with which the data will be evaluated
• ... : the specific parameters of each search method

It is important to note that the FSinR package does not split the data into training data and test data, but instead applies the feature selection over the entire data set passed to it as a parameter. Then, in a modeling process, the data should be separated by the user prior to the whole process of the training and test sets. Missing data must be processed prior to the use of the package. The following are some examples of using the package along with a brief description of how it works.

Wrapper example

To demonstrate in a simple manner how wrapper methods work, the iris dataset will be used in this example. The dataset consists in 150 instances of 4 variables (Sepal.Length, Sepal.Width, Petal.Length, Petal.Width) that determine the type of iris plant. The target variable, Species, has 3 possible classes (setosa, versicolor, virginica). For a correct use of the package, the data on which the feature selection is performed must be the training data. But the main objective of this vignette is to illustrate the use of the package, and not the complete modeling process, so in this case the whole dataset will be used without partitioning.

library(caret)
library(FSinR)

data(iris)

In the package, wrapper methods are passed as an evaluation measure to the search algorithms. The possible wrapper methods to use are the 238 models available in caret. In addition, the caret package offers the possibility of establishing a group of options to personalize the models (eg. resampling techniques, evaluation measurement, grid parameters, ...) using the trainControl and train functions. In FSinR, the wrapper function is used to set all these parameters and use them to generate the wrapper model using as background the methods of caret. The wrapper function has as parameters:

• learner: model name of those available in caret
• resamplingParams: list of parameters for trainControl function
• fittingParams: list of parameters for train function (x, y, method and trainControl not neccesary)

In the example we use a knn model, since the iris problem is a classification problem. The FSinR package is able to detect automatically depending on the metric whether the objective of the problem is to maximize or minimize. To tune the model, the resampling method is established as a 10-fold crossvalidation, the dataset is centered and scaled, the accuracy is used as a metric, and a grid of the k parameter is performed.

resamplingParams <- list(method = "cv", number = 10)
fittingParams <- list(preProc = c("center", "scale"), metric="Accuracy", tuneGrid = expand.grid(k = c(1:20)))

wra <- wrapper("knn", resamplingParams, fittingParams)

For more details, the way in which caret train and tune a model can be seen here. The link contain tutorials on how to use the caret functions, and also show the parameters that accept the functions and the possible values they can take. A list of available models in caret can be found here.

The wrapper model is obtained as a result of the call to the wrapper function, and is passed as a parameter to the search function. The search algorithm used in this example is ts (taboo search). As mentioned earlier, the search methods have 3 parameters that are always present, which are the dataset, the class name, and the wrapper or filter method. And in addition, each algorithm has its own parameters for optimal modeling. An example of a call to the default search function is as follows:

result.search.fs <- ts(iris, "Species", wra)

But it is also possible to set some specific parameters of the search method. In this case, the number of algorithm iterations, the size of the taboo list, as well as an intensification phase and a diversification phase, both of 5 iterations each, are established. Although in this example has not been taken into account, it is important to note that the number of neighbors that are considered and evaluated in each iteration of the algorithm, numNeigh, is set by default to as many as there are, then it should be noted that a high value of this parameter considerably increases the calculation time. Most FSinR package search algorithms include a parameter called verbose, which if set to TRUE shows the development and information of the iterations of the algorithms per console.

result.search.fs <- ts(iris, 'Species', wra, tamTabuList = 4, iter = 5, intensification=2, iterIntensification=5, diversification=1, iterDiversification=5, verbose=FALSE)

The search algorithm call returns a list of the most important results and the most important details of the process.

result.search.fs$bestFeatures
result.search.fs$bestFitness

In the example, the output only shows the best subset of features chosen in the feature selection process and the accuracy measurement obtained. In this particular case, the taboo search result also returns the status of the taboo list in each iteration, as well as the subset of features chosen in each iteration. Although in this example is not shown to be extensive.

The wrapper method generated above can also be used directly, without being inside a search algorithm. To do this, the data set, the name of the variable to be predicted, and a vector with the names of the features to be taken into account are passed to the method as parameters.

wrapperMeasure <- wrapper(iris,"Species",c("Sepal.Length", "Sepal.Width", "Petal.Length", "Petal.Width"))
wrapperMeasure

This returns as a result the evaluation measure of the wrapper method on the data set with all variables passed as parameter. As it can be seen, this value is lower, and therefore worse, than the one obtained with the feature selection process.

The above example is a classification problem, but the package methods can also be applied to regression problems. For this, we use the dataset mtcars which contains 10 design variables from 32 automobiles. The variable to predict, mpg, is a numerical variable.

data(mtcars)

The wrapper function has to be generated again. In this case it can be seen how to use a repeated 10-fold cross validation and how the metric also changes to a metric for regression problems, RMSE. In addition, in this case a neural network will be used to fit the model, and therefore has its specific parameters, size and decay.

resamplingParams <- list(method="repeatedcv", number = 10, repeats = 3)
fittingParams <- list(preProc = c("center", "scale"), metric="RMSE", tuneGrid = expand.grid(size = seq(1,12,by=2), decay=0), trace=FALSE)

wra <- wrapper("nnet",resamplingParams, fittingParams)

The search algorithm is then executed. In this case a genetic algorithm is used using the ga function and some parameters are defined.

result.search.fs <- ga(mtcars, 'mpg', wra, popSize = 10, pcrossover = 0.8, pmutation = 0.1, maxiter=5, verbose=TRUE)

The search algorithm returns again a list with the best subset of features found and the evaluation measure obtained. In addition, this algorithm also returns the final population of individuals along with their evaluation measure.

result.search.fs

Filter example

For the example of the filter method, the iris dataset is used again as in the previous example. Again, it is important to note that the purpose of this example is to show how the package works, rather than a complete modeling process, so variable selection methods are applied to the entire dataset without partitioning. Otherwise the dataset should be partitioned into training and test data, and the feature selection should be applied over the training data.

library(FSinR)

data(iris)

The following filter methods are implemented in the package:

• Chi-squared
• Cramer V
• F-score
• Relief
• Rough Sets consistency
• Binary consistency
• Inconsistent Examples consistency
• Inconsistent Examples Pairs consistency
• Determination Coefficient (R^2^, to continous features)
• Mutual information
• Gain ratio
• Symmetrical uncertain
• Gini index
• Jd evaluation
• MDLC evaluation
• RFSM evaluation

These methods can be passed as parameters to the search algorithms. But unlike wrapper methods, filter methods are directly implemented in the package for use, and there is no need to generate a function prior to use in the search methods as was the case with wrapper in the wrapper example.

Therefore, the search method is executed directly with the name of the filter function as parameter. In this case the search algorithm used in the example is Sequential forward selection, sfs. This algorithm is simple and therefore will not set additional parameters to the required. And on the other hand, the chosen filter measure is the Gini Index.

result.search.fs <- sfs(iris, "Species", giniIndex)

This search algorithm returns a list with the best subset of features it has found, and the value of the measurement obtained.

result.search.fs

As with wrapper measures, filter measures can be used directly without the need to include them in a search algorithm. To do this, the data set, the name of the variable to be predicted, and a vector with the names of the features to be taken into account are passed to the method as parameters.

filterMeasure <- giniIndex(iris, "Species", c("Sepal.Length", "Sepal.Width", "Petal.Length", "Petal.Width"))

Filter methods can also be applied to regression problems. For this as in the previous wrapper regression example the dataset mtcars is used. As a search method the same algorithm is used as in the previous example, sfs, and as a filter method also the same method giniIndex.

result.search.fs <- sfs(mtcars, "mpg", giniIndex)
result.search.fs

FSinR documentation built on Nov. 23, 2020, 5:10 p.m.