This vignette is supposed to give you a short and easy introduction into the benchmarkVis package.

The purpose of the package is to provide a flexible and simple to use framework to visualize benchmark results.

Benchmarking is a good way to compare the performances of different algorithms. To evaluate the results often the same procedures are used. But even though different benchmarks normally contain similar information, the structure can differ significantly. This increases the effort to visualize and analyze them. You have to do the same creation steps over and over again with just a few little changes. At this point the benchmarkVis package comes into play. It aims to convert various formats into a default data table which can be visualized in multiple ways.

The package offers several differant plots which range from simple bar charts to complex parallel coordinates plots. All the functionalities are also provided from an interactive shiny application to simplify the workflow even more. This includes comprehensive methods to manipulate the input benchmark data.

The benchmarks themselves are not generated with this package. They have to be imported from other tools. But no problem, the benchmarkVis package offers several easy ways to integrate benchmark results from your favorite software. Therefore a clearly structured input data format has been defined which is combinable with csv and json files, R data.table or directly with R benchmark packages like mlr, microbenchmark or rbenchmark through different wrappers.

Install the package

To install the package simply type:

devtools::install_github("collinleiber/benchmarkVis")

Afterwards it can be loaded:

library(benchmarkVis)

Data Structure

In this section we will describe the input data structure of the benchmarkVis package. It consists of a table of the following structure:

Compatible data table

| problem | problem.parameter | algorithm | algorithm.parameter | replication | replication.parameter | measure.* | list.* | |---|---|---|---|---|---|---|---| | character | list | character | list | character | list | numeric | numeric vector | | mandatory | optional | mandatory | optional | optional | optional | optional | optional |

As you can see, each column has a fixed name and data type. Also some of the columns are optional while others are mandatory. The table can contain any number of measures and lists. It is important that at least one column of type measure or list is contained and that the column names start with "measure." / "list.".

Table components

To get the components of your input data table you can use following methods:

getMeasures(data.table)
getLists(data.table)
getMainColumns(data.table)
getParameterColumns(data.table)
getParameters(data.table, parameter.column)

The main columns always consist of problem and algorithm and can also contain replication.

Algorithm tuning

One special case occurs if you try to tune your algorithm by changing its parameters through multiple iterations. If this is the case you need to add the numeric field iteration to the algorithm.parameter list of the corresponding algorithm. It is important that no value occurs multiple times for the same combination of problem, algorithm and replication.

To see all tuning combinations in your data table just execute:

getTunings(data.table)

Input Data Types

There are several ways to load your benchmark data into the benchmarkVis application. These are:

  1. Create the table directly with R
  2. Use csv file
  3. Use json file
  4. Use a provided wrapper

1. Create table directly with R

You can create the data table directly in R:

library(data.table)
problem = c("Problem A", "Problem B", "Problem C")
problem.parameter = list(list(parameter.a = 2, parameter.b = "xyz"), list(parameter.a = 4), list())
algorithm = c("algorithm1", "algorithm2", "algorithm3")
algorithm.parameter = list(list(), list(parameter = "test"), list())
replication = c("none", "none", "none")
replication.parameter = list(list(), list(), list())
measure.abc = c(16, 23, 52)
list.measure = list(c(1, 2, 3), c(1, 2, 5), c(3, 5, 3))
dt = data.table(problem, problem.parameter, algorithm, algorithm.parameter, replication, replication.parameter, measure.abc, list.measure)

dt

To check if your created data table is really compatible with the benchmarkVis package it is recommended to use following method:

checkStructure(dt)

If it returns TRUE everything is fine else it will result in an error. For all other input strategies this structure check is executed automatically.

You can use the created data table with the shiny application by saving it as a csv, json or rds file. Just use:

csvExport(dt, file.path = "PATH")

or

jsonExport(dt, file.path = "PATH")

or

rdsExport(dt, file.path = "PATH")

2. Use csv file

A second option is to create your input benchmark result as a csv file. This is a very flexible way to use your benchmark result with the benchmarkVis package. If you choose this there are two things you have to consider:

An example input line would look as follows:

"problemA","list(para1= 'xyz', para2 = 15)","algorithm1","list(para1 = 'test')","none","list()",12,"c(1,2,3)"

If you are still not exactly sure how to design a compatible csv file take a look at the example file ml.example.csv at Link.

3. Use json file

Using a json file works similar to a csv file. One difference is that vectors can direclty be defined as json arrays instead of a string. The input parameter lists still have to be desinged as stated in "#2. Use csv file".

An example json entry would be (notice that all entries have to be inside a json array):

{
    "problem": "problemA",
    "problem.parameter": "list(para1= 'xyz', para2 = 15)",
    "algorithm": "algorithm1",
    "algorithm.parameter": "list(para1 = 'test')",
    "replication": "none",
    "replication.parameter": "list()",
    "measure.abc": 12,
    "list.measure": [1, 2, 3]
  }

You can find a complete json example at ml.example2.json at Link.

4. Use a provided wrapper

The last option is to use a R benchmark package for which a wrapper is provided. The wrappers will automatically transform a regular result of the specified package into the desired data table form. To see a full list of compatible wrappers use following function:

listWrappers()

If you want to use a benchmark result with the shiny application, you have to save the unwrapped result as a "RDS" file first.

saveRDS(table, file = "PATH/file.rds")

Then it can later be loaded with a file wrapper.

Alternatively you could use the wrapper and save the created table as a json or csv file.

Load Benchmark

As stated before, there are several ways to load your data into the benchmarkVis package.

Use a csv, json or Rds file

If you have a csv/json/Rds file which is structured as stated in 1. Data Structure you can simply use following command to load your benchmark result:

csvImport("PATH.TO.CSV.File")

or

jsonImport("PATH.TO.JSON.File")

or

rdsImport("PATH.TO.RDS.File")

After the import a structure check will be run to see if everything is correct.

Use a wrapper

To import a benchmark result created with a package for which a wrapper is provided you can simply use the corresponding method. One example (in this case the microbenchmark wrapper) would be:

library(microbenchmark)
x = runif(100)
benchmark = microbenchmark(sqrt(x), x ^ 0.5)

table = useMicrobenchmarkWrapper(benchmark)

table

You can get a full list of all wrappers with:

listWrappers()

All the *FileWrapper methods are used with an RDS file.

Also for this import strategy a structure check will be executed.

Use a R data.table

If you created your data table directly in R and already checked the structure with

checkStructure(dt)

you are ready to go.

Create Plots

The main feature of the package is to create different kinds of plots. These can roughly be devided into four groups:

Following list gives an overview about all implemented plots and their corresponding group.

| Measure Plot | List Plot | Parameter Plot | Tuning Plot | |---|---|---|---| | Box Plot | Box Plot | Box Plot | | | Scatter Plot | Scatter Plot | Scatter Plot | | | Density Plot | Density Plot | Density Plot | | | Dual Measure Plot | Dual Measure Plot | Dual Plot | | | Measure Matrix Plot | Measure Matrix Plot | Measure Matrix Plot | | | Density Rank Plot | Density Rank Plot | | | | Rank Matrix Bar Plot | Rank Matrix Bar Plot | | | | | Line Plot | | Line Plot | | Summary Plot | | | | | Parallel Coordinates Plot | | | | | Radar Plot | | | | | Rank Plot | | | | | Bar Plot | | | |

There is no guarantee that the list provided above is up to date. To see a full list with all possible charts use the method:

listPlots()

If you just want to get a list with the plots that are usable with your input data table use:

getValidPlots(dt)

Create Plot

The creation of a benchmark visualization is pretty simple. Examples:

createBarPlot(microbenchmark.example, "measure.mean")
createListLinePlot(microbenchmark.example, "list.values")

Plot parameters

If you don't know the needed arguments of a specific plot, take a look into the package documentation. As most of the plots are created with plotly you can interactivly change the visualization.

To get more information about the components of your data table execute following commands:

getMeasures(mlr.benchmark.example)
getLists(mlr.benchmark.example)
getMainColumns(mlr.benchmark.example)
getParameterColumns(mlr.benchmark.example)
getParameters(mlr.benchmark.example, "algorithm.parameter")
getTunings(mlr.tuning.example)

Shiny application

All the functionalities of the package are also provided from an interactive shiny application. Therefore simply load the benchmarkVis package and run the R command:

runShinyApp()

Alternatively - as a developer - open the ui.R or server.ui file from the shiny directory from within RStudio and click the "Run App" button in the upper right. If you do not use RStudio you can load the benchmarkVis package and run the R command shiny::runApp('shiny') in the console.

The shiny application is structured with tabs each having its own functionality. After the app has been started, Import tab is the first one you see. Here you can import your data. First select your import data type and then choose the corresponding csv, json or Rds file. When the upload is finished you can select the Data table tab in the menu on the left. There you can apply aggregation and transformation functions on table columns to transform the table to suit your needs regarding what exactly you want to analyse and plot. The transformed table can be saved as csv or PDF file to better share your results (you can also print out the table directly from the app).

Next you need to switch to the Plots tab, where you can create different charts, for both original data and your transformed table. After a plot type has been chosen and all parameters have been defined the plot will be created in the bottom of the screen. Through the usage of the plotly package there is a wide range of interaction possibilities. If you want to save a plot for a later comparison in the app you can do so by clicking Save this plot in 'Saved Plots'. Saved Plots and Compare Plots tabs allow for inspection of the pre-selected plots.

Wiki

If there are questions left, take a look into our Wiki. There you can find more tips to work with the benchmarkVis package. Should there be no answer to your question or do you find any errors in the software feel free to open an Issue.



collinleiber/benchmarkVis documentation built on May 28, 2019, 8:23 a.m.