BenchmarkAggr | R Documentation |
An R6 class for aggregated benchmark results.
This class is used to easily carry out and guide analysis of models after aggregating
the results after resampling. This can either be constructed using mlr3 objects,
for example the result of mlr3::BenchmarkResult$aggregate
or via as_benchmark_aggr,
or by passing in a custom dataset of results. Custom datasets must include at the very least,
a character column for learner ids, a character column for task ids, and numeric columns for
one or more measures.
Currently supported for multiple independent datasets only.
data
(data.table::data.table)
Aggregated data.
learners
(character())
Unique learner names.
tasks
(character())
Unique task names.
measures
(character())
Unique measure names.
nlrns
(integer())
Number of learners.
ntasks
(integer())
Number of tasks.
nmeas
(integer())
Number of measures.
nrow
(integer())
Number of rows.
col_roles
(character()
)
Column roles, currently cannot be changed after construction.
new()
Creates a new instance of this R6 class.
BenchmarkAggr$new( dt, task_id = "task_id", learner_id = "learner_id", independent = TRUE, strip_prefix = TRUE, ... )
dt
(matrix(1))
'
matrix
like object coercable to data.table::data.table, should
include column names "task_id" and "learner_id", and at least one measure (numeric).
If ids are not already factors then coerced internally.
task_id
(character(1)
)
String specifying name of task id column.
learner_id
(character(1)
)
String specifying name of learner id column.
independent
(logical(1))
Are tasks independent of one another? Affects which tests can be used for analysis.
strip_prefix
(logical(1)
)
If TRUE
(default) then mlr prefixes, e.g. regr.
, classif.
, are automatically
stripped from the learner_id
.
...
ANY
Additional arguments, currently unused.
print()
Prints the internal data via data.table::print.data.table.
BenchmarkAggr$print(...)
...
ANY
Passed to data.table::print.data.table.
summary()
Prints the internal data via data.table::print.data.table.
BenchmarkAggr$summary(...)
...
ANY
Passed to data.table::print.data.table.
rank_data()
Ranks the aggregated data given some measure.
BenchmarkAggr$rank_data(meas = NULL, minimize = TRUE, task = NULL, ...)
meas
(character(1))
Measure to rank the data against, should be in $measures
. Can be NULL
if only one measure
in data.
minimize
(logical(1))
Should the measure be minimized? Default is TRUE
.
task
(character(1))
If NULL
then returns a matrix of ranks where columns are tasks and rows are
learners, otherwise returns a one-column matrix of a specified task, should
be in $tasks
.
...
ANY
ANY
Passed to data.table::frank()
.
friedman_test()
Computes Friedman test over all tasks, assumes datasets are independent.
BenchmarkAggr$friedman_test(meas = NULL, p.adjust.method = NULL)
meas
(character(1))
Measure to rank the data against, should be in $measures
. If no measure is provided
then returns a matrix of tests for all measures.
p.adjust.method
(character(1))
Passed to p.adjust if meas = NULL
for multiple testing correction. If NULL
then no correction applied.
friedman_posthoc()
Posthoc Friedman Nemenyi tests. Computed with
PMCMRplus::frdAllPairsNemenyiTest. If global $friedman_test
is non-significant then
this is returned and no post-hocs computed. Also returns critical difference
BenchmarkAggr$friedman_posthoc( meas = NULL, p.value = 0.05, friedman_global = TRUE )
meas
(character(1))
Measure to rank the data against, should be in $measures
. Can be NULL
if only one measure
in data.
p.value
(numeric(1))
p.value for which the global test will be considered significant.
friedman_global
(logical(1)
)
Should a friedman global test be performed before conducting the posthoc
test? If FALSE
, a warning is issued in case the corresponding friedman
global test fails instead of an error. Default is TRUE
(raises an
error if global test fails).
subset()
Subsets the data by given tasks or learners. Returns data as data.table::data.table.
BenchmarkAggr$subset(task = NULL, learner = NULL)
task
(character()
)
Task(s) to subset the data by.
learner
(character()
)
Learner(s) to subset the data by.
clone()
The objects of this class are cloneable with this method.
BenchmarkAggr$clone(deep = FALSE)
deep
Whether to make a deep clone.
'r format_bib("demsar_2006")
# Not restricted to mlr3 objects
df = data.frame(tasks = factor(rep(c("A", "B"), each = 5),
levels = c("A", "B")),
learners = factor(paste0("L", 1:5)),
RMSE = runif(10), MAE = runif(10))
as_benchmark_aggr(df, task_id = "tasks", learner_id = "learners")
if (requireNamespaces(c("mlr3", "rpart"))) {
library(mlr3)
task = tsks(c("boston_housing", "mtcars"))
learns = lrns(c("regr.featureless", "regr.rpart"))
bm = benchmark(benchmark_grid(task, learns, rsmp("cv", folds = 2)))
# coercion
as_benchmark_aggr(bm)
}
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.