generateCritDifferencesData: Generate data for critical-differences plot.

In guillermozbta/mir: Machine Learning in R

Description

Generates data that can be used to plot a critical differences plot. Computes the critical differences according to either the "Bonferroni-Dunn" test or the "Nemenyi" test.
"Bonferroni-Dunn" usually yields higher power as it does not compare all algorithms to each other, but all algorithms to a baseline instead.
Learners are drawn on the y-axis according to their average rank.
For test = "nemenyi" a bar is drawn, connecting all groups of not significantly different learners.
For test = "bd" an interval is drawn arround the algorithm selected as baseline. All learners within this interval are not signifcantly different from the baseline.
Calculation:

CD = q_alpha sqrt(k(k+1)/(6N))

Where q_α is based on the studentized range statistic. See references for details.

Usage

generateCritDifferencesData(bmr, measure = NULL, p.value = 0.05,
  baseline = NULL, test = "bd")

Arguments

bmr	[BenchmarkResult] Benchmark result.
measure	[Measure] Performance measure. Default is the first measure used in the benchmark experiment.
p.value	[numeric(1)] P-value for the critical difference. Default: 0.05
baseline	[character(1)]: [learner.id] Select a learner.id as baseline for the test = "bd" ("Bonferroni-Dunn") critical differences diagram.The critical difference Interval will then be positioned arround this learner. Defaults to best performing algorithm. For test = "nemenyi", no baseline is needed as it performs all pairwise comparisons.
test	[character(1)] Test for which the critical differences are computed. “bd” for the Bonferroni-Dunn Test, which is comparing all classifiers to a baseline, thus performing a comparison of one classifier to all others. Algorithms not connected by a single line are statistically different. then the baseline. “nemenyi” for the posthoc.friedman.nemenyi.test which is comparing all classifiers to each other. The null hypothesis that there is a difference between the classifiers can not be rejected for all classifiers that have a single grey bar connecting them.

Value

[critDifferencesData]. List containing:

data	[data.frame] containing the info for the descriptive part of the plot
friedman.nemenyi.test	[list] of class pairwise.htest contains the calculated posthoc.friedman.nemenyi.test
cd.info	[list] containing info on the critical difference and its positioning
baseline	baseline chosen for plotting
p.value	p.value used for the posthoc.friedman.nemenyi.test and for computation of the critical difference

See Also

Other generate_plot_data: generateCalibrationData, generateFeatureImportanceData, generateFilterValuesData, generateFunctionalANOVAData, generateLearningCurveData, generatePartialDependenceData, generateThreshVsPerfData, getFilterValues, plotFilterValues

Other benchmark: BenchmarkResult, batchmark, benchmark, convertBMRToRankMatrix, friedmanPostHocTestBMR, friedmanTestBMR, getBMRAggrPerformances, getBMRFeatSelResults, getBMRFilteredFeatures, getBMRLearnerIds, getBMRLearnerShortNames, getBMRLearners, getBMRMeasureIds, getBMRMeasures, getBMRModels, getBMRPerformances, getBMRPredictions, getBMRTaskDescs, getBMRTaskIds, getBMRTuneResults, plotBMRBoxplots, plotBMRRanksAsBarChart, plotBMRSummary, plotCritDifferences, reduceBatchmarkResults

guillermozbta/mir documentation built on May 11, 2019, 6:27 p.m.