performanceEstimation: An Infra-Structure for Performance Estimation of Predictive Models

An infra-structure for estimating the predictive performance of predictive models. In this context, it can also be used to compare and/or select among different alternative ways of solving one or more predictive tasks. The main goal of the package is to provide a generic infra-structure to estimate the values of different metrics of predictive performance using different estimation procedures. These estimation tasks can be applied to any solutions (workflows) to the predictive tasks. The package provides easy to use standard workflows that allow the usage of any available R modeling algorithm together with some pre-defined data pre-processing steps and also prediction post- processing methods. It also provides means for addressing issues related with the statistical significance of the observed differences.

AuthorLuis Torgo [aut, cre]
Date of publication2016-10-13 20:37:05
MaintainerLuis Torgo <ltorgo@dcc.fc.up.pt>
LicenseGPL (>= 2)
Version1.1.0
https://github.com/ltorgo/performanceEstimation

View on CRAN

Man pages

bootEstimates: Performance estimation using (e0 or .632) bootstrap

Bootstrap-class: Class "Bootstrap"

CDdiagram.BD: CD diagrams for the post-hoc Boferroni-Dunn test

CDdiagram.Nemenyi: CD diagrams for the post-hoc Nemenyi test

classificationMetrics: Calculate some standard classification evaluation metrics of...

ComparisonResults-class: Class "ComparisonResults"

CV-class: Class "CV"

cvEstimates: Performance estimation using cross validation

EstCommon-class: Class '"EstCommon"'

EstimationMethod-class: Class "EstimationMethod"

EstimationResults-class: Class "EstimationResults"

estimationSummary: Obtain a set of descriptive statistics of the scores of a...

EstimationTask-class: Class '"EstimationTask"'

getIterationsInfo: Obtaining the information returned by a workflow when applied...

getIterationsPreds: Obtaining the predictions returned by a workflow when applied...

getScores: Obtaining the metric scores on the different iterations for a...

getWorkflow: Obtain the workflow object corresponding to an ID

hldEstimates: Performance estimation using holdout and random resampling

Holdout-class: Class "Holdout"

is.classification: Check if a certain predictive task is a classification...

is.regression: Check if a certain predictive task is a regression problem

knnImp: Fill in NA values with the values of the nearest neighbours

LOOCV-class: Class "LOOCV"

loocvEstimates: Performance estimation using Leave One Out Cross Validation

mcEstimates: Performance estimation for time series prediction tasks using...

mergeEstimationRes: Merging several 'ComparisonResults' class objects

metricNames: The evaluation metrics estimated in an experiment

metricsSummary: Obtains a summary of the individual metric scores obtained by...

MonteCarlo-class: Class "MonteCarlo"

pairedComparisons: Statistical hypothesis testing on the observed paired...

performanceEstimation: Estimate the predictive performance of modeling alternatives...

PredTask-class: Class "PredTask"

rankWorkflows: Provide a ranking of workflows involved in an estimation...

regressionMetrics: Calculate some standard regression evaluation metrics of...

responseValues: Obtain the target variable values of a prediction task

results2table: Obtains a dplyr data frame table object containing all the...

runWorkflow: Run a workflow on a predictive task

signifDiffs: Obtains a list with the set of paired differences that are...

smote: SMOTE algorithm for unbalanced classification problems

standardPOST: A function for applying post-processing steps to the...

standardPRE: A function for applying data pre-processing steps

standardWF: A function implementing a standard workflow for prediction...

subset-methods: Methods for Function 'subset' in Package...

taskNames: The prediction tasks involved in an estimation experiment

timeseriesWF: A function implementing sliding and growing window standard...

topPerformer: Obtain the workflow that best performed in terms of a metric...

topPerformers: Obtain the best scores from a performance estimation...

Workflow-class: Class "Workflow"

workflowNames: The IDs of the workflows involved in an estimation experiment

workflowVariants: Generate (parameter) variants of a workflow

Functions

bootEstimates Man page
Bootstrap Man page
Bootstrap-class Man page
CDdiagram.BD Man page
CDdiagram.Nemenyi Man page
classificationMetrics Man page
ComparisonResults Man page
ComparisonResults-class Man page
CV Man page
CV-class Man page
cvEstimates Man page
EstCommon-class Man page
EstimationMethod Man page
EstimationMethod-class Man page
EstimationResults Man page
EstimationResults-class Man page
estimationSummary Man page
EstimationTask Man page
EstimationTask-class Man page
getIterationsInfo Man page
getIterationsPreds Man page
getScores Man page
getWorkflow Man page
hldEstimates Man page
Holdout Man page
Holdout-class Man page
is.classification Man page
is.regression Man page
knnImp Man page
LOOCV Man page
LOOCV-class Man page
loocvEstimates Man page
mcEstimates Man page
mergeEstimationRes Man page
metricNames Man page
metricsSummary Man page
MonteCarlo Man page
MonteCarlo-class Man page
pairedComparisons Man page
performanceEstimation Man page
plot,ComparisonResults-method Man page
plot,EstimationResults-method Man page
PredTask Man page
PredTask-class Man page
rankWorkflows Man page
regressionMetrics Man page
responseValues Man page
results2table Man page
runWorkflow Man page
show,Bootstrap-method Man page
show,ComparisonResults-method Man page
show,CV-method Man page
show,EstimationResults-method Man page
show,EstimationTask-method Man page
show,Holdout-method Man page
show,LOOCV-method Man page
show,MonteCarlo-method Man page
show,PredTask-method Man page
show,Workflow-method Man page
signifDiffs Man page
smote Man page
standardPOST Man page
standardPRE Man page
standardWF Man page
subset,ANY-method Man page
subset,ComparisonResults-method Man page
subset,ComparisonResults-method Man page
subset-methods Man page
summary,ComparisonResults-method Man page
summary,EstimationResults-method Man page
summary,Workflow-method Man page
taskNames Man page
timeseriesWF Man page
topPerformer Man page
topPerformers Man page
Workflow Man page
Workflow-class Man page
workflowNames Man page
workflowVariants Man page

Files

performanceEstimation
performanceEstimation/inst
performanceEstimation/inst/CITATION
performanceEstimation/inst/doc
performanceEstimation/inst/doc/performanceEstimation.pdf
performanceEstimation/inst/doc/performanceEstimation.html
performanceEstimation/inst/doc/performanceEstimation.R
performanceEstimation/inst/doc/performanceEstimation.Rmd
performanceEstimation/NAMESPACE
performanceEstimation/R
performanceEstimation/R/workflows.R performanceEstimation/R/resultsManipulation.R performanceEstimation/R/resultsAnalysis.R performanceEstimation/R/methods.R performanceEstimation/R/experiments.R performanceEstimation/R/classDefs.R performanceEstimation/R/smote.R performanceEstimation/R/evaluationMetrics.R
performanceEstimation/MD5
performanceEstimation/DESCRIPTION
performanceEstimation/man
performanceEstimation/man/workflowNames.Rd performanceEstimation/man/CV-class.Rd performanceEstimation/man/results2table.Rd performanceEstimation/man/CDdiagram.Nemenyi.Rd performanceEstimation/man/standardPOST.Rd performanceEstimation/man/responseValues.Rd performanceEstimation/man/bootEstimates.Rd performanceEstimation/man/standardPRE.Rd performanceEstimation/man/taskNames.Rd performanceEstimation/man/ComparisonResults-class.Rd performanceEstimation/man/runWorkflow.Rd performanceEstimation/man/signifDiffs.Rd performanceEstimation/man/timeseriesWF.Rd performanceEstimation/man/hldEstimates.Rd performanceEstimation/man/regressionMetrics.Rd performanceEstimation/man/topPerformers.Rd performanceEstimation/man/CDdiagram.BD.Rd performanceEstimation/man/mergeEstimationRes.Rd performanceEstimation/man/metricsSummary.Rd performanceEstimation/man/subset-methods.Rd performanceEstimation/man/classificationMetrics.Rd performanceEstimation/man/LOOCV-class.Rd performanceEstimation/man/metricNames.Rd performanceEstimation/man/getWorkflow.Rd performanceEstimation/man/getScores.Rd performanceEstimation/man/smote.Rd performanceEstimation/man/Holdout-class.Rd performanceEstimation/man/is.regression.Rd performanceEstimation/man/PredTask-class.Rd performanceEstimation/man/pairedComparisons.Rd performanceEstimation/man/standardWF.Rd performanceEstimation/man/EstimationTask-class.Rd performanceEstimation/man/MonteCarlo-class.Rd performanceEstimation/man/Bootstrap-class.Rd performanceEstimation/man/loocvEstimates.Rd performanceEstimation/man/workflowVariants.Rd performanceEstimation/man/cvEstimates.Rd performanceEstimation/man/knnImp.Rd performanceEstimation/man/mcEstimates.Rd performanceEstimation/man/getIterationsInfo.Rd performanceEstimation/man/topPerformer.Rd performanceEstimation/man/Workflow-class.Rd performanceEstimation/man/is.classification.Rd performanceEstimation/man/rankWorkflows.Rd performanceEstimation/man/EstimationMethod-class.Rd performanceEstimation/man/getIterationsPreds.Rd performanceEstimation/man/performanceEstimation.Rd performanceEstimation/man/EstimationResults-class.Rd performanceEstimation/man/EstCommon-class.Rd performanceEstimation/man/estimationSummary.Rd
performanceEstimation/CHANGES

Questions? Problems? Suggestions? or email at ian@mutexlabs.com.

Please suggest features or report bugs with the GitHub issue tracker.

All documentation is copyright its authors; we didn't write any of that.