performanceEstimation: An Infra-Structure for Performance Estimation of Predictive Models

An infra-structure for estimating the predictive performance of predictive models. In this context, it can also be used to compare and/or select among different alternative ways of solving one or more predictive tasks. The main goal of the package is to provide a generic infra-structure to estimate the values of different metrics of predictive performance using different estimation procedures. These estimation tasks can be applied to any solutions (workflows) to the predictive tasks. The package provides easy to use standard workflows that allow the usage of any available R modeling algorithm together with some pre-defined data pre-processing steps and also prediction post- processing methods. It also provides means for addressing issues related with the statistical significance of the observed differences.

Author
Luis Torgo [aut, cre]
Date of publication
2016-10-13 20:37:05
Maintainer
Luis Torgo <ltorgo@dcc.fc.up.pt>
License
GPL (>= 2)
Version
1.1.0
URLs

View on CRAN

Man pages

bootEstimates
Performance estimation using (e0 or .632) bootstrap
Bootstrap-class
Class "Bootstrap"
CDdiagram.BD
CD diagrams for the post-hoc Boferroni-Dunn test
CDdiagram.Nemenyi
CD diagrams for the post-hoc Nemenyi test
classificationMetrics
Calculate some standard classification evaluation metrics of...
ComparisonResults-class
Class "ComparisonResults"
CV-class
Class "CV"
cvEstimates
Performance estimation using cross validation
EstCommon-class
Class '"EstCommon"'
EstimationMethod-class
Class "EstimationMethod"
EstimationResults-class
Class "EstimationResults"
estimationSummary
Obtain a set of descriptive statistics of the scores of a...
EstimationTask-class
Class '"EstimationTask"'
getIterationsInfo
Obtaining the information returned by a workflow when applied...
getIterationsPreds
Obtaining the predictions returned by a workflow when applied...
getScores
Obtaining the metric scores on the different iterations for a...
getWorkflow
Obtain the workflow object corresponding to an ID
hldEstimates
Performance estimation using holdout and random resampling
Holdout-class
Class "Holdout"
is.classification
Check if a certain predictive task is a classification...
is.regression
Check if a certain predictive task is a regression problem
knnImp
Fill in NA values with the values of the nearest neighbours
LOOCV-class
Class "LOOCV"
loocvEstimates
Performance estimation using Leave One Out Cross Validation
mcEstimates
Performance estimation for time series prediction tasks using...
mergeEstimationRes
Merging several 'ComparisonResults' class objects
metricNames
The evaluation metrics estimated in an experiment
metricsSummary
Obtains a summary of the individual metric scores obtained by...
MonteCarlo-class
Class "MonteCarlo"
pairedComparisons
Statistical hypothesis testing on the observed paired...
performanceEstimation
Estimate the predictive performance of modeling alternatives...
PredTask-class
Class "PredTask"
rankWorkflows
Provide a ranking of workflows involved in an estimation...
regressionMetrics
Calculate some standard regression evaluation metrics of...
responseValues
Obtain the target variable values of a prediction task
results2table
Obtains a dplyr data frame table object containing all the...
runWorkflow
Run a workflow on a predictive task
signifDiffs
Obtains a list with the set of paired differences that are...
smote
SMOTE algorithm for unbalanced classification problems
standardPOST
A function for applying post-processing steps to the...
standardPRE
A function for applying data pre-processing steps
standardWF
A function implementing a standard workflow for prediction...
subset-methods
Methods for Function 'subset' in Package...
taskNames
The prediction tasks involved in an estimation experiment
timeseriesWF
A function implementing sliding and growing window standard...
topPerformer
Obtain the workflow that best performed in terms of a metric...
topPerformers
Obtain the best scores from a performance estimation...
Workflow-class
Class "Workflow"
workflowNames
The IDs of the workflows involved in an estimation experiment
workflowVariants
Generate (parameter) variants of a workflow

Files in this package

performanceEstimation
performanceEstimation/inst
performanceEstimation/inst/CITATION
performanceEstimation/inst/doc
performanceEstimation/inst/doc/performanceEstimation.pdf
performanceEstimation/inst/doc/performanceEstimation.html
performanceEstimation/inst/doc/performanceEstimation.R
performanceEstimation/inst/doc/performanceEstimation.Rmd
performanceEstimation/NAMESPACE
performanceEstimation/R
performanceEstimation/R/workflows.R
performanceEstimation/R/resultsManipulation.R
performanceEstimation/R/resultsAnalysis.R
performanceEstimation/R/methods.R
performanceEstimation/R/experiments.R
performanceEstimation/R/classDefs.R
performanceEstimation/R/smote.R
performanceEstimation/R/evaluationMetrics.R
performanceEstimation/MD5
performanceEstimation/DESCRIPTION
performanceEstimation/man
performanceEstimation/man/workflowNames.Rd
performanceEstimation/man/CV-class.Rd
performanceEstimation/man/results2table.Rd
performanceEstimation/man/CDdiagram.Nemenyi.Rd
performanceEstimation/man/standardPOST.Rd
performanceEstimation/man/responseValues.Rd
performanceEstimation/man/bootEstimates.Rd
performanceEstimation/man/standardPRE.Rd
performanceEstimation/man/taskNames.Rd
performanceEstimation/man/ComparisonResults-class.Rd
performanceEstimation/man/runWorkflow.Rd
performanceEstimation/man/signifDiffs.Rd
performanceEstimation/man/timeseriesWF.Rd
performanceEstimation/man/hldEstimates.Rd
performanceEstimation/man/regressionMetrics.Rd
performanceEstimation/man/topPerformers.Rd
performanceEstimation/man/CDdiagram.BD.Rd
performanceEstimation/man/mergeEstimationRes.Rd
performanceEstimation/man/metricsSummary.Rd
performanceEstimation/man/subset-methods.Rd
performanceEstimation/man/classificationMetrics.Rd
performanceEstimation/man/LOOCV-class.Rd
performanceEstimation/man/metricNames.Rd
performanceEstimation/man/getWorkflow.Rd
performanceEstimation/man/getScores.Rd
performanceEstimation/man/smote.Rd
performanceEstimation/man/Holdout-class.Rd
performanceEstimation/man/is.regression.Rd
performanceEstimation/man/PredTask-class.Rd
performanceEstimation/man/pairedComparisons.Rd
performanceEstimation/man/standardWF.Rd
performanceEstimation/man/EstimationTask-class.Rd
performanceEstimation/man/MonteCarlo-class.Rd
performanceEstimation/man/Bootstrap-class.Rd
performanceEstimation/man/loocvEstimates.Rd
performanceEstimation/man/workflowVariants.Rd
performanceEstimation/man/cvEstimates.Rd
performanceEstimation/man/knnImp.Rd
performanceEstimation/man/mcEstimates.Rd
performanceEstimation/man/getIterationsInfo.Rd
performanceEstimation/man/topPerformer.Rd
performanceEstimation/man/Workflow-class.Rd
performanceEstimation/man/is.classification.Rd
performanceEstimation/man/rankWorkflows.Rd
performanceEstimation/man/EstimationMethod-class.Rd
performanceEstimation/man/getIterationsPreds.Rd
performanceEstimation/man/performanceEstimation.Rd
performanceEstimation/man/EstimationResults-class.Rd
performanceEstimation/man/EstCommon-class.Rd
performanceEstimation/man/estimationSummary.Rd
performanceEstimation/CHANGES