performanceEstimation: An Infra-Structure for Performance Estimation of Predictive Models

An infra-structure for estimating the predictive performance of predictive models. In this context, it can also be used to compare and/or select among different alternative ways of solving one or more predictive tasks. The main goal of the package is to provide a generic infra-structure to estimate the values of different metrics of predictive performance using different estimation procedures. These estimation tasks can be applied to any solutions (workflows) to the predictive tasks. The package provides easy to use standard workflows that allow the usage of any available R modeling algorithm together with some pre-defined data pre-processing steps and also prediction post- processing methods. It also provides means for addressing issues related with the statistical significance of the observed differences.

AuthorLuis Torgo [aut, cre]
Date of publication2016-10-13 20:37:05
MaintainerLuis Torgo <ltorgo@dcc.fc.up.pt>
LicenseGPL (>= 2)
Version1.1.0
https://github.com/ltorgo/performanceEstimation

View on CRAN

Man pages

bootEstimates: Performance estimation using (e0 or .632) bootstrap

Bootstrap-class: Class "Bootstrap"

CDdiagram.BD: CD diagrams for the post-hoc Boferroni-Dunn test

CDdiagram.Nemenyi: CD diagrams for the post-hoc Nemenyi test

classificationMetrics: Calculate some standard classification evaluation metrics of...

ComparisonResults-class: Class "ComparisonResults"

CV-class: Class "CV"

cvEstimates: Performance estimation using cross validation

EstCommon-class: Class '"EstCommon"'

EstimationMethod-class: Class "EstimationMethod"

EstimationResults-class: Class "EstimationResults"

estimationSummary: Obtain a set of descriptive statistics of the scores of a...

EstimationTask-class: Class '"EstimationTask"'

getIterationsInfo: Obtaining the information returned by a workflow when applied...

getIterationsPreds: Obtaining the predictions returned by a workflow when applied...

getScores: Obtaining the metric scores on the different iterations for a...

getWorkflow: Obtain the workflow object corresponding to an ID

hldEstimates: Performance estimation using holdout and random resampling

Holdout-class: Class "Holdout"

is.classification: Check if a certain predictive task is a classification...

is.regression: Check if a certain predictive task is a regression problem

knnImp: Fill in NA values with the values of the nearest neighbours

LOOCV-class: Class "LOOCV"

loocvEstimates: Performance estimation using Leave One Out Cross Validation

mcEstimates: Performance estimation for time series prediction tasks using...

mergeEstimationRes: Merging several 'ComparisonResults' class objects

metricNames: The evaluation metrics estimated in an experiment

metricsSummary: Obtains a summary of the individual metric scores obtained by...

MonteCarlo-class: Class "MonteCarlo"

pairedComparisons: Statistical hypothesis testing on the observed paired...

performanceEstimation: Estimate the predictive performance of modeling alternatives...

PredTask-class: Class "PredTask"

rankWorkflows: Provide a ranking of workflows involved in an estimation...

regressionMetrics: Calculate some standard regression evaluation metrics of...

responseValues: Obtain the target variable values of a prediction task

results2table: Obtains a dplyr data frame table object containing all the...

runWorkflow: Run a workflow on a predictive task

signifDiffs: Obtains a list with the set of paired differences that are...

smote: SMOTE algorithm for unbalanced classification problems

standardPOST: A function for applying post-processing steps to the...

standardPRE: A function for applying data pre-processing steps

standardWF: A function implementing a standard workflow for prediction...

subset-methods: Methods for Function 'subset' in Package...

taskNames: The prediction tasks involved in an estimation experiment

timeseriesWF: A function implementing sliding and growing window standard...

topPerformer: Obtain the workflow that best performed in terms of a metric...

topPerformers: Obtain the best scores from a performance estimation...

Workflow-class: Class "Workflow"

workflowNames: The IDs of the workflows involved in an estimation experiment

workflowVariants: Generate (parameter) variants of a workflow

Files in this package

performanceEstimation
performanceEstimation/inst
performanceEstimation/inst/CITATION
performanceEstimation/inst/doc
performanceEstimation/inst/doc/performanceEstimation.pdf
performanceEstimation/inst/doc/performanceEstimation.html
performanceEstimation/inst/doc/performanceEstimation.R
performanceEstimation/inst/doc/performanceEstimation.Rmd
performanceEstimation/NAMESPACE
performanceEstimation/R
performanceEstimation/R/workflows.R performanceEstimation/R/resultsManipulation.R performanceEstimation/R/resultsAnalysis.R performanceEstimation/R/methods.R performanceEstimation/R/experiments.R performanceEstimation/R/classDefs.R performanceEstimation/R/smote.R performanceEstimation/R/evaluationMetrics.R
performanceEstimation/MD5
performanceEstimation/DESCRIPTION
performanceEstimation/man
performanceEstimation/man/workflowNames.Rd performanceEstimation/man/CV-class.Rd performanceEstimation/man/results2table.Rd performanceEstimation/man/CDdiagram.Nemenyi.Rd performanceEstimation/man/standardPOST.Rd performanceEstimation/man/responseValues.Rd performanceEstimation/man/bootEstimates.Rd performanceEstimation/man/standardPRE.Rd performanceEstimation/man/taskNames.Rd performanceEstimation/man/ComparisonResults-class.Rd performanceEstimation/man/runWorkflow.Rd performanceEstimation/man/signifDiffs.Rd performanceEstimation/man/timeseriesWF.Rd performanceEstimation/man/hldEstimates.Rd performanceEstimation/man/regressionMetrics.Rd performanceEstimation/man/topPerformers.Rd performanceEstimation/man/CDdiagram.BD.Rd performanceEstimation/man/mergeEstimationRes.Rd performanceEstimation/man/metricsSummary.Rd performanceEstimation/man/subset-methods.Rd performanceEstimation/man/classificationMetrics.Rd performanceEstimation/man/LOOCV-class.Rd performanceEstimation/man/metricNames.Rd performanceEstimation/man/getWorkflow.Rd performanceEstimation/man/getScores.Rd performanceEstimation/man/smote.Rd performanceEstimation/man/Holdout-class.Rd performanceEstimation/man/is.regression.Rd performanceEstimation/man/PredTask-class.Rd performanceEstimation/man/pairedComparisons.Rd performanceEstimation/man/standardWF.Rd performanceEstimation/man/EstimationTask-class.Rd performanceEstimation/man/MonteCarlo-class.Rd performanceEstimation/man/Bootstrap-class.Rd performanceEstimation/man/loocvEstimates.Rd performanceEstimation/man/workflowVariants.Rd performanceEstimation/man/cvEstimates.Rd performanceEstimation/man/knnImp.Rd performanceEstimation/man/mcEstimates.Rd performanceEstimation/man/getIterationsInfo.Rd performanceEstimation/man/topPerformer.Rd performanceEstimation/man/Workflow-class.Rd performanceEstimation/man/is.classification.Rd performanceEstimation/man/rankWorkflows.Rd performanceEstimation/man/EstimationMethod-class.Rd performanceEstimation/man/getIterationsPreds.Rd performanceEstimation/man/performanceEstimation.Rd performanceEstimation/man/EstimationResults-class.Rd performanceEstimation/man/EstCommon-class.Rd performanceEstimation/man/estimationSummary.Rd
performanceEstimation/CHANGES

Questions? Problems? Suggestions? or email at ian@mutexlabs.com.

Please suggest features or report bugs with the GitHub issue tracker.

All documentation is copyright its authors; we didn't write any of that.