mmetric: Compute classification or regression error metrics.

Description Usage Arguments Details Value Note Author(s) References See Also Examples

View source: R/metrics.R

Description

Compute classification or regression error metrics.

Usage

1
mmetric(y, x = NULL, metric, D = 0.5, TC = -1, val = NULL, aggregate = "no")

Arguments

y

if there are predictions (!is.null(x)), y should be a numeric vector or factor with the target desired responses (or output values).
Else, y should be a list returned by the mining function.

x

the predictions (should be a numeric vector if task="reg", matrix if task="prob" or factor if task="class" (used if y is not a list).

metric

a R function or a character.
Note: if a R function, then it should be set to provide lower values for better models if the intention is to be used within the search argument of fit and mining (i.e., "<" meaning).
Valid character options are (">" means "better" if higher value; "<" means "better" if lower value):

  • ALL – returns all classification or regression metrics (context dependent, multi-metric).

  • if vector – returns all metrics included in the vector, vector elements can be any of the options below (multi-metric).

  • CONF – confusion matrix (classification, matrix).

  • ACC – classification accuracy rate (classification, ">", [0-%100]).

  • CE – classification error or misclassification error rate (classification, "<", [0-%100]).

  • MAEO – mean absolute error for ordinal classification (classification, "<", [0-Inf[).

  • MSEO – mean squared error for ordinal classification (classification, "<", [0-Inf[).

  • KENDALL – Kendalls's coefficient for ordinal classification or (mean if) ranking (classification, ">", [-1;1]). Note: if ranking, y is a matrix and mean metric is computed.

  • SPEARMAN – Mean Spearman's rho coefficient for ranking (classification, ">", [-1;1]). Note: if ranking, y is a matrix and mean metric is computed.

  • BER – balanced error rate (classification, "<", [0-%100]).

  • KAPPA – kappa index (classification, "<", [0-%100]).

  • CRAMERV – Cramer's V (classification, ">", [0,1.0]).

  • ACCLASS – classification accuracy rate per class (classification, ">", [0-%100]).

  • TPR – true positive rate, sensitivity or recall (classification, ">", [0-%100]).

  • TNR – true negative rate or specificity (classification, ">", [0-%100]).

  • PRECISION – precision (classification, ">", [0-%100]).

  • F1 – F1 score (classification, ">", [0-%100]).

  • MCC – Matthews correlation coefficient (classification, ">", [-1,1]).

  • BRIER – overall Brier score (classification "prob", "<", [0,1.0]).

  • BRIERCLASS – Brier score per class (classification "prob", "<", [0,1.0]).

  • ROC – Receiver Operating Characteristic curve (classification "prob", list with several components).

  • AUC – overall area under the curve (of ROC curve, classification "prob", ">", domain values: [0,1.0]).

  • AUCCLASS – area under the curve per class (of ROC curve, classification "prob", ">", domain values: [0,1.0]).

  • NAUC – normalized AUC (given a fixed val=FPR, classification "prob", ">", [0,1.0]).

  • TPRATFPR – the TPR (given a fixed val=FPR, classification "prob", ">", [0,1.0]).

  • LIFT – accumulative percent of responses captured (LIFT accumulative curve, classification "prob", list with several components).

  • ALIFT – area of the accumulative percent of responses captured (LIFT accumulative curve, classification "prob", ">", [0,1.0]).

  • NALIFT – normalized ALIFT (given a fixed val=percentage of examples, classification "prob", ">", [0,1.0]).

  • ALIFTATPERC – ALIFT value (given a fixed val=percentage of examples, classification "prob", ">", [0,1.0]).

  • SAE – sum absolute error/deviation (regression, "<", [0,Inf[).

  • MAE – mean absolute error (regression, "<", [0,Inf[).

  • MdAE – median absolute error (regression, "<", [0,Inf[).

  • GMAE – geometric mean absolute error (regression, "<", [0,Inf[).

  • MaxAE – maximum absolute error (regression, "<", [0,Inf[).

  • NMAE – normalized mean absolute error (regression, "<", [0%,Inf[). Note: by default, this metric assumes the range of y as the denominator of NMAE; a different range can be set by setting the optional val argument (see example).

  • RAE – relative absolute error (regression, "<", [0%,Inf[).

  • SSE – sum squared error (regression, "<", [0,Inf[).

  • MSE – mean squared error (regression, "<", [0,Inf[).

  • MdSE – median squared error (regression, "<", [0,Inf[).

  • RMSE – root mean squared error (regression, "<", [0,Inf[).

  • GMSE – geometric mean squared error (regression, "<", [0,Inf[).

  • HRMSE – Heteroscedasticity consistent root mean squared error (regression, "<", [0,Inf[).

  • RSE – relative squared error (regression, "<", [0%,Inf[).

  • RRSE – root relative squared error (regression, "<", [0%,Inf[).

  • ME – mean error (regression, "<", [0,Inf[).

  • SMinkowski3 – sum of Minkowski loss function (q=3, heavier penalty for large errors when compared with SSE, regression, "<", [0%,Inf[).

  • MMinkowski3 – mean of Minkowski loss function (q=3, heavier penalty for large errors when compared with SSE, regression, "<", [0%,Inf[).

  • MdMinkowski3 – median of Minkowski loss function (q=3, heavier penalty for large errors when compared with SSE, regression, "<", [0%,Inf[).

  • COR – correlation (regression, ">", [-1,1]).

  • q2 – =1-correlation^2 test error metric, as used by M.J. Embrechts (regression, "<", [0,1.0]).

  • R2 – coefficient of determination R^2 (regression, ">", squared pearson correlation coefficient: [0,1]).

  • R22 – 2nd variant of coefficient of determination R^2 (regression, ">", most general definition that however can lead to negative values: ]-Inf,1]. In previous rminer versions, this variant was known as "R2").

  • Q2 – R^2/SD test error metric, as used by M.J. Embrechts (regression, "<", [0,Inf[).

  • REC – Regression Error Characteristic curve (regression, list with several components).

  • NAREC – normalized REC area (given a fixed val=tolerance, regression, ">", [0,1.0]).

  • TOLERANCE – the tolerance (y-axis value) of a REC curve (given a fixed val=tolerance, regression, ">", [0,1.0]).

  • MAPE – Mean Absolute Percentage mmetric forecasting metric (regression, "<", [0%,Inf[).

  • MdAPE – Median Absolute Percentage mmetric forecasting metric (regression, "<"), [0%,Inf[).

  • RMSPE – Root Mean Square Percentage mmetric forecasting metric (regression, "<", [0%,Inf[).

  • RMdSPE – Root Median Square Percentage mmetric forecasting metric (regression, "<", [0%,Inf[).

  • SMAPE – Symmetric Mean Absolute Percentage mmetric forecasting metric (regression, "<", [0%,200%]).

  • SMdAPE – Symmetric Median Absolute Percentage mmetric forecasting metric (regression, "<", [0%,200%]).

  • MRAE – Mean Relative Absolute mmetric forecasting metric (val should contain the last in-sample/training data value (for random walk) or full benchmark time series related with out-of-sample values, regression, "<", [0,Inf[).

  • MdRAE – Median Relative Absolute mmetric forecasting metric (val should contain the last in-sample/training data value (for random walk) or full benchmark time series, regression, "<", [0,Inf[).

  • GMRAE – Geometric Mean Relative Absoluate mmetric forecasting metric (val should contain the last in-sample/training data value (for random walk) or full benchmark time series, regression, "<", [0,Inf[).

  • THEILSU2 – Theils'U2 forecasting metric (val should contain the last in-sample/training data value (for random walk) or full benchmark time series, regression, "<", [0,Inf[).

  • MASE – MASE forecasting metric (val should contain the time series in-samples or training data, regression, "<", [0,Inf[).

D

decision threshold (for task="prob", probabilistic classification) within [0,1]. The class is TRUE if prob>D.

TC

target class index or vector of indexes (for multi-class classification class) within 1,...,Nc, where Nc is the number of classes:<cr>

  • if TC==-1 (the default value), then it is assumed:

    • if metric is "CONF" – D is ignored and highest probability class is assumed (if TC>0, the metric is computed for positive TC class and D is used).

    • if metric is "ACC", "CE", "BER", "KAPPA", "CRAMERV", "BRIER", or "AUC" – the global metric (for all classes) is computed (if TC>0, the metric is computed for positive TC class).

    • if metric is "ACCLASS", "TPR", "TNR", "Precision", "F1", "MCC", "ROC", "BRIERCLASS", "AUCCLASS" – it returns one result per class (if TC>0, it returns negative (e.g. "TPR1") and positive (TC, e.g. "TPR2") result).

    • if metric is "NAUC", "TPRATFPR", "LIFT", "ALIFT", "NALIFT" or "ALIFTATPERC" – TC is set to the index of the last class.

val

auxiliary value:

  • when two or more metrics need different val values, then val should be a vector list, see example.

  • if numeric or vector – check the metric argument for specific details of each metric val meaning.

aggregate

character with type of aggregation performed when y is a mining list. Valid options are:

  • no – returns all metrics for all mining runs. If metric includes "CONF", "ROC", "LIFT" or "REC", it returns a vector list, else if metric includes a single metric, it returns a vector; else it retuns a data.frame (runs x metrics).

  • sum – sums all run results.

  • mean – averages all run results.

  • note: both "sum" and "mean" only work if only metric=="CONF" is used or if metric does not contain "ROC", "LIFT" or "REC".

Details

Compute classification or regression error metrics:

Value

Returns the computed error metric(s):

Note

See also http://hdl.handle.net/1822/36210 and http://www3.dsi.uminho.pt/pcortez/rminer.html

Author(s)

Paulo Cortez http://www3.dsi.uminho.pt/pcortez

References

See Also

fit, predict.fit, mining, mgraph, savemining and Importance.

Examples

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
### regression examples: y - desired values; x - predictions
y=c(95.01,96.1,97.2,98.0,99.3,99.7);x=95:100
print(mmetric(y,x,"ALL"))
print(mmetric(y,x,"MAE"))
m=mmetric(y,x,c("MAE","RMSE","RAE","RSE"))
print(m)
# getting NMAE:
m=mmetric(y,x,"NMAE")
cat("NMAE:",round(m,digits=3),"(denominator=",diff(range(y)),")\n")
m=mmetric(y,x,"NMAE",val=5) # usage of different range
cat("NMAE:",round(m,digits=3),"(denominator=",5,")\n")

cat(names(m)[3],"=",round(m[3],digit=2),"\n",sep="")
print(mmetric(y,x,c("COR","R2","Q2")))
print(mmetric(y,x,c("TOLERANCE","NAREC"),val=0.5)) # if admitted/accepted absolute error is 0.5
print(mmetric(y,x,"THEILSU2",val=94.1)) # val = 1-ahead random walk, c(y,94.1), same as below
print(mmetric(y,x,"THEILSU2",val=c(94.1,y[1:5]))) # val = 1-ahead random walk (previous y values)
print(mmetric(y,x,"MASE",val=c(88.1,89.9,93.2,94.1))) # val = in-samples
val=vector("list",length=4)
val[[2]]=0.5;val[[3]]=94.1;val[[4]]=c(88.1,89.9,93.2,94.1)
print(mmetric(y,x,c("MAE","NAREC","THEILSU2","MASE"),val=val))
# user defined error function example:
# myerror = number of samples with absolute error above 0.1% of y: 
myerror=function(y,x){return (sum(abs(y-x)>(0.001*y)))}
print(mmetric(y,x,metric=myerror))
# example that returns a list since "REC" is included:
print(mmetric(y,x,c("MAE","REC","TOLERANCE"),val=1))

### pure binary classification 
y=factor(c("a","a","a","a","b","b","b","b"))
x=factor(c("a","a","b","a","b","a","b","a"))
print(mmetric(y,x,"CONF")$conf)
print(mmetric(y,x,"ALL"))
print(mmetric(y,x,metric=c("ACC","TPR","ACCLASS")))

### probabilities binary classification 
y=factor(c("a","a","a","a","b","b","b","b"))
px=matrix(nrow=8,ncol=2)
px[,1]=c(1.0,0.9,0.8,0.7,0.6,0.5,0.4,0.3)
px[,2]=1-px[,1]
print(px)
print(mmetric(y,px,"CONF")$conf)
print(mmetric(y,px,"CONF",D=0.5,TC=2)$conf)
print(mmetric(y,px,"CONF",D=0.3,TC=2)$conf)
print(mmetric(y,px,metric="ALL",D=0.3,TC=2))
print(mmetric(y,px,metric=c("ACC","AUC","AUCCLASS","BRIER","BRIERCLASS","CE"),D=0.3,TC=2))

### pure multi-class classification 
y=factor(c("a","a","b","b","c","c"))
x=factor(c("a","a","b","c","b","c"))
print(mmetric(y,x,metric="CONF")$conf)
print(mmetric(y,x,metric="CONF",TC=-1)$conf)
print(mmetric(y,x,metric="CONF",TC=1)$conf)
print(mmetric(y,x,metric="ALL"))
print(mmetric(y,x,metric=c("ACC","ACCLASS","KAPPA")))
print(mmetric(y,x,metric=c("ACC","ACCLASS","KAPPA"),TC=1))

### probabilities multi-class 
y=factor(c("a","a","b","b","c","c"))
px=matrix(nrow=6,ncol=3)
px[,1]=c(1.0,0.7,0.5,0.3,0.1,0.7)
px[,2]=c(0.0,0.2,0.4,0.7,0.3,0.2)
px[,3]=1-px[,1]-px[,2]
print(px)
print(mmetric(y,px,metric=c("AUC","AUCCLASS","NAUC"),TC=-1,val=0.1))
print(mmetric(y,px,metric=c("AUC","NAUC"),TC=3,val=0.1))
print(mmetric(y,px,metric=c("ACC","ACCLASS"),TC=-1))
print(mmetric(y,px,metric=c("CONF"),TC=3,D=0.5)$conf)
print(mmetric(y,px,metric=c("ACCLASS"),TC=3,D=0.5))
print(mmetric(y,px,metric=c("CONF"),TC=3,D=0.7)$conf)
print(mmetric(y,px,metric=c("ACCLASS"),TC=3,D=0.7))

### ordinal multi-class (example in Ricardo Sousa PhD thesis 2012)
y=ordered(c(rep("a",4),rep("b",6),rep("d",3)),levels=c("a","b","c","d"))
x=ordered(c(rep("c",(4+6)),rep("d",3)),levels=c("a","b","c","d"))
print(mmetric(y,x,metric="CONF")$conf)
print(mmetric(y,x,metric=c("CE","MAEO","MSEO","KENDALL")))
# note: only y needs to be ordered
x=factor(c(rep("b",4),rep("a",6),rep("d",3)),levels=c("a","b","c","d"))
print(mmetric(y,x,metric="CONF")$conf)
print(mmetric(y,x,metric=c("CE","MAEO","MSEO","KENDALL")))

### ranking (multi-class) 
y=matrix(nrow=1,ncol=12);x=y
# http://www.youtube.com/watch?v=D56dvoVrBBE
y[1,]=1:12
x[1,]=c(2,1,4,3,6,5,8,7,10,9,12,11)
print(mmetric(y,x,metric="KENDALL"))
print(mmetric(y,x,metric="ALL"))

y=matrix(nrow=2,ncol=7);x=y
y[1,]=c(2,6,5,4,3,7,1)
y[2,]=7:1
x[1,]=1:7
x[2,]=1:7
print(mmetric(y,x,metric="ALL"))

### mining, several runs, prob multi-class
## Not run: 
data(iris)
M=mining(Species~.,iris,model="rpart",Runs=2)
R=mmetric(M,metric="CONF",aggregate="no")
print(R[[1]]$conf)
print(R[[2]]$conf)
print(mmetric(M,metric="CONF",aggregate="mean"))
print(mmetric(M,metric="CONF",aggregate="sum"))
print(mmetric(M,metric=c("ACC","ACCLASS"),aggregate="no"))
print(mmetric(M,metric=c("ACC","ACCLASS"),aggregate="mean"))
print(mmetric(M,metric="ALL",aggregate="no"))
print(mmetric(M,metric="ALL",aggregate="mean"))

## End(Not run)

### mining, several runs, regression
## Not run: 
data(sin1reg)
S=sample(1:nrow(sin1reg),40)
M=mining(y~.,data=sin1reg[S,],model="ksvm",search=2^3,Runs=10)
R=mmetric(M,metric="MAE")
print(mmetric(M,metric="MAE",aggregate="mean"))
miR=meanint(R) # mean and t-student confidence intervals
cat("MAE=",round(miR$mean,digits=2),"+-",round(miR$int,digits=2),"\n")
print(mmetric(M,metric=c("MAE","RMSE")))
print(mmetric(M,metric=c("MAE","RMSE"),aggregate="mean"))
R=mmetric(M,metric="REC",aggregate="no")
print(R[[1]]$rec)
print(mmetric(M,metric=c("TOLERANCE","NAREC"),val=0.2))
print(mmetric(M,metric=c("TOLERANCE","NAREC"),val=0.2,aggregate="mean"))

## End(Not run)

rminer documentation built on May 1, 2019, 7:48 p.m.