ScorlogPFS: ScorlogPFS
In GabrieleDAmore/ForecastErrorParadox: ForecastErrorParadox package
Description
Usage
Arguments
Value
Note
Author(s)
Source
Examples
Description
Diebold-Mariano Test (1995) to statistically test forecast accuracy equivalence for 2 sets of predictions belonging to the Paradox Free Space (see D'Amore (2020))
Usage
1
ScorlogPFS(a, b, h, DATA, p, q, meanp, stdp, meanq, stdq)
Arguments
a
Single numerical value indicating the avoidable forecast error of model Y (see example 3.2 on D'Amore (2020)).
b
Single numerical value indicating the avoidable forecast error of model Z (see example 3.2 on D'Amore (2020)).
h
It's the the kernel bandwidth smoothing parameter of the uniform kernel density function employed for modeling training data.
DATA
Numeric vector of i.i.d. observations used as test dataset.
p
It's the root name of the guessed density function (model Y) for modeling the training dataset (it must be dnorm for ForecastErrorParadox version 0.1.0).
q
It's the root name of the guessed density function (model Z) for modeling the training dataset (it must be dnorm for ForecastErrorParadox version 0.1.0).
meanp
Mean of p.
stdp
Standard deviation of p.
meanq
Mean of q.
stdq
Standard deviation of p.
Value
Diebold-Mariano Test (1995).
AvWScY = Average Weighted Scoring rule of model Y.
AvWScZ = Average Weighted Scoring rule of model Z.
checkdataExAnte = Table with data before imputation.
checkdataExPost = Table with data after imputation.
WScoreYPFS = Weighted Scoring rule of model Y computed with the testing data (see equation 13 in D'Amore(2020)).
WScoreZPFS = Weighted Scoring rule of model Z computed with the testing data (see equation 13 in D'Amore(2020)).
lenghtexpost = length of the sample before imputation.
TestStat = Diebold-Mariano Test Statistic.
TestRes = Test result.
pval = p-value of the classic Diebold-Mariano Test (1995).
dmtest2sid= Two-sided Modified Diebold-Mariano Test. The alternative hypothesis is that method 1 and method 2 have different levels of accuracy.
dmtestgre= One-sided Modified Diebold-Mariano Test. The alternative hypothesis is that method 2 is more accurate than method 1.
dmtestles= One-sided Modified Diebold-Mariano Test. The alternative hypothesis is that method 2 is less accurate than method 1.
h= adaptive bandwidth of the kernel density function.
Note
During the computation, R tends to approximate very small logarithmic values with non-finite values. For this reason, we replace such kind of approximations with finite values obtained by imputation method.
Author(s)
Gabriele D'Amore
Source
Gneiting, T. and A.E. Raftery (2007): ‘Strictly proper scoring rules, prediction and estimation’, Journal of the American Statistical Association 102, 359-378.
Diebold, F. X. (2015). Comparing predictive accuracy, twenty years later: A personal perspective on the use and abuse of Diebold–Mariano tests. Journal of Business & Economic Statistics, 33(1), 1-1.
Harvey, D., Leybourne, S., & Newbold, P. (1997). Testing the equality of prediction mean squared errors. International Journal of forecasting, 13(2), 281-291.
D'Amore, G. (2020): ‘When Scoring Rules fall into an informative paradox’, (Forthcoming).
Examples
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
set.seed(8)
library(forecast)
library(Hmisc)
library(missForest)
DATA = rnorm(100, mean = 0, sd = 1)
h<-bw.ucv(DATA)
######(Ex.1)######
a=-1
b=1
scoretest<-ScorlogPFS(a,b,h,DATA, dnorm, dnorm, meanp = -1, stdp = 1, meanq =1, stdq=1)
#p-value of the classic Diebold-Mariano Test (1995)
scoretest$pval
#Average of the difference of the Weighted Scoring rules of the two models (Y and Z)
scoretest$AvWScY-scoretest$AvWScZ
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 1 and method 2 have different levels of accuracy.
scoretest$dmtest2sid
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 2 is more accurate than method 1
scoretest$dmtestgre
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 2 is less accurate than method 1
scoretest$dmtestles
######(Ex.2)######
a=0.8
b=3
scoretest<-ScorlogPFS(a,b,h,DATA, dnorm, dnorm, meanp = 0.8, stdp = 1, meanq =3, stdq=1)
#p-value of the classic Diebold-Mariano Test (1995)
scoretest$pval
#Average of the difference of the Weighted Scoring rules of the two models (Y and Z)
scoretest$AvWScY-scoretest$AvWScZ
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 1 and method 2 have different levels of accuracy.
scoretest$dmtest2sid
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 2 is more accurate than method 1
scoretest$dmtestgre
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 2 is less accurate than method 1
scoretest$dmtestles
######(Ex.3)######
a=0.4
b=-2
scoretest<-ScorlogPFS(a,b,h,DATA, dnorm, dnorm, meanp = 0.4, stdp = 0.3, meanq =-2, stdq=5)
#p-value of the classic Diebold-Mariano Test (1995)
scoretest$pval
#Average of the difference of the Weighted Scoring rules of the two models (Y and Z)
scoretest$AvWScY-scoretest$AvWScZ
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 1 and method 2 have different levels of accuracy.
scoretest$dmtest2sid
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 2 is more accurate than method 1
scoretest$dmtestgre
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 2 is less accurate than method 1
scoretest$dmtestles
######(Ex.4)######
a=(2)^(1/2)*(- log(5/9))^(1/2)/2
b=0
scoretest<-ScorlogPFS(a,b,h,DATA, dnorm, dnorm, meanp = (2)^(1/2)*(- log(5/9))^(1/2)/2, stdp = 1, meanq =0, stdq=3)
#p-value of the classic Diebold-Mariano Test (1995)
scoretest$pval
#Average of the difference of the Weighted Scoring rules of the two models (Y and Z)
scoretest$AvWScY-scoretest$AvWScZ
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 1 and method 2 have different levels of accuracy.
scoretest$dmtest2sid
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 2 is more accurate than method 1
scoretest$dmtestgre
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 2 is less accurate than method 1
scoretest$dmtestles
######(Ex.5)######
a=0
b=0
scoretest<-ScorlogPFS(a,b,h,DATA, dnorm, dnorm, meanp = 0, stdp = 2, meanq =0, stdq=0.5)
#p-value of the classic Diebold-Mariano Test (1995)
scoretest$pval
#Average of the difference of the Weighted Scoring rules of the two models (Y and Z)
scoretest$AvWScY-scoretest$AvWScZ
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 1 and method 2 have different levels of accuracy.
scoretest$dmtest2sid
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 2 is more accurate than method 1
scoretest$dmtestgre
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 2 is less accurate than method 1
scoretest$dmtestles
GabrieleDAmore/ForecastErrorParadox documentation built on Oct. 20, 2020, 1:18 a.m.
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Description Usage Arguments Value Note Author(s) Source Examples
Description
Diebold-Mariano Test (1995) to statistically test forecast accuracy equivalence for 2 sets of predictions belonging to the Paradox Free Space (see D'Amore (2020))
Usage
1 | ScorlogPFS(a, b, h, DATA, p, q, meanp, stdp, meanq, stdq)
|
Arguments
a |
Single numerical value indicating the avoidable forecast error of model Y (see example 3.2 on D'Amore (2020)). |
b |
Single numerical value indicating the avoidable forecast error of model Z (see example 3.2 on D'Amore (2020)). |
h |
It's the the kernel bandwidth smoothing parameter of the uniform kernel density function employed for modeling training data. |
DATA |
Numeric vector of i.i.d. observations used as test dataset. |
p |
It's the root name of the guessed density function (model Y) for modeling the training dataset (it must be dnorm for ForecastErrorParadox version 0.1.0). |
q |
It's the root name of the guessed density function (model Z) for modeling the training dataset (it must be dnorm for ForecastErrorParadox version 0.1.0). |
meanp |
Mean of p. |
stdp |
Standard deviation of p. |
meanq |
Mean of q. |
stdq |
Standard deviation of p. |
Value
Diebold-Mariano Test (1995).
AvWScY = Average Weighted Scoring rule of model Y.
AvWScZ = Average Weighted Scoring rule of model Z.
checkdataExAnte = Table with data before imputation.
checkdataExPost = Table with data after imputation.
WScoreYPFS = Weighted Scoring rule of model Y computed with the testing data (see equation 13 in D'Amore(2020)).
WScoreZPFS = Weighted Scoring rule of model Z computed with the testing data (see equation 13 in D'Amore(2020)).
lenghtexpost = length of the sample before imputation.
TestStat = Diebold-Mariano Test Statistic.
TestRes = Test result.
pval = p-value of the classic Diebold-Mariano Test (1995).
dmtest2sid= Two-sided Modified Diebold-Mariano Test. The alternative hypothesis is that method 1 and method 2 have different levels of accuracy.
dmtestgre= One-sided Modified Diebold-Mariano Test. The alternative hypothesis is that method 2 is more accurate than method 1.
dmtestles= One-sided Modified Diebold-Mariano Test. The alternative hypothesis is that method 2 is less accurate than method 1.
h= adaptive bandwidth of the kernel density function.
Note
During the computation, R tends to approximate very small logarithmic values with non-finite values. For this reason, we replace such kind of approximations with finite values obtained by imputation method.
Author(s)
Gabriele D'Amore
Source
Gneiting, T. and A.E. Raftery (2007): ‘Strictly proper scoring rules, prediction and estimation’, Journal of the American Statistical Association 102, 359-378.
Diebold, F. X. (2015). Comparing predictive accuracy, twenty years later: A personal perspective on the use and abuse of Diebold–Mariano tests. Journal of Business & Economic Statistics, 33(1), 1-1.
Harvey, D., Leybourne, S., & Newbold, P. (1997). Testing the equality of prediction mean squared errors. International Journal of forecasting, 13(2), 281-291.
D'Amore, G. (2020): ‘When Scoring Rules fall into an informative paradox’, (Forthcoming).
Examples
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 | set.seed(8)
library(forecast)
library(Hmisc)
library(missForest)
DATA = rnorm(100, mean = 0, sd = 1)
h<-bw.ucv(DATA)
######(Ex.1)######
a=-1
b=1
scoretest<-ScorlogPFS(a,b,h,DATA, dnorm, dnorm, meanp = -1, stdp = 1, meanq =1, stdq=1)
#p-value of the classic Diebold-Mariano Test (1995)
scoretest$pval
#Average of the difference of the Weighted Scoring rules of the two models (Y and Z)
scoretest$AvWScY-scoretest$AvWScZ
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 1 and method 2 have different levels of accuracy.
scoretest$dmtest2sid
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 2 is more accurate than method 1
scoretest$dmtestgre
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 2 is less accurate than method 1
scoretest$dmtestles
######(Ex.2)######
a=0.8
b=3
scoretest<-ScorlogPFS(a,b,h,DATA, dnorm, dnorm, meanp = 0.8, stdp = 1, meanq =3, stdq=1)
#p-value of the classic Diebold-Mariano Test (1995)
scoretest$pval
#Average of the difference of the Weighted Scoring rules of the two models (Y and Z)
scoretest$AvWScY-scoretest$AvWScZ
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 1 and method 2 have different levels of accuracy.
scoretest$dmtest2sid
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 2 is more accurate than method 1
scoretest$dmtestgre
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 2 is less accurate than method 1
scoretest$dmtestles
######(Ex.3)######
a=0.4
b=-2
scoretest<-ScorlogPFS(a,b,h,DATA, dnorm, dnorm, meanp = 0.4, stdp = 0.3, meanq =-2, stdq=5)
#p-value of the classic Diebold-Mariano Test (1995)
scoretest$pval
#Average of the difference of the Weighted Scoring rules of the two models (Y and Z)
scoretest$AvWScY-scoretest$AvWScZ
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 1 and method 2 have different levels of accuracy.
scoretest$dmtest2sid
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 2 is more accurate than method 1
scoretest$dmtestgre
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 2 is less accurate than method 1
scoretest$dmtestles
######(Ex.4)######
a=(2)^(1/2)*(- log(5/9))^(1/2)/2
b=0
scoretest<-ScorlogPFS(a,b,h,DATA, dnorm, dnorm, meanp = (2)^(1/2)*(- log(5/9))^(1/2)/2, stdp = 1, meanq =0, stdq=3)
#p-value of the classic Diebold-Mariano Test (1995)
scoretest$pval
#Average of the difference of the Weighted Scoring rules of the two models (Y and Z)
scoretest$AvWScY-scoretest$AvWScZ
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 1 and method 2 have different levels of accuracy.
scoretest$dmtest2sid
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 2 is more accurate than method 1
scoretest$dmtestgre
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 2 is less accurate than method 1
scoretest$dmtestles
######(Ex.5)######
a=0
b=0
scoretest<-ScorlogPFS(a,b,h,DATA, dnorm, dnorm, meanp = 0, stdp = 2, meanq =0, stdq=0.5)
#p-value of the classic Diebold-Mariano Test (1995)
scoretest$pval
#Average of the difference of the Weighted Scoring rules of the two models (Y and Z)
scoretest$AvWScY-scoretest$AvWScZ
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 1 and method 2 have different levels of accuracy.
scoretest$dmtest2sid
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 2 is more accurate than method 1
scoretest$dmtestgre
# Modified out-of-sample one-step forecasts Diebold-Mariano Test proposed by Harvey, Leybourne and Newbold (1997)
# the alternative hypothesis is that method 2 is less accurate than method 1
scoretest$dmtestles
|
R Package Documentation
Browse R Packages
We want your feedback!
Note that we can't provide technical support on individual packages. You should contact the package authors for that.
Embedding an R snippet on your website
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.