importance_pvalues: ranger variable importance p-values

Description Usage Arguments Details Value Author(s) References See Also Examples

View source: R/importance.R

Description

Compute variable importance with p-values. For high dimensional data, the fast method of Janitza et al. (2016) can be used. The permutation approach of Altmann et al. (2010) is computationally intensive but can be used with all kinds of data. See below for details.

Usage

1
2
3
4
5
6
7
8
importance_pvalues(
  x,
  method = c("janitza", "altmann"),
  num.permutations = 100,
  formula = NULL,
  data = NULL,
  ...
)

Arguments

x

ranger or holdoutRF object.

method

Method to compute p-values. Use "janitza" for the method by Janitza et al. (2016) or "altmann" for the non-parametric method by Altmann et al. (2010).

num.permutations

Number of permutations. Used in the "altmann" method only.

formula

Object of class formula or character describing the model to fit. Used in the "altmann" method only.

data

Training data of class data.frame or matrix. Used in the "altmann" method only.

...

Further arguments passed to ranger(). Used in the "altmann" method only.

Details

The method of Janitza et al. (2016) uses a clever trick: With an unbiased variable importance measure, the importance values of non-associated variables vary randomly around zero. Thus, all non-positive importance values are assumed to correspond to these non-associated variables and they are used to construct a distribution of the importance under the null hypothesis of no association to the response. Since only the non-positive values of this distribution can be observed, the positive values are created by mirroring the negative distribution. See Janitza et al. (2016) for details.

The method of Altmann et al. (2010) uses a simple permutation test: The distribution of the importance under the null hypothesis of no association to the response is created by several replications of permuting the response, growing an RF and computing the variable importance. The authors recommend 50-100 permutations. However, much larger numbers have to be used to estimate more precise p-values. We add 1 to the numerator and denominator to avoid zero p-values.

Value

Variable importance and p-value for each variable.

Author(s)

Marvin N. Wright

References

Janitza, S., Celik, E. & Boulesteix, A.-L., (2016). A computationally fast variable importance test for random forests for high-dimensional data. Adv Data Anal Classif https://doi.org/10.1007/s11634-016-0276-4.
Altmann, A., Tolosi, L., Sander, O. & Lengauer, T. (2010). Permutation importance: a corrected feature importance measure, Bioinformatics 26:1340-1347.

See Also

ranger

Examples

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
## Janitza's p-values with corrected Gini importance
n <- 50
p <- 400
dat <- data.frame(y = factor(rbinom(n, 1, .5)), replicate(p, runif(n)))
rf.sim <- ranger(y ~ ., dat, importance = "impurity_corrected")
importance_pvalues(rf.sim, method = "janitza")

## Permutation p-values 
## Not run: 
rf.iris <- ranger(Species ~ ., data = iris, importance = 'permutation')
importance_pvalues(rf.iris, method = "altmann", formula = Species ~ ., data = iris)

## End(Not run)

Example output

        importance      pvalue
X1    0.0426282874 0.075242718
X2   -0.0043034297 0.570388350
X3   -0.0106764273 0.669902913
X4    0.0026888889 0.456310680
X5    0.0138952801 0.293689320
X6   -0.0123671457 0.701456311
X7   -0.0028737101 0.546116505
X8    0.0038207039 0.434466019
X9    0.0134288608 0.293689320
X10   0.0040423529 0.434466019
X11   0.0072063492 0.378640777
X12  -0.0486702798 0.936893204
X13   0.0281320784 0.172330097
X14   0.0246270108 0.208737864
X15  -0.0279561834 0.827669903
X16   0.0258352941 0.199029126
X17  -0.0389369119 0.912621359
X18   0.0205075434 0.240291262
X19  -0.0362137552 0.900485437
X20  -0.0164238815 0.735436893
X21  -0.0131935844 0.703883495
X22   0.0141715518 0.291262136
X23  -0.0001165923 0.504854369
X24   0.0105093840 0.332524272
X25  -0.0345114095 0.888349515
X26  -0.0285306044 0.832524272
X27   0.1079454849 0.009708738
X28   0.2500531898 0.000000000
X29   0.0160434275 0.269417476
X30  -0.0110834619 0.682038835
X31  -0.0148915556 0.718446602
X32   0.0069836356 0.381067961
X33  -0.0252316340 0.798543689
X34  -0.0083867551 0.638349515
X35  -0.0060292775 0.601941748
X36  -0.0389210536 0.910194175
X37   0.0977027649 0.012135922
X38   0.0037052227 0.441747573
X39  -0.0602931674 0.958737864
X40  -0.0325734848 0.868932039
X41  -0.0119933580 0.696601942
X42  -0.0057452355 0.594660194
X43   0.0802092902 0.024271845
X44  -0.1005949662 0.990291262
X45   0.0352816045 0.111650485
X46   0.0839933489 0.019417476
X47   0.0202107409 0.247572816
X48  -0.0050458090 0.580097087
X49   0.1109770955 0.009708738
X50  -0.0025705137 0.541262136
X51  -0.0335289479 0.878640777
X52  -0.0769434897 0.973300971
X53  -0.0071020750 0.621359223
X54  -0.0042031319 0.567961165
X55  -0.0811385819 0.978155340
X56  -0.0062814371 0.611650485
X57  -0.0339577354 0.883495146
X58  -0.0158356032 0.728155340
X59  -0.0335429437 0.881067961
X60  -0.0061033670 0.606796117
X61  -0.0088561648 0.643203883
X62   0.0214932975 0.230582524
X63  -0.0007729902 0.509708738
X64   0.0123143614 0.303398058
X65   0.0240387487 0.211165049
X66  -0.0230294653 0.788834951
X67   0.0395102300 0.082524272
X68   0.0350336533 0.111650485
X69   0.0137750815 0.293689320
X70  -0.0309653827 0.851941748
X71   0.0043680729 0.429611650
X72  -0.0265262271 0.813106796
X73  -0.0029165388 0.548543689
X74   0.0050737678 0.419902913
X75   0.0090668896 0.354368932
X76  -0.0063294641 0.614077670
X77  -0.0891111604 0.985436893
X78   0.0026111111 0.458737864
X79  -0.0038105339 0.565533981
X80  -0.0259235255 0.803398058
X81  -0.0060028860 0.599514563
X82   0.0311001000 0.148058252
X83   0.0208064807 0.237864078
X84  -0.0229822316 0.786407767
X85  -0.1492159612 0.997572816
X86   0.0124444444 0.298543689
X87   0.0316173143 0.143203883
X88  -0.0268287278 0.817961165
X89   0.0069473329 0.381067961
X90  -0.0025144013 0.533980583
X91  -0.0292070393 0.839805825
X92   0.0432688723 0.075242718
X93   0.0401365782 0.082524272
X94   0.0283878401 0.169902913
X95   0.0123852632 0.298543689
X96  -0.0358061217 0.898058252
X97  -0.0046212917 0.572815534
X98   0.0283343197 0.169902913
X99   0.0322470450 0.135922330
X100  0.0180946352 0.259708738
X101 -0.0101782028 0.665048544
X102 -0.0446040645 0.929611650
X103  0.0031036085 0.449029126
X104  0.0041585462 0.434466019
X105  0.0384883022 0.094660194
X106 -0.0037404673 0.560679612
X107 -0.0290994822 0.834951456
X108  0.0418705302 0.080097087
X109 -0.0317495382 0.859223301
X110 -0.0332231718 0.871359223
X111 -0.0356603942 0.895631068
X112  0.0435556125 0.072815534
X113 -0.0333945818 0.876213592
X114 -0.0159342356 0.730582524
X115 -0.0333346288 0.873786408
X116 -0.0215642706 0.771844660
X117 -0.0293807407 0.844660194
X118 -0.0094135925 0.657766990
X119 -0.0501021624 0.939320388
X120 -0.0510750621 0.941747573
X121  0.0050598967 0.419902913
X122 -0.0112189355 0.686893204
X123 -0.0109333624 0.674757282
X124 -0.0389822894 0.915048544
X125 -0.0133058608 0.706310680
X126 -0.0371867981 0.902912621
X127  0.0237343401 0.211165049
X128  0.0127221408 0.298543689
X129 -0.0115433360 0.691747573
X130 -0.0163171023 0.733009709
X131 -0.0276825347 0.825242718
X132 -0.0550987655 0.951456311
X133  0.0010366729 0.485436893
X134  0.0208661496 0.235436893
X135  0.0385735998 0.094660194
X136 -0.0205028340 0.759708738
X137  0.0520369528 0.058252427
X138  0.0012983092 0.485436893
X139  0.0476792394 0.065533981
X140  0.0226802015 0.218446602
X141  0.0179454094 0.259708738
X142 -0.0291364567 0.837378641
X143 -0.0198360375 0.750000000
X144  0.0261745336 0.194174757
X145  0.0126697092 0.298543689
X146  0.0608101163 0.041262136
X147  0.0429205848 0.075242718
X148  0.0217204939 0.228155340
X149  0.0055048713 0.415048544
X150  0.0270532373 0.182038835
X151  0.0106630850 0.332524272
X152 -0.0281642686 0.830097087
X153 -0.0586077027 0.956310680
X154 -0.0092497657 0.650485437
X155  0.0012720763 0.485436893
X156 -0.0016353303 0.521844660
X157  0.0290806991 0.167475728
X158 -0.0062135168 0.609223301
X159  0.0001149091 0.497572816
X160  0.0419757955 0.080097087
X161 -0.0321991503 0.864077670
X162 -0.0048743590 0.577669903
X163 -0.0816582711 0.980582524
X164  0.0157901634 0.274271845
X165  0.0070171123 0.381067961
X166 -0.0057202980 0.592233010
X167 -0.0096918768 0.662621359
X168 -0.0203054742 0.754854369
X169 -0.0025490471 0.538834951
X170 -0.0090984484 0.648058252
X171  0.0679690389 0.038834951
X172 -0.0408622221 0.919902913
X173 -0.0265974368 0.815533981
X174 -0.0053129759 0.582524272
X175  0.0143037207 0.291262136
X176 -0.0293652165 0.842233010
X177 -0.1157817678 0.992718447
X178  0.0330688505 0.131067961
X179 -0.0014531413 0.519417476
X180  0.0111490792 0.317961165
X181  0.0356580415 0.106796117
X182  0.0000000000 0.502427184
X183  0.0115462081 0.308252427
X184  0.0394106829 0.082524272
X185  0.0114284567 0.310679612
X186 -0.0086002335 0.640776699
X187  0.0047015904 0.427184466
X188  0.0200998390 0.247572816
X189  0.0063853896 0.385922330
X190  0.0244374481 0.208737864
X191  0.0333664717 0.126213592
X192 -0.0026209491 0.543689320
X193 -0.0530500371 0.949029126
X194  0.0581832491 0.046116505
X195 -0.0060559307 0.604368932
X196  0.0395792294 0.082524272
X197 -0.0032000000 0.553398058
X198 -0.0198539029 0.752427184
X199  0.0213691449 0.230582524
X200 -0.0769550779 0.975728155
X201 -0.0221600193 0.776699029
X202 -0.0123402452 0.699029126
X203  0.0931699295 0.014563107
X204 -0.0054409524 0.584951456
X205  0.0319891511 0.138349515
X206  0.0027926888 0.456310680
X207  0.0192132315 0.254854369
X208  0.1354771047 0.004854369
X209 -0.0057184141 0.589805825
X210 -0.0640027102 0.961165049
X211 -0.0208136561 0.764563107
X212  0.0014630516 0.480582524
X213  0.1591434392 0.002427184
X214 -0.0148315152 0.716019417
X215 -0.0260599709 0.805825243
X216  0.0187638142 0.257281553
X217  0.0607485607 0.041262136
X218 -0.0848956568 0.983009709
X219  0.0413038734 0.080097087
X220 -0.0107758381 0.672330097
X221 -0.0093504632 0.652912621
X222  0.0087272727 0.359223301
X223 -0.0021070012 0.529126214
X224 -0.0156970379 0.725728155
X225 -0.0686921622 0.966019417
X226 -0.0009044689 0.514563107
X227 -0.0378979453 0.905339806
X228  0.0751292040 0.031553398
X229  0.0046119423 0.429611650
X230 -0.0094653398 0.660194175
X231  0.0038400000 0.434466019
X232 -0.0468919718 0.934466019
X233 -0.0064676058 0.616504854
X234  0.0141875883 0.291262136
X235  0.0128089559 0.298543689
X236  0.1018436999 0.009708738
X237 -0.0271169551 0.820388350
X238  0.0427946045 0.075242718
X239 -0.0094103598 0.655339806
X240  0.0029090909 0.453883495
X241  0.0366548520 0.099514563
X242 -0.0526463260 0.946601942
X243  0.0067307692 0.383495146
X244  0.0367966955 0.099514563
X245 -0.0035912650 0.558252427
X246  0.0386434665 0.092233010
X247 -0.0204610048 0.757281553
X248 -0.0211818034 0.766990291
X249  0.0972497018 0.012135922
X250 -0.0067630952 0.618932039
X251  0.0097410256 0.337378641
X252  0.0186541901 0.257281553
X253  0.0386026560 0.092233010
X254 -0.0226900981 0.783980583
X255  0.0036917217 0.441747573
X256 -0.0685839544 0.963592233
X257  0.0288643906 0.167475728
X258 -0.0224378998 0.779126214
X259 -0.0025489347 0.536407767
X260 -0.0018285493 0.524271845
X261  0.0451701151 0.067961165
X262 -0.0196608379 0.747572816
X263  0.0071619916 0.378640777
X264  0.0591349737 0.043689320
X265  0.0076666667 0.368932039
X266  0.0215676402 0.228155340
X267 -0.0262372464 0.808252427
X268  0.0161389428 0.269417476
X269 -0.0312994160 0.854368932
X270  0.1694708946 0.000000000
X271 -0.0718760567 0.968446602
X272  0.0203944749 0.245145631
X273  0.0319721389 0.138349515
X274 -0.0385842045 0.907766990
X275 -0.0037515455 0.563106796
X276  0.0514907749 0.058252427
X277 -0.0255785673 0.800970874
X278 -0.0141438783 0.708737864
X279  0.0171050019 0.259708738
X280 -0.0524977981 0.944174757
X281 -0.0181858164 0.742718447
X282  0.0120104558 0.303398058
X283  0.0007166434 0.492718447
X284  0.0238602885 0.211165049
X285  0.0118373258 0.308252427
X286 -0.0433711456 0.927184466
X287 -0.0319360143 0.861650485
X288  0.0031529935 0.449029126
X289  0.0122683358 0.303398058
X290 -0.0225192262 0.781553398
X291 -0.0220084848 0.774271845
X292  0.0660452560 0.038834951
X293 -0.1205212779 0.995145631
X294  0.0367780392 0.099514563
X295 -0.0166308754 0.740291262
X296 -0.0072890290 0.623786408
X297 -0.0112969188 0.689320388
X298  0.0394036315 0.082524272
X299  0.0260854671 0.194174757
X300 -0.0393921700 0.917475728
X301 -0.0119331158 0.694174757
X302 -0.0014323922 0.516990291
X303  0.0202068147 0.247572816
X304 -0.0421172161 0.924757282
X305 -0.0022967572 0.531553398
X306  0.0129417179 0.298543689
X307  0.0082374956 0.368932039
X308 -0.0971299497 0.987864078
X309 -0.0298431990 0.849514563
X310  0.0189644956 0.254854369
X311 -0.0102658312 0.667475728
X312 -0.0056496552 0.587378641
X313 -0.0352902363 0.890776699
X314 -0.0088967201 0.645631068
X315  0.0113586453 0.310679612
X316  0.0243960689 0.208737864
X317 -0.0047430592 0.575242718
X318  0.0257887461 0.199029126
X319  0.0718378843 0.033980583
X320 -0.0766449640 0.970873786
X321  0.0213915018 0.230582524
X322 -0.0207515031 0.762135922
X323  0.0116883117 0.308252427
X324 -0.0313613281 0.856796117
X325 -0.0342138535 0.885922330
X326  0.0468667019 0.067961165
X327 -0.0152087542 0.720873786
X328 -0.0189493216 0.745145631
X329 -0.0076159695 0.631067961
X330 -0.0356077593 0.893203883
X331 -0.0111721577 0.684466019
X332 -0.0250379061 0.793689320
X333 -0.0419878656 0.922330097
X334  0.0007556980 0.492718447
X335 -0.0240813984 0.791262136
X336  0.0066728050 0.383495146
X337 -0.0074420017 0.628640777
X338  0.0639871169 0.041262136
X339  0.0051512021 0.419902913
X340  0.0438375249 0.072815534
X341  0.0326896487 0.131067961
X342  0.0112080116 0.315533981
X343 -0.0109723953 0.677184466
X344  0.0976987326 0.012135922
X345  0.1641345621 0.002427184
X346  0.0129619271 0.298543689
X347  0.0097790835 0.337378641
X348 -0.1682383394 1.000000000
X349  0.0210801619 0.235436893
X350  0.0067368631 0.383495146
X351 -0.0082684182 0.633495146
X352  0.0174589678 0.259708738
X353  0.0510414243 0.060679612
X354  0.0222218301 0.223300971
X355  0.0000000000 0.502427184
X356  0.0050117097 0.422330097
X357  0.0147023982 0.286407767
X358 -0.0251449328 0.796116505
X359  0.0095104212 0.339805825
X360 -0.0034749816 0.555825243
X361  0.0403102242 0.082524272
X362 -0.0073544709 0.626213592
X363  0.0450791582 0.067961165
X364  0.0466307627 0.067961165
X365 -0.0144513238 0.711165049
X366  0.0353819195 0.109223301
X367 -0.0576835553 0.953883495
X368 -0.0449747375 0.932038835
X369  0.0534092874 0.050970874
X370  0.0319594255 0.138349515
X371 -0.0322719686 0.866504854
X372  0.0034335094 0.446601942
X373  0.0389795990 0.087378641
X374 -0.0008393228 0.512135922
X375  0.0076966295 0.368932039
X376  0.0131193657 0.298543689
X377 -0.0029278212 0.550970874
X378 -0.0165009864 0.737864078
X379 -0.0146352174 0.713592233
X380 -0.0272301009 0.822815534
X381  0.0158895749 0.271844660
X382  0.1038301174 0.009708738
X383  0.0086493506 0.359223301
X384 -0.0019458185 0.526699029
X385  0.0091573213 0.351941748
X386  0.0115879495 0.308252427
X387  0.0375370922 0.097087379
X388 -0.0155223869 0.723300971
X389  0.0292128393 0.160194175
X390  0.0153855815 0.279126214
X391 -0.0294397325 0.847087379
X392 -0.0082722430 0.635922330
X393 -0.0005331987 0.507281553
X394  0.0526518239 0.053398058
X395 -0.0264879067 0.810679612
X396 -0.0212177054 0.769417476
X397  0.0841988258 0.019417476
X398 -0.0110443410 0.679611650
X399 -0.0057574431 0.597087379
X400  0.0135245886 0.293689320
              importance     pvalue
Sepal.Length 0.033611880 0.00990099
Sepal.Width  0.005845831 0.33663366
Petal.Length 0.295364371 0.00990099
Petal.Width  0.305109602 0.00990099

ranger documentation built on Jan. 11, 2020, 9:21 a.m.