README.md

R Markdown

This publivate package includes helper functions that I commonly use while modeling. So far, it includes:

  1. Creating dummy variables (with S3 method!)
  2. Frequency encoding
  3. Rare variable grouping
  4. Common DBI data request/fetch functions
  5. Easy data upload to a SQL server

More to come, I'm sure.

rm(list = ls())
library(data.table)
library(helperFuncs)

## Loading required package: odbc

## Loading required package: DBI

raw <- fread("Data/train.csv")

# Create a dummyencode Object
dumEnc <- dummyEncode(
  raw
  , vars = c("Alley","LotShape","LandContour","Utilities","LotConfig","LandSlope","Neighborhood")
)

encDT <- predict(dumEnc, raw)
encInpDT <- predict(dumEnc, raw, inPlace = TRUE)

freqObj <- frequencyEncode(
    raw
  , c("Alley","LotShape","LandContour","Utilities","LotConfig","LandSlope","Neighborhood")
  )

rareObj <- rareEncode(
  raw
  , vars = c("Alley","LotShape","LandContour","Utilities","LotConfig","LandSlope","Neighborhood")
  , minPerc = 0.1
)


freqDT <- applyTrans(raw, freqObj, inPlace = FALSE)
rareDT<- applyTrans(raw, rareObj, inPlace = TRUE)


# Results from different encodings:
head(encDT, 10)

##     Alley.na Alley.Grvl Alley.Pave LotShape.Reg LotShape.IR1 LotShape.IR2
##  1:        1          0          0            1            0            0
##  2:        1          0          0            1            0            0
##  3:        1          0          0            0            1            0
##  4:        1          0          0            0            1            0
##  5:        1          0          0            0            1            0
##  6:        1          0          0            0            1            0
##  7:        1          0          0            1            0            0
##  8:        1          0          0            0            1            0
##  9:        1          0          0            1            0            0
## 10:        1          0          0            1            0            0
##     LotShape.IR3 LandContour.Lvl LandContour.Bnk LandContour.Low
##  1:            0               1               0               0
##  2:            0               1               0               0
##  3:            0               1               0               0
##  4:            0               1               0               0
##  5:            0               1               0               0
##  6:            0               1               0               0
##  7:            0               1               0               0
##  8:            0               1               0               0
##  9:            0               1               0               0
## 10:            0               1               0               0
##     LandContour.HLS Utilities.AllPub Utilities.NoSeWa LotConfig.Inside
##  1:               0                1                0                1
##  2:               0                1                0                0
##  3:               0                1                0                1
##  4:               0                1                0                0
##  5:               0                1                0                0
##  6:               0                1                0                1
##  7:               0                1                0                1
##  8:               0                1                0                0
##  9:               0                1                0                1
## 10:               0                1                0                0
##     LotConfig.FR2 LotConfig.Corner LotConfig.CulDSac LotConfig.FR3
##  1:             0                0                 0             0
##  2:             1                0                 0             0
##  3:             0                0                 0             0
##  4:             0                1                 0             0
##  5:             1                0                 0             0
##  6:             0                0                 0             0
##  7:             0                0                 0             0
##  8:             0                1                 0             0
##  9:             0                0                 0             0
## 10:             0                1                 0             0
##     LandSlope.Gtl LandSlope.Mod LandSlope.Sev Neighborhood.CollgCr
##  1:             1             0             0                    1
##  2:             1             0             0                    0
##  3:             1             0             0                    1
##  4:             1             0             0                    0
##  5:             1             0             0                    0
##  6:             1             0             0                    0
##  7:             1             0             0                    0
##  8:             1             0             0                    0
##  9:             1             0             0                    0
## 10:             1             0             0                    0
##     Neighborhood.Veenker Neighborhood.Crawfor Neighborhood.NoRidge
##  1:                    0                    0                    0
##  2:                    1                    0                    0
##  3:                    0                    0                    0
##  4:                    0                    1                    0
##  5:                    0                    0                    1
##  6:                    0                    0                    0
##  7:                    0                    0                    0
##  8:                    0                    0                    0
##  9:                    0                    0                    0
## 10:                    0                    0                    0
##     Neighborhood.Mitchel Neighborhood.Somerst Neighborhood.NWAmes
##  1:                    0                    0                   0
##  2:                    0                    0                   0
##  3:                    0                    0                   0
##  4:                    0                    0                   0
##  5:                    0                    0                   0
##  6:                    1                    0                   0
##  7:                    0                    1                   0
##  8:                    0                    0                   1
##  9:                    0                    0                   0
## 10:                    0                    0                   0
##     Neighborhood.OldTown Neighborhood.BrkSide Neighborhood.Sawyer
##  1:                    0                    0                   0
##  2:                    0                    0                   0
##  3:                    0                    0                   0
##  4:                    0                    0                   0
##  5:                    0                    0                   0
##  6:                    0                    0                   0
##  7:                    0                    0                   0
##  8:                    0                    0                   0
##  9:                    1                    0                   0
## 10:                    0                    1                   0
##     Neighborhood.NridgHt Neighborhood.NAmes Neighborhood.SawyerW
##  1:                    0                  0                    0
##  2:                    0                  0                    0
##  3:                    0                  0                    0
##  4:                    0                  0                    0
##  5:                    0                  0                    0
##  6:                    0                  0                    0
##  7:                    0                  0                    0
##  8:                    0                  0                    0
##  9:                    0                  0                    0
## 10:                    0                  0                    0
##     Neighborhood.IDOTRR Neighborhood.MeadowV Neighborhood.Edwards
##  1:                   0                    0                    0
##  2:                   0                    0                    0
##  3:                   0                    0                    0
##  4:                   0                    0                    0
##  5:                   0                    0                    0
##  6:                   0                    0                    0
##  7:                   0                    0                    0
##  8:                   0                    0                    0
##  9:                   0                    0                    0
## 10:                   0                    0                    0
##     Neighborhood.Timber Neighborhood.Gilbert Neighborhood.StoneBr
##  1:                   0                    0                    0
##  2:                   0                    0                    0
##  3:                   0                    0                    0
##  4:                   0                    0                    0
##  5:                   0                    0                    0
##  6:                   0                    0                    0
##  7:                   0                    0                    0
##  8:                   0                    0                    0
##  9:                   0                    0                    0
## 10:                   0                    0                    0
##     Neighborhood.ClearCr Neighborhood.NPkVill Neighborhood.Blmngtn
##  1:                    0                    0                    0
##  2:                    0                    0                    0
##  3:                    0                    0                    0
##  4:                    0                    0                    0
##  5:                    0                    0                    0
##  6:                    0                    0                    0
##  7:                    0                    0                    0
##  8:                    0                    0                    0
##  9:                    0                    0                    0
## 10:                    0                    0                    0
##     Neighborhood.BrDale Neighborhood.SWISU Neighborhood.Blueste
##  1:                   0                  0                    0
##  2:                   0                  0                    0
##  3:                   0                  0                    0
##  4:                   0                  0                    0
##  5:                   0                  0                    0
##  6:                   0                  0                    0
##  7:                   0                  0                    0
##  8:                   0                  0                    0
##  9:                   0                  0                    0
## 10:                   0                  0                    0

head(freqDT, 10)

##     Alley LotShape LandContour Utilities LotConfig LandSlope Neighborhood
##  1:     3        4           4         2         5         3           24
##  2:     3        4           4         2         2         3            3
##  3:     3        3           4         2         5         3           24
##  4:     3        3           4         2         4         3           14
##  5:     3        3           4         2         2         3           12
##  6:     3        3           4         2         5         3           13
##  7:     3        4           4         2         5         3           21
##  8:     3        3           4         2         4         3           17
##  9:     3        4           4         2         5         3           23
## 10:     3        4           4         2         4         3           15

head(rareDT, 10)

##     Id MSSubClass MSZoning LotFrontage LotArea Street Alley LotShape
##  1:  1         60       RL          65    8450   Pave    na      Reg
##  2:  2         20       RL          80    9600   Pave    na      Reg
##  3:  3         60       RL          68   11250   Pave    na      IR1
##  4:  4         70       RL          60    9550   Pave    na      IR1
##  5:  5         60       RL          84   14260   Pave    na      IR1
##  6:  6         50       RL          85   14115   Pave    na      IR1
##  7:  7         20       RL          75   10084   Pave    na      Reg
##  8:  8         60       RL          NA   10382   Pave    na      IR1
##  9:  9         50       RM          51    6120   Pave    na      Reg
## 10: 10        190       RL          50    7420   Pave    na      Reg
##     LandContour Utilities LotConfig LandSlope Neighborhood Condition1
##  1:         Lvl    AllPub    Inside       Gtl      CollgCr       Norm
##  2:         Lvl    AllPub rareGroup       Gtl    rareGroup      Feedr
##  3:         Lvl    AllPub    Inside       Gtl      CollgCr       Norm
##  4:         Lvl    AllPub    Corner       Gtl    rareGroup       Norm
##  5:         Lvl    AllPub rareGroup       Gtl    rareGroup       Norm
##  6:         Lvl    AllPub    Inside       Gtl    rareGroup       Norm
##  7:         Lvl    AllPub    Inside       Gtl    rareGroup       Norm
##  8:         Lvl    AllPub    Corner       Gtl    rareGroup       PosN
##  9:         Lvl    AllPub    Inside       Gtl    rareGroup     Artery
## 10:         Lvl    AllPub    Corner       Gtl    rareGroup     Artery
##     Condition2 BldgType HouseStyle OverallQual OverallCond YearBuilt
##  1:       Norm     1Fam     2Story           7           5      2003
##  2:       Norm     1Fam     1Story           6           8      1976
##  3:       Norm     1Fam     2Story           7           5      2001
##  4:       Norm     1Fam     2Story           7           5      1915
##  5:       Norm     1Fam     2Story           8           5      2000
##  6:       Norm     1Fam     1.5Fin           5           5      1993
##  7:       Norm     1Fam     1Story           8           5      2004
##  8:       Norm     1Fam     2Story           7           6      1973
##  9:       Norm     1Fam     1.5Fin           7           5      1931
## 10:     Artery   2fmCon     1.5Unf           5           6      1939
##     YearRemodAdd RoofStyle RoofMatl Exterior1st Exterior2nd MasVnrType
##  1:         2003     Gable  CompShg     VinylSd     VinylSd    BrkFace
##  2:         1976     Gable  CompShg     MetalSd     MetalSd       None
##  3:         2002     Gable  CompShg     VinylSd     VinylSd    BrkFace
##  4:         1970     Gable  CompShg     Wd Sdng     Wd Shng       None
##  5:         2000     Gable  CompShg     VinylSd     VinylSd    BrkFace
##  6:         1995     Gable  CompShg     VinylSd     VinylSd       None
##  7:         2005     Gable  CompShg     VinylSd     VinylSd      Stone
##  8:         1973     Gable  CompShg     HdBoard     HdBoard      Stone
##  9:         1950     Gable  CompShg     BrkFace     Wd Shng       None
## 10:         1950     Gable  CompShg     MetalSd     MetalSd       None
##     MasVnrArea ExterQual ExterCond Foundation BsmtQual BsmtCond
##  1:        196        Gd        TA      PConc       Gd       TA
##  2:          0        TA        TA     CBlock       Gd       TA
##  3:        162        Gd        TA      PConc       Gd       TA
##  4:          0        TA        TA     BrkTil       TA       Gd
##  5:        350        Gd        TA      PConc       Gd       TA
##  6:          0        TA        TA       Wood       Gd       TA
##  7:        186        Gd        TA      PConc       Ex       TA
##  8:        240        TA        TA     CBlock       Gd       TA
##  9:          0        TA        TA     BrkTil       TA       TA
## 10:          0        TA        TA     BrkTil       TA       TA
##     BsmtExposure BsmtFinType1 BsmtFinSF1 BsmtFinType2 BsmtFinSF2 BsmtUnfSF
##  1:           No          GLQ        706          Unf          0       150
##  2:           Gd          ALQ        978          Unf          0       284
##  3:           Mn          GLQ        486          Unf          0       434
##  4:           No          ALQ        216          Unf          0       540
##  5:           Av          GLQ        655          Unf          0       490
##  6:           No          GLQ        732          Unf          0        64
##  7:           Av          GLQ       1369          Unf          0       317
##  8:           Mn          ALQ        859          BLQ         32       216
##  9:           No          Unf          0          Unf          0       952
## 10:           No          GLQ        851          Unf          0       140
##     TotalBsmtSF Heating HeatingQC CentralAir Electrical 1stFlrSF 2ndFlrSF
##  1:         856    GasA        Ex          Y      SBrkr      856      854
##  2:        1262    GasA        Ex          Y      SBrkr     1262        0
##  3:         920    GasA        Ex          Y      SBrkr      920      866
##  4:         756    GasA        Gd          Y      SBrkr      961      756
##  5:        1145    GasA        Ex          Y      SBrkr     1145     1053
##  6:         796    GasA        Ex          Y      SBrkr      796      566
##  7:        1686    GasA        Ex          Y      SBrkr     1694        0
##  8:        1107    GasA        Ex          Y      SBrkr     1107      983
##  9:         952    GasA        Gd          Y      FuseF     1022      752
## 10:         991    GasA        Ex          Y      SBrkr     1077        0
##     LowQualFinSF GrLivArea BsmtFullBath BsmtHalfBath FullBath HalfBath
##  1:            0      1710            1            0        2        1
##  2:            0      1262            0            1        2        0
##  3:            0      1786            1            0        2        1
##  4:            0      1717            1            0        1        0
##  5:            0      2198            1            0        2        1
##  6:            0      1362            1            0        1        1
##  7:            0      1694            1            0        2        0
##  8:            0      2090            1            0        2        1
##  9:            0      1774            0            0        2        0
## 10:            0      1077            1            0        1        0
##     BedroomAbvGr KitchenAbvGr KitchenQual TotRmsAbvGrd Functional
##  1:            3            1          Gd            8        Typ
##  2:            3            1          TA            6        Typ
##  3:            3            1          Gd            6        Typ
##  4:            3            1          Gd            7        Typ
##  5:            4            1          Gd            9        Typ
##  6:            1            1          TA            5        Typ
##  7:            3            1          Gd            7        Typ
##  8:            3            1          TA            7        Typ
##  9:            2            2          TA            8       Min1
## 10:            2            2          TA            5        Typ
##     Fireplaces FireplaceQu GarageType GarageYrBlt GarageFinish GarageCars
##  1:          0        <NA>     Attchd        2003          RFn          2
##  2:          1          TA     Attchd        1976          RFn          2
##  3:          1          TA     Attchd        2001          RFn          2
##  4:          1          Gd     Detchd        1998          Unf          3
##  5:          1          TA     Attchd        2000          RFn          3
##  6:          0        <NA>     Attchd        1993          Unf          2
##  7:          1          Gd     Attchd        2004          RFn          2
##  8:          2          TA     Attchd        1973          RFn          2
##  9:          2          TA     Detchd        1931          Unf          2
## 10:          2          TA     Attchd        1939          RFn          1
##     GarageArea GarageQual GarageCond PavedDrive WoodDeckSF OpenPorchSF
##  1:        548         TA         TA          Y          0          61
##  2:        460         TA         TA          Y        298           0
##  3:        608         TA         TA          Y          0          42
##  4:        642         TA         TA          Y          0          35
##  5:        836         TA         TA          Y        192          84
##  6:        480         TA         TA          Y         40          30
##  7:        636         TA         TA          Y        255          57
##  8:        484         TA         TA          Y        235         204
##  9:        468         Fa         TA          Y         90           0
## 10:        205         Gd         TA          Y          0           4
##     EnclosedPorch 3SsnPorch ScreenPorch PoolArea PoolQC Fence MiscFeature
##  1:             0         0           0        0   <NA>  <NA>        <NA>
##  2:             0         0           0        0   <NA>  <NA>        <NA>
##  3:             0         0           0        0   <NA>  <NA>        <NA>
##  4:           272         0           0        0   <NA>  <NA>        <NA>
##  5:             0         0           0        0   <NA>  <NA>        <NA>
##  6:             0       320           0        0   <NA> MnPrv        Shed
##  7:             0         0           0        0   <NA>  <NA>        <NA>
##  8:           228         0           0        0   <NA>  <NA>        Shed
##  9:           205         0           0        0   <NA>  <NA>        <NA>
## 10:             0         0           0        0   <NA>  <NA>        <NA>
##     MiscVal MoSold YrSold SaleType SaleCondition SalePrice
##  1:       0      2   2008       WD        Normal    208500
##  2:       0      5   2007       WD        Normal    181500
##  3:       0      9   2008       WD        Normal    223500
##  4:       0      2   2006       WD       Abnorml    140000
##  5:       0     12   2008       WD        Normal    250000
##  6:     700     10   2009       WD        Normal    143000
##  7:       0      8   2007       WD        Normal    307000
##  8:     350     11   2009       WD        Normal    200000
##  9:       0      4   2008       WD       Abnorml    129900
## 10:       0      1   2008       WD        Normal    118000


AnotherSamWilson/helperFuncs documentation built on July 25, 2019, 3:56 p.m.