library(knitr) library(vtype)
The purpose of the package is automatically detecting type of variables in not quality controlled data. The prediction is based on a pre-trained random forest model, trained on over 5000 medical variables with OOB accuracy of 99%. The accuracy depends heavily on the type and coding style of data. For example, often categorical variables are coded as integers 1 to x, if the number of categories is very large, there is no way to distinguish it from a continuous integer variable. Some types are per definition very sensitive to errors in data, like ID, missing or constant, where a single alternative non-missing value makes it not constant or not missing anymore. The data is assumed to be cross sectional, where ID is unique (no multiple entries per ID).
It can be used as a first step by data quality control to help sort the variables in advance and get some information about the possible formats.
The data set 'sim_nqc_data' contains 100 observations and 14 artificial variables with some not well formatted or missing values. The data is complete artificial and was not used for training or validation of the random forest model.
knitr::kable(head(sim_nqc_data, 20), caption='Artificial not quality controlled data')
The application is straightforward, it requires data in data.frame format.
It is important that all unusual missing values in the data, e.g. the code 9999 for missing values are covered.
Values as NA, NaN, Inf, NULL and spaces are automatic considered as invalid (missing) values.
The second column type
is the estimated type of the variable, and the column probability
indicates how certain the type is.
The format
gives additional information about the possible format of the variable, especially useful for date variables.
Class
is just a translation of type into broader categories.
tab <- vtype(sim_nqc_data, miss_values='9999') knitr::kable(tab, caption='Application example of vtype')
Very small sample size can reduce the prediction performance significantly.
The id
variable is now detected as integer, age
as categorical and decades
as a date variable.
knitr::kable(vtype(sim_nqc_data[1:10,]), caption='Application example with small sample size')
knitr::kable(vtype(mtcars), caption='Application example on data without errors')
Any scripts or data that you put into this service are public.
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.