README.md
In pdwaggoner/hdImpute: A Batch Process for High Dimensional Imputation

`hdImpute`: Batched high dimensional imputation

hdImpute is a correlation-based batch process for addressing high dimensional imputation problems. There are relatively few algorithms designed to handle imputation of missing data in high dimensional contexts in a relatively fast, efficient manner. Further, of the existing algorithms, even fewer are flexible enough to natively handle mixed-type data, often requiring a great deal of preprocessing to get the data into proper shape, and then postprocessing to return data to its original form. Such decisions as well as assumptions made by many algorithms regarding for example, the data generating process, limit the performance, flexibility, and usability of the algorithm. Built on top of a recent set of complementary algorithms for nonparametric imputation via chained random forests, missForest and missRanger, I offer a batch-based approach for subsetting the data based on ranked cross-feature correlations, and then imputing each batch separately, and then joining imputed subsets in the final step. The process is extremely fast and accurate after a bit of tuning to find the optimal batch size. As a result, high dimensional imputation is more accessible, and researchers are not forced to decide between speed or accuracy.

See the R-Bloggers post overviewing a basic implementation of hdImpute in R here

See the detailed complementary paper (Computational Statistics, 2023) introducing hdImpute along with several experimental results here (journal site) or here (full paper)

A complementary version of hdImpute is being actively developed in Python. Take a look here and please feel free to directly contribute!

Dev:

devtools::install_github("pdwaggoner/hdImpute")

Stable (on CRAN):

install.packages("hdImpute")
library(hdImpute)

hdImpute includes five core functions, and two helpers. The first three are to proceed by individual stages ((1) build the correlation matrix, (2) flatten and rank the matrix to give a ranked feature list, and (3) build batches, impute, and join). The fourth function (hdImpute()) runs all stages simultaneously, which is slightly less flexible, but much simpler. Finally, the latest release (v0.2.1) includes a fifth function to evaluate the quality of imputations by computing the mean absolute differences ("MAD scores") for each variable in the original data compared to the imputed version of the data.

feature_cor(): creates the correlation matrix
flatten_mat(): flattens the correlation matrix from the previous stage, and ranks the features based on absolute correlations. Thus, the input for flatten_mat() should be the stored output from feature_cor().
impute_batches(): creates batches based on the feature rankings from flatten_mat(), and then imputes missing values for each batch, until all batches are completed. Then, joins the batches to give a completed, imputed data set.
hdImpute(): does everything for you. At a minimum, pass the raw data object (data) along with specifying the batch size (batch) to hdImpute() to return a complete, imputed data set (same as you'd get from the individual stages in the above three functions).
mad(): computes variable-wise mean absolute differences (MAD) between original and imputed dataframes. Returns the MAD scores for each variable as a tibble to ensure tidy compliance and easy interaction with other Tidyverse functions (e.g., ggplot() for visualizing imputation error).

There are several vignettes with deeper dives into the package functionality, which include a few ideas for how to use the software for any imputation project.

This software is being actively developed, with many more features to come. Wide engagement with it and collaboration is welcomed! Here's a sampling of how to contribute: