In const-ae/sparseMatrixStats: Summary Statistics for Rows and Columns of Sparse Matrices

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%"
)

sparseMatrixStats

The goal of sparseMatrixStats is to make the API of matrixStats available for sparse matrices.

Installation

You can install the release version of sparseMatrixStats from BioConductor:

if (!requireNamespace("BiocManager", quietly = TRUE))
    install.packages("BiocManager")

BiocManager::install("sparseMatrixStats")

Alternatively, you can get the development version of the package from GitHub with:

# install.packages("devtools")
devtools::install_github("const-ae/sparseMatrixStats")

If you have trouble with the installation, see the end of the README.

Example

set.seed(1)

library(sparseMatrixStats)

mat <- matrix(0, nrow=10, ncol=6)
mat[sample(seq_len(60), 4)] <- 1:4
# Convert dense matrix to sparse matrix
sparse_mat <- as(mat, "dgCMatrix")
sparse_mat

The package provides an interface to quickly do common operations on the rows or columns. For example calculate the variance:

apply(mat, 2, var)
matrixStats::colVars(mat)
sparseMatrixStats::colVars(sparse_mat)

On this small example data, all methods are basically equally fast, but if we have a much larger dataset, the optimizations for the sparse data start to show.

I generate a dataset with 10,000 rows and 50 columns that is 99% empty

big_mat <- matrix(0, nrow=1e4, ncol=50)
big_mat[sample(seq_len(1e4 * 50), 5000)] <- rnorm(5000)
# Convert dense matrix to sparse matrix
big_sparse_mat <- as(big_mat, "dgCMatrix")

I use the bench package to benchmark the performance difference:

bench::mark(
  sparseMatrixStats=sparseMatrixStats::colVars(big_sparse_mat),
  matrixStats=matrixStats::colVars(big_mat),
  apply=apply(big_mat, 2, var)
)

As you can see sparseMatrixStats is ca. 35 times fast than matrixStats, which in turn is 7 times faster than the apply() version.

API

The package now supports all functions from the matrixStats API for column sparse matrices (dgCMatrix). And thanks to the MatrixGenerics it can be easily integrated along-side matrixStats and DelayedMatrixStats. Note that the rowXXX() functions are called by transposing the input and calling the corresponding colXXX() function. Special optimized implementations are available for rowSums2(), rowMeans2(), and rowVars().

matrixStats_functions <- sort(
  c("colsum", "rowsum", grep("^(col|row)", 
                             getNamespaceExports("matrixStats"), 
                             value = TRUE)))
DelayedMatrixStats_functions <- grep("^(col|row)", getNamespaceExports("DelayedMatrixStats"), value=TRUE)
DelayedArray_functions <- grep("^(col|row)", getNamespaceExports("DelayedArray"), value=TRUE)
sparseMatrixStats_functions <- grep("^(col|row)", getNamespaceExports("sparseMatrixStats"), value=TRUE)

notes <- c("colAnyMissings"="Not implemented because it is deprecated in favor of `colAnyNAs()`",
           "rowAnyMissings"="Not implemented because it is deprecated in favor of `rowAnyNAs()`",
           "colsum"="Base R function",
           "rowsum"="Base R function",
           "colWeightedMedians"="Only equivalent if `interpolate=FALSE`",
           "rowWeightedMedians"="Only equivalent if `interpolate=FALSE`",
           "colWeightedMads"="Sparse version behaves slightly differently, because it always uses `interpolate=FALSE`.",
           "rowWeightedMads"="Sparse version behaves slightly differently, because it always uses `interpolate=FALSE`.")

api_df <- data.frame(
  Method = paste0(matrixStats_functions, "()"),
  matrixStats = ifelse(matrixStats_functions %in% matrixStats_functions, "✔", "❌"),
  sparseMatrixStats = ifelse(matrixStats_functions %in%sparseMatrixStats_functions, "✔", "❌"),
  Notes = ifelse(matrixStats_functions %in% names(notes), notes[matrixStats_functions], ""),
  stringsAsFactors = FALSE
)
knitr::kable(api_df, row.names = FALSE)

Installation Problems

sparseMatrixStats uses features from C++14 and as the standard is more than 6 years old, I thought this wouldn't cause problems. In most circumstances this is true, but there are reoccuring reports, that the installation fails for some people and that is of course annoying. The typical error message is:

Error: C++14 standard requested but CXX14 is not defined

The main reason that the installation fails is that the compiler is too old. Sufficient support for C++14 came in

• clang version 3.4
• gcc version 4.9

Accordingly, you must have a compiler available that is at least that new. If you run on the command line

```{bash eval=FALSE, include=TRUE} $ gcc --version

and it says 4.8, you will have to install a newer compiler. At the end of the section, I have collected a few tips to install an appropriate version on different distributions.

If you have recent version of `gcc` (>=4.9) or `clang` (>= 3.4) installed, but you still see the error message

Error: C++14 standard requested but CXX14 is not defined

the problem is that R doesn't yet know about it.

The solution is to either create a `~/.R/Makevars` file and define

CXX14 = g++ CXX14FLAGS = -g -O2 $(LTO) CXX14PICFLAGS = -fpic CXX14STD = -std=gnu++14

or simply call 
```r
withr::with_makevars(
 new = c(CXX14 = "g++", CXX14FLAGS = "-g -O2 $(LTO)", 
         CXX14PICFLAGS = "-fpic", CXX14STD = "-std=gnu++14"), 
 code = {
   BiocManager::install("sparseMatrixStats")
 })

Update Compiler

CentOS / Scientic Linux / RHEL

One of the main culprits causing trouble is CentOS 7. It is popular in scientific computing and is still supported until 2024. It does, however, by default come with a very old version of gcc (4.8.5).

To install a more recent compiler, we can use devtoolset. First, we enable the Software Collection Tools and then install for example gcc version 7:

```{bash eval=FALSE} $ yum install centos-release-scl $ yum install devtoolset-7-gcc*

We can now either activate the new compiler for an R session   
```{bash eval=FALSE}
$ scl enable devtoolset-7 R

and then call

withr::with_makevars(
 new = c(CXX14 = "g++", CXX14FLAGS = "-g -O2 $(LTO)", 
         CXX14PICFLAGS = "-fpic", CXX14STD = "-std=gnu++14"), 
 code = {
   BiocManager::install("sparseMatrixStats")
 })

or we refer to the full path of the newly installed g++ from a standard R session

withr::with_makevars(
 new = c(CXX14 = "/opt/rh/devtoolset-7/root/usr/bin/g++", 
         CXX14FLAGS = "-g -O2 $(LTO)", CXX14PICFLAGS = "-fpic",
         CXX14STD = "-std=gnu++14"), 
 code = {
   BiocManager::install("sparseMatrixStats")
 })

Note, that our shenanigans are only necessary once, when we install sparseMatrixStats. After the successful installation of the package, we can use R as usual.

Debian

All Debian releases later than Jessie (i.e. Stretch, Buster, Bullseye) are recent enough and should install sparseMatrixStats without problems.

I was able to install sparseMatrixStats on Debian Jessie (which comes with gcc version 4.9.2) by providing the necessary Makefile arguments

withr::with_makevars(
 new = c(CXX14 = "g++", 
         CXX14FLAGS = "-g -O2 $(LTO)", CXX14PICFLAGS = "-fpic",
         CXX14STD = "-std=gnu++14"), 
 code = {
   BiocManager::install("sparseMatrixStats")
 })

Debian Wheezy comes with gcc 4.7, which does not support C++14. On the other hand, the last R release that was backported to Wheezy is 3.2.5 (see information on CRAN). Thus, if you are still on Wheezy, I would encourage you to update your OS.

Ubuntu

Since 16.04, Ubuntu comes with a recent enough compiler.

Ubuntu 14.04 comes with gcc 4.8.5, but updating to gcc-5 is easy:

```{bash eval=FALSE, include=TRUE} $ sudo add-apt-repository ppa:ubuntu-toolchain-r/test $ sudo apt-get update $ sudo apt-get install gcc-5 g++-5

After that, you can install `sparseMatrixStats` with a custom Makevars variables that refer to the new compiler
```r
withr::with_makevars(
 new = c(CXX14 = "g++-5", 
         CXX14FLAGS = "-g -O2 $(LTO)", CXX14PICFLAGS = "-fpic",
         CXX14STD = "-std=gnu++14"), 
 code = {
   BiocManager::install("sparseMatrixStats")
 })

MacOS

No trouble reported so far. Just do:

BiocManager::install("sparseMatrixStats")

Windows

It is important that you have RTools40 installed. After that, you shouldn't have any troubles installing sparseMatrixStats directly from Bioconductor:

BiocManager::install("sparseMatrixStats")

But I still have a problems

1. Please make sure to carefully read the full problem section.
2. Make sure that you are using at least R 4.0.0.
3. Make sure your compiler is new enough to support C++14 (ie. gcc >= 4.9 and clang >= 3.4)

If your problems nonetheless persist, please file an issue including the following information:

• Operating system with exact version (e.g. 'Linux Ubuntu 18.04')
• Compiler and compiler version (e.g. 'gcc version 7.2.5')
• The output of sessionInfo()
• Information if you have a ~/.R/Makevars file and what it contains
• The exact call that you use to install sparseMatrixStats including the full error message

const-ae/sparseMatrixStats documentation built on April 10, 2024, 5:27 p.m.