analysisPipelines: analysisPipelines

In analysisPipelines: Compose Interoperable Analysis Pipelines & Put Them in Production

Description

The package aims at enabling data scientists to compose pipelines of analysis which consist of data manipulation, exploratory analysis & reporting, as well as modeling steps. It also aims to enable data scientists to use tools of their choice through an R interface, and compose interoperable pipelines between R, Spark, and Python. Credits to Mu Sigma for supporting the development of the package.

Note

To enable pipelines involving Spark tasks, the package uses the 'SparkR' package. Using Spark as an engine requires the SparkR package to be installed. SparkR is distributed natively with Apache Spark and is not distributed on CRAN. The SparkR version needs to directly map to the Spark version (hence the native distribution), and care needs to be taken to ensure that this is configured properly. To install from Github, run the following command, if you know the Spark version:

• devtools::install_github('apache/spark@v2.x.x', subdir='R/pkg')

The other option is to install SparkR by running the following terminal commands if Spark has already been installed:

• $ export SPARK_HOME=/path/to/spark/directory

• $ cd $SPARK_HOME/R/lib/SparkR/

• $ R -e "devtools::install('.')"

analysisPipelines documentation built on July 1, 2020, 7:09 p.m.