README.md
In jpmml/sparklyr2pmml: R library for converting Apache Spark ML pipelines to PMML

Sparklyr2PMML

R library for converting Apache Spark ML pipelines to PMML.

Features

This package is a thin R wrapper for the JPMML-SparkML library.

Prerequisites

Apache Spark 3.0.X, 3.1.X, 3.2.X, 3.3.X, 3.4.X or 3.5.X.
R 3.3 or newer.

Installation

Install from GitHub using the devtools package:

library("devtools")

install_github("jpmml/sparklyr2pmml")

Configuration and usage

Sparklyr2PMML must be paired with JPMML-SparkML based on the following compatibility matrix:

Active development branches:

| Apache Spark version | JPMML-SparkML branch | Latest JPMML-SparkML version | |----------------------|----------------------|------------------------------| | 3.4.X | 3.0.X | 3.0.0 | | 3.5.X | master | 3.1.0 |

Stale development branches:

| Apache Spark version | JPMML-SparkML branch | Latest JPMML-SparkML version | |----------------------|----------------------|------------------------------| | 3.0.X | 2.0.X | 2.0.6 | | 3.1.X | 2.1.X | 2.1.6 | | 3.2.X | 2.2.X | 2.2.6 | | 3.3.X | 2.3.X | 2.3.5 | | 3.4.X | 2.4.X | 2.4.4 | | 3.5.X | 2.5.X | 2.5.3 |

Launch Sparklyr; use the sparklyr.connect.packages configuration option to specify the coordinates of relevant JPMML-SparkML modules:

org.jpmml:pmml-sparkml:${version} - Core module.
org.jpmml:pmml-sparkml-lightgbm:${version} - LightGBM via SynapseML extension module.
org.jpmml:pmml-sparkml-xgboost:${version} - XGBoost via XGBoost4J-Spark extension module.

Launching core:

library("sparklyr")

config = spark_config()
config[["sparklyr.connect.packages"]] = "org.jpmml:pmml-sparkml:${version}"

sc = spark_connect(master = "local", config = config)

Fitting a Spark ML pipeline:

library("dplyr")
library("sparklyr")

data(iris)

iris_df = copy_to(sc, iris)

iris_pipeline = ml_pipeline(sc) %>%
    ft_r_formula(Species ~ .) %>%
    ml_decision_tree_classifier()

iris_pipeline_model = ml_fit(iris_pipeline, iris_df)

Exporting the fitted Spark ML pipeline to a PMML file:

library("sparklyr2pmml")

pmmlBuilder = PMMLBuilder(sc, iris_df, iris_pipeline_model)

buildFile(pmmlBuilder, "DecisionTreeIris.pmml")

License

Sparklyr2PMML is licensed under the terms and conditions of the GNU Affero General Public License, Version 3.0.

If you would like to use Sparklyr2PMML in a proprietary software project, then it is possible to enter into a licensing agreement which makes Sparklyr2PMML available under the terms and conditions of the BSD 3-Clause License instead.