convertKafkaValueFromJson: Connect to a Spark session

In analysisPipelines: Compose Interoperable Analysis Pipelines & Put Them in Production

Description

Connect to a Spark session

Usage

convertKafkaValueFromJson(streamObj, schema)

Arguments

streamObj	Spark Structured Streaming DataFrame which is returned by the castKafkaStreamAsString function
schema	A structType object created from SparkR specifying the schema of the json data present in the value attribute of the incoming Kafka stream

Details

Takes in a Structured Stream from Kafka created from read.stream(source = 'kafka', ...) and returns a Structured Streaming DataFrame where the key and value from the Kafka stream are cast to string

Value

Spark Structured Streaming DataFrame with the json data in the value attribute of the Kafka stream parsed into a DataFrame format

See Also

Other Spark utilities: castKafkaStreamAsString, sparkRSessionCreateIfNotPresent

analysisPipelines documentation built on July 1, 2020, 7:09 p.m.