eval_sparklyr <- FALSE if(Sys.getenv("GLOBAL_EVAL") != "") eval_sparklyr <- Sys.getenv("GLOBAL_EVAL")
sparklyr
library(dplyr) library(sparklyr)
Learn to open a new Spark session
Load the sparklyr
library
r
library(sparklyr)
Use spark_connect()
to create a new local Spark session
r
sc <- spark_connect(master = "local")
Click on the Spark
button to view the current Spark session's UI
Click on the Log
button to see the message history
Practice uploading data to Spark
Load the dplyr
library
r
library(dplyr)
Copy the mtcars
dataset into the session
r
spark_mtcars <- copy_to(sc, mtcars, "my_mtcars")
In the Connections pane, expande the my_mtcars
table
Go to the Spark UI, note the new jobs
In the UI, click the Storage button, note the new table
Click on the In-memory table my_mtcars link
dplyr
See how Spark handles dplyr
commands
Run the following code snipett
r
spark_mtcars %>%
group_by(am) %>%
summarise(mpg_mean = mean(mpg, na.rm = TRUE))
Go to the Spark UI and click the SQL button
Click on the top item inside the Completed Queries table
At the bottom of the diagram, expand Details
Introduction to how Spark Feature Transformers can be called from R
Use ft_binarizer()
to create a new column, called over_20
, that indicates if that row's mpg
value is over or under 20MPG
```r
```
Pipe the code into count()
to see how the data splits between the two values
```r
```
Start a new code chunk. This time use ft_quantile_discretizer()
to create a new column called mpg_quantile
```r
```
Add the num_buckets
argument to ft_quantile_discretizer()
, set its value to 5
```r
```
count()
to see how the data splits between the quantiles
```r```
Introduce Spark ML models by running a couple of them in R
Use ml_kmeans()
to run a model based on the following formula: wt ~ mpg
. Assign the results to a variable called k_mtcars
r
k_mtcars <-
Use k_mtcars$summary
to view the results of the model. Pull the cluster sizes by using ...$cluster_sizes()
r
k_mtcars$summary$cluster_sizes()
Start a new code chunk. This time use ml_linear_regression()
to produce a Linear Regression model of the same formula used in the previous model. Assign the results to a variable called lr_mtcars
r
lr_mtcars <-
Use summary()
to view the results of the model
```r
```
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.