databricks_execute: Remote execution of commands on a Databricks cluster.

In RafiKurlansik/bricksteR: R Package for Making Databricks Easy to Use.

Remote execution of commands on a Databricks cluster.

Description

This function sends commands to an execution context on an existing Databricks cluster via REST API. It requires a context_id from create_execution_context. Commands must be compatible with the language of the execution context - 'r', 'python', 'scala', or 'sql'. Will attempt to return a data.frame but if the execution hasn't finished will return the status of execution. If your command does not return a data.frame output may vary considerably, or fail.

Usage

databricks_execute(command, context, verbose = F, ...)

Arguments

command	A string containing commands for remote execution on Databricks.
context	The list generated by create_execution_context
verbose	If TRUE, will print the API response to the console. Defaults to FALSE.
...	Additional options to be passed to data.table::fread which is used to parse the API response.

Details

The API endpoint for creating the execution context is is '1.2/commands/execute'. For all details on API calls please see the official documentation at https://docs.databricks.com/dev-tools/api/latest/.

Value

A list with two components:

• response - The full API response.

• data - The data as a data.frame.

Examples

# Using netrc
context <- create_execution_context(workspace = "https://eastus2.azuredatabricks.net",
                  language = "r",
                  cluster_id = "1017-337483-jars232")

## Use the context to execute a command on Databricks
command <- "iris[1, ]"
result <- databricks_execute(command, context)

## Access dataframe
result$data

RafiKurlansik/bricksteR documentation built on Oct. 13, 2022, 6:58 a.m.