create_and_run_job: Create and run a job on Databricks

In RafiKurlansik/bricksteR: R Package for Making Databricks Easy to Use.

Create and run a job on Databricks

Description

Single function to both create a job and immediately run it. Each call will register a new job in the Databricks Jobs UI, so for subsequent runs it is best to submit the job_id to run_job.

Usage

create_and_run_job(
  name = "R Job",
  file = NULL,
  notebook_path,
  job_config = "default",
  workspace,
  token = NULL,
  ...
)

Arguments

name	A string representing the name of the job. It is encouraged to choose a unique name for each job.
file	The path to a local .R or .Rmd file. Will be imported to the workspace at the notebook_path.
notebook_path	A string representing the path to a Databricks notebook in the workspace.
job_config	A JSON formatted string or file specifying the details of the job, i.e., the name, cluster spec, and so on.
workspace	A string representing the web workspace of your Databricks instance. E.g., "https://eastus2.azuredatabricks.net" or "https://demo.cloud.databricks.com".
token	A valid authentication token generated via User Settings in Databricks or via the Databricks REST API 2.0. If none is provided, netrc will be used.
...	additional arguments to be passed, i.e., overwrite = 'false' when importing a file to run as a job.

Details

Uses '2.0/jobs/create', '2.0/jobs/run-now' and '2.0/runs/get' API endpoints. For all details on API calls please see the official documentation at https://docs.databricks.com/dev-tools/api/latest/.

Value

A list with three elements containing the values of create_job, run_job, and get_run_status:

• job - The full API response, including status code.

• run - The JSON result.

• run_status - The JSON result transformed into a list.

Examples

results <- create_and_run_job(name = "Forecasting Report",
                              notebook_path = "/Shared/forecasting_report_notebook".
                              workspace = "https://eastus2.azuredatabricks.net",
                              token = "dapi2930482309ddavascscq")

# Get each response
job <- results$job
run <- results$run
run_status <- results$run_status

RafiKurlansik/bricksteR documentation built on Oct. 13, 2022, 6:58 a.m.