create_job: Create a new Job on Databricks

In RafiKurlansik/bricksteR: R Package for Making Databricks Easy to Use.

Create a new Job on Databricks

Description

This function will create a new job on Databricks, but will not run it. To run a job, see run_job or runs_submit.

Usage

create_job(
  name = "R Job",
  file = NULL,
  notebook_path,
  job_config = "default",
  workspace,
  token = NULL,
  verbose = T,
  ...
)

Arguments

name	A string representing the name of the job. It is encouraged to choose a unique name for each job.
file	The path to a local .R or .Rmd file. Will be imported to the workspace at the notebook_path.
notebook_path	A string representing the path to a Databricks notebook in the workspace.
job_config	A JSON formatted string or file specifying the details of the job, i.e., the name, cluster spec, and so on.
workspace	A string representing the web workspace of your Databricks instance. E.g., "https://eastus2.azuredatabricks.net" or "https://demo.cloud.databricks.com".
token	A valid authentication token generated via User Settings in Databricks or via the Databricks REST API 2.0. If none is provided, netrc will be used.
verbose	If true, will pretty print the success or failure of the request and add a 'job_id' variable to the R environment. Defaults to TRUE.
...	additional arguments to be passed, i.e., overwrite = 'false' when importing a file to run as a job.

Details

The API endpoint for creating a job is '2.0/jobs/create'. For all details on API calls please see the official documentation at https://docs.databricks.com/dev-tools/api/latest/.

Value

A list with two elements - the complete API response and the job ID.

Examples

# Default JSON used
create_job(path = "/Shared/R/brickster_tutorial", # A notebook in the workspace
 workspace = "https://dbc-z64b06b4-d212.cloud.databricks.com", # The workspace of your Databricks instance
  token = "dapi30912309sdfdsa9iu09") # The valid auth token

# Passing custom JSON
job_config <- '{"name": "New R Job",
                   "new_cluster": {
                       "spark_version": "5.5.x-scala2.11",
                       "node_type_id": "r3.xlarge",
                       "aws_attributes": {
                           "availability": "ON_DEMAND"
                       },
                       "num_workers": 2,
                       "email_notifications": {
                           "on_start": [],
                           "on_success": [],
                           "on_failure": []
                       },
                       "notebook_task": {
                           "notebook_path": "/Shared/R/brickster_tutorial"
                       }
                   }
                 }'

# Specifying the path now unnecessary
create_job(job_config,
workspace = "https://dbc-z64b06b4-d212.cloud.databricks.com",
token = "dapi310240980a9dgqwebdsfadsf21")

RafiKurlansik/bricksteR documentation built on Oct. 13, 2022, 6:58 a.m.