orderly_runner_: Object for managing running jobs on the redis queue

In vimc/orderly.server: Serve Orderly

Object for managing running jobs on the redis queue

Description

Object for managing running jobs on the redis queue

Object for managing running jobs on the redis queue

Public fields

root

Orderly root

config

Orderly config

allow_ref

Allow git to change branch/ref for run

default_branch

Default git branch

con

Redis connection

cleanup_on_exit

If TRUE workers are killed on exit

queue

The redis queue

queue_id

Redis queue ID

keys

Set of redis keys for mapping between key, report_id and task_id

Methods

Public methods

• orderly_runner_$new()

• orderly_runner_$start_workers()

• orderly_runner_$assert_ref_switching_allowed()

• orderly_runner_$submit_task_report()

• orderly_runner_$submit()

• orderly_runner_$submit_workflow()

• orderly_runner_$status()

• orderly_runner_$workflow_status()

• orderly_runner_$queue_status()

• orderly_runner_$check_timeout()

• orderly_runner_$kill()

• orderly_runner_$destroy()

• orderly_runner_$cleanup()

---

Method new()

Create object, read configuration and setup redis connection.

Usage

orderly_runner_$new(
  root,
  identity,
  queue_id,
  workers,
  cleanup_on_exit = workers > 0,
  worker_timeout = Inf
)

Arguments

root

Orderly root.

identity

Remote identity, as listed in orderly_config.yml

queue_id

ID of an existing queue to connect to, creates a new one if NULL.

workers

Number of workers to spawn.

cleanup_on_exit

If TRUE workers are killed on exit.

worker_timeout

How long worker should live for before it is killed. Expect this is only finite during local testing.

---

Method start_workers()

Start n workers for this queue and optionally set a timeout.

Usage

orderly_runner_$start_workers(workers, timeout)

Arguments

workers

Number of workers to spawn.

timeout

How long worker should live for before it is killed. Expect this is only finite during local testing.

Returns

TRUE, called for side effects.

---

Method assert_ref_switching_allowed()

Check if ref switching is allowed in this runner. Errors if ref is non NULL and ref switching disallowed, otherwise does nothing.

Usage

orderly_runner_$assert_ref_switching_allowed(ref)

Arguments

ref

Input ref to check

Returns

TRUE, called for side effects.

---

Method submit_task_report()

Queue a job to run an orderly report.

Usage

orderly_runner_$submit_task_report(
  name,
  parameters = NULL,
  ref = NULL,
  instance = NULL,
  changelog = NULL,
  poll = 0.1,
  timeout = 60 * 60 * 3,
  depends_on = NULL
)

Arguments

name

Name of report to be queued.

parameters

List of parameters to pass to report.

ref

The git sha to run the report.

instance

The db instance for the report to pull data from.

changelog

Description of changes to the report.

poll

How frequently to poll for the report ID being available.

timeout

Timeout for the report run default 3 hours.

depends_on

Keys of any tasks which this report depends on

Returns

The key for the job, note this is not the task id. The task id can be retrieved from redis using the key.

---

Method submit()

Submit an arbitrary job on the queue

Usage

orderly_runner_$submit(expr, depends_on = NULL)

Arguments

expr

A quoted R expression.

depends_on

Task ids for any dependencies of this job.

Returns

Task id

---

Method submit_workflow()

Queue a workflow.

Usage

orderly_runner_$submit_workflow(
  reports,
  ref = NULL,
  changelog = NULL,
  poll = 0.1,
  timeout = 60 * 60 * 3
)

Arguments

reports

Details of reports to be run.

ref

The git sha to run the workflow.

changelog

Description of changes to the reports - applied to all reports.

poll

How frequently to poll for the report ID being available. Applied to each of the reports run in the workflow

timeout

Timeout for each of the reports run as part of the workflow default 3 hours.

Returns

The key for the workflow and each individual report

---

Method status()

Get the status of a job

Usage

orderly_runner_$status(key, output = FALSE)

Arguments

key

The job key.

output

If TRUE include the output from job running.

Returns

List containing the key, status, report_id (if available), output and the position of the job in the queue.

---

Method workflow_status()

Get the status of a workflow.

Usage

orderly_runner_$workflow_status(workflow_key, output = FALSE)

Arguments

workflow_key

The workflow key.

output

If TRUE include the output from each job in the workflow.

Returns

List containing the workflow_key, status and status of each job in the workflow.

---

Method queue_status()

Get the running and queued tasks in front of key in the queue.

If key is NULL then includes all queued tasks.

Usage

orderly_runner_$queue_status(key = NULL)

Arguments

key

The job key, if NULL returns all queued tasks.

Returns

List containing the key, status and report name of any running tasks and any queued tasks in front of key in the queue.

---

Method check_timeout()

Check if any running tasks have passed their timeouts. This is run by the API on a preroute - we check for timeouts everytime someone interacts with the API. Not intended to be run directly.

Usage

orderly_runner_$check_timeout()

Returns

List of killed reports.

---

Method kill()

Kill a job

Usage

orderly_runner_$kill(key)

Arguments

key

The job key.

---

Method destroy()

Destroy the queue. Not expected to be called directly, used in tests.

Usage

orderly_runner_$destroy()

---

Method cleanup()

Cleanup workers and destroy the queue. Not expected to be called directly, gets registered as finaliser of the object.

Usage

orderly_runner_$cleanup()

vimc/orderly.server documentation built on July 27, 2023, 5:07 p.m.