queue_didehpc: Create a queue object

In dide-tools/didehpc: DIDE HPC Support

Create a queue object

Description

Create a queue object. This is an R6::R6Class object which you interact with by calling "methods" which are described below, and on the help page for queuer::queue_base, from which this derives.

Usage

queue_didehpc(
  context,
  config = didehpc_config(),
  root = NULL,
  initialise = TRUE,
  provision = NULL,
  login = NULL
)

Arguments

context	A context
config	Optional dide configuration information.
root	A root directory, not usually needed
initialise	Passed through to the base queue. If you set this to FALSE you will not be able to submit tasks. By default if FALSE this also sets provision to later and login to FALSE.
provision	A provisioning strategy to use. Options are verylazy (the default) which installs packages if any declared package is not present, or if the remote library has already been provisioned. This is lazier than the lazy policy and faster as it avoids fetching package metadata, which may take a few seconds. If you have manually adjusted your library (especially by removing packages) you will probably want to use lazy or upgrade to account for dependencies of your declared packages. lazy: which tells pkgdepends to be "lazy" - this prefers to minimise installation time and does not upgrade packages unless required. upgrade: which tells pkgdepends to always try and upgrade later: don't do anything on creation fake: don't do anything but mark the queue as being already provisioned (this option can come in useful if you really don't want to risk any accidental package installation)
login	Logical, indicating if we should immediately login. If TRUE, then you will be prompted to login immediately, rather than when a request to the web portal is made.

Super class

queuer::queue_base -> queue_didehpc

Public fields

config

Your didehpc_config() for this queue. Do not change this after queue creation as changes may not take effect as expected.

client

A web_client object used to communicate with the web portal. See the help page for its documentation, but you will typically not need to interact with this.

Methods

Public methods

• queue_didehpc_$new()

• queue_didehpc_$login()

• queue_didehpc_$cluster_load()

• queue_didehpc_$reconcile()

• queue_didehpc_$submit()

• queue_didehpc_$submit_workers()

• queue_didehpc_$stop_workers()

• queue_didehpc_$rrq_controller()

• queue_didehpc_$unsubmit()

• queue_didehpc_$dide_id()

• queue_didehpc_$dide_log()

• queue_didehpc_$provision_context()

• queue_didehpc_$install_packages()

Inherited methods

• queuer::queue_base$enqueue()
• queuer::queue_base$enqueue_()
• queuer::queue_base$enqueue_bulk()
• queuer::queue_base$initialize_context()
• queuer::queue_base$lapply()
• queuer::queue_base$mapply()
• queuer::queue_base$task_bundle_get()
• queuer::queue_base$task_bundle_info()
• queuer::queue_base$task_bundle_list()
• queuer::queue_base$task_bundle_retry_failed()
• queuer::queue_base$task_delete()
• queuer::queue_base$task_get()
• queuer::queue_base$task_list()
• queuer::queue_base$task_result()
• queuer::queue_base$task_retry_failed()
• queuer::queue_base$task_status()
• queuer::queue_base$task_times()

---

Method new()

Constructor

Usage

queue_didehpc_$new(
  context,
  config,
  root,
  initialise,
  provision,
  login,
  client = NULL
)

Arguments

context, config, root, initialise, provision, login

See above

client

A web_client object, primarily useful for testing the package

---

Method login()

Log onto the web portal. This will be called automatically at either when creating the object (by default) or when you make your first request to the portal. However, you can call this to refresh the session too.

Usage

queue_didehpc_$login(refresh = TRUE)

Arguments

refresh

Logical, indicating if we should try logging on again, even if it looks like we already have. This will refresh the session, which is typically what you want to do.

---

Method cluster_load()

Report on the overall cluster usage

Usage

queue_didehpc_$cluster_load(cluster = NULL, nodes = TRUE)

Arguments

cluster

Cluster to show; if TRUE show the entire cluster (via load_overall), if NULL defaults to the value config$cluster

nodes

Show the individual nodes when printing

---

Method reconcile()

Attempt to reconcile any differences in task state between our database and the HPC queue. This is needed when tasks have crashed, or something otherwise bad has happened and you have tasks stuck in PENDING or RUNNING that are clearly not happy. This function does not play well with workers and you should not use it if using them.

Usage

queue_didehpc_$reconcile(task_ids = NULL)

Arguments

task_ids

A vector of tasks to check

---

Method submit()

Submit a task to the queue. Ordinarily you do not call this directly, it is called by the ⁠$enqueue()⁠ method of queuer::queue_base when you create a task. However, you can use this to resubmit a task that has failed if you think it will run successfully a second time (e.g., because you cancelled it the first time around).

Usage

queue_didehpc_$submit(task_ids, names = NULL, depends_on = NULL)

Arguments

task_ids

A vector of task identifiers to submit.

names

Optional names for the tasks.

depends_on

Optional vector of dependencies, named by task id

---

Method submit_workers()

Submit workers to the queue. This only works if use_rrq or use_workers is TRUE in your configuration. See vignette("workers") for more information.

Usage

queue_didehpc_$submit_workers(n, timeout = 600, progress = NULL)

Arguments

n

The number of workers to submit

timeout

The time to wait, in seconds, for all workers to come online. An error will be thrown if this time is exceeded.

progress

Logical, indicating if a progress bar should be printed while waiting for workers.

---

Method stop_workers()

Stop workers running on the cluster. See vignette("workers") for more information. By default workers will timeout after 10 minutes of inactivity.

Usage

queue_didehpc_$stop_workers(worker_ids = NULL)

Arguments

worker_ids

Vector of worker names to try and stop. By default all workers are stopped.

---

Method rrq_controller()

Return an rrq::rrq_controller object, if you have set use_rrq or use_workers in your configuration. This is a lightweight queue using your workers which is typically much faster than submitting via ⁠$enqueue()⁠. See vignette("workers") for more information.

Usage

queue_didehpc_$rrq_controller()

---

Method unsubmit()

Unsubmit tasks from the cluster. This removes the tasks from the queue if they have not been started yet, and stops them if currently running. It will have no effect if the tasks are completed (successfully or errored)

Usage

queue_didehpc_$unsubmit(task_ids)

Arguments

task_ids

Can be a task id (string), a vector of task ids, a task, a list of tasks, a bundle returned by enqueue_bulk, or a list of bundles.

---

Method dide_id()

Find the DIDE task id of your task. This is the number displayed in the web portal.

Usage

queue_didehpc_$dide_id(task_ids)

Arguments

task_ids

Vector of task identifiers to look up

---

Method dide_log()

Return the pre-context log of a task. Use this to find out what has happened to a task that has unexpectedly failed, but for which ⁠$log()⁠ is uninformative.

Usage

queue_didehpc_$dide_log(task_id)

Arguments

task_id

A single task id to check

---

Method provision_context()

Provision your context for running on the cluster. This sets up the remote set of packages that your tasks will use. See vignette("packages") for more information.

Usage

queue_didehpc_$provision_context(
  policy = "verylazy",
  dryrun = FALSE,
  quiet = FALSE,
  show_progress = NULL,
  show_log = TRUE
)

Arguments

policy

The installation policy to use, as interpreted by pkgdepends::pkg_solution - so this should be verylazy/lazy (install missing packages but don't upgrade unless needed) or upgrade (upgrade packages as possible). In addition you can also use later which does nothing, or fake which pretends that it ran the provisioning. See vignette("packages") for details on these options.

dryrun

Do a dry run installation locally - this just checks that your requested set of packages is plausible, but does this without submitting a cluster job so it may be faster.

quiet

Logical, controls printing of informative messages

show_progress

Logical, controls printing of a spinning progress bar

show_log

Logical, controls printing of the log from the cluster

---

Method install_packages()

Install packages on the cluster. This can be used to more directly install packages on the cluster than the ⁠$provision_context⁠ method that you would typically use. See vignette("packages") for more information.

Usage

queue_didehpc_$install_packages(
  packages,
  repos = NULL,
  policy = "lazy",
  dryrun = FALSE,
  show_progress = NULL,
  show_log = TRUE
)

Arguments

packages

A character vector of packages to install. These can be names of CRAN packages or GitHub references etc; see pkgdepends::new_pkg_installation_proposal() and vignette("packages") for more details

repos

A character vector of repositories to use when installing. A suitable CRAN repo will be added if not detected.

policy

The installation policy to use, as interpreted by pkgdepends::pkg_solution - so this should be lazy (install missing packages but don't upgrade unless needed) or upgrade (upgrade packages as possible). In addition you can also use later which does nothing, or fake which pretends that it ran the provisioning. See vignette("packages") for details on these options.

dryrun

Do a dry run installation locally - this just checks that your requested set of packages is plausible, but does this without submitting a cluster job so it may be faster.

show_progress

Logical, controls printing of a spinning progress bar

show_log

Logical, controls printing of the log from the cluster

dide-tools/didehpc documentation built on Aug. 20, 2023, 9:27 a.m.