restbench-tasks: Create/Restore a client-side task proxy

Description Usage Arguments Details Value Fields Methods Examples

Description

Create/Restore a client-side task proxy

Usage

1
2
3
new_task2(fun, ..., task_name = "Noname", globals = TRUE, env = parent.frame())

restore_task2(task_name)

Arguments

fun, ...

function to apply and additional parameters to be passed to batchMap function

task_name

a readable name for the task; default is 'Noname'.

globals

variables to export to the computing processes; see details.

env

environment where the 'globals' are defined

Details

A task proxy manages a folder at task_dir containing all the information necessary to perform the task. A task is a collection of "jobs" (see its definition in batchMap). The evaluation of the jobs is always in different processes with the current R session, sometimes even different computers.

Sometimes the function fun may contains external objects that need to be exported. For example, function(x){ a+x } has a variable a that might be defined in somewhere else. It will raise errors if we simply execute the function. Therefore, the task requires exporting the variable a before executing jobs. This task could be done automatically or manually. If globals is TRUE, then all the external variables including the packages needed will be automatically detected and exported. If globals is a character vector, then only the variables indicated will be correctly exported If globals is FALSE, then you need to manually specify them in task$set_globals method

Value

If the task does not exist, the returns NULL, otherwise returns a locked environment proxy that wraps batch task with the following fields and methods.

Fields

task_name

task ID, a combination of user ID, task readable name, and a 16-character random string

readable_name

a friendly name of the task

task_dir

task directory where the jobs are stored

task_root

root directory where all tasks are stored

reg

'batchtools' registry that reads batch job status

njobs

total number of jobs in the task

protocol, host, port, path_*

server to be submitted to, used by 'submit()' method

submitted

whether the task has been submitted or not

submitted_to

if the task has been submitted, then where

results

the result of the task, should almost surely use collect() instead

collected

whether the results has been loaded from task directory into the memory

Methods

submit

submit or re-submit a task. If 'pack' is true, then the task files will be archived before sending to the server. This option is required when server runs remotely.

resolved

check whether the task has been resolved, a combination of server_status() and local_status(). Will raise errors if server if not running

locally_resolved

check whether task is resolved locally without query the server. Only used to check local files. It's possible that server has finished but locally not resolved. See download() to synchronize local files with the finished tasks on the server

server_status

get task status from servers

local_status

get task status from servers

download

download the finished tasks from the server and overwrite the local files. Raises error if the task is not finished remotely. Always check server status before downloading.

collect

collect results. It checks the status both locally and remotely; download the task if finished; load up the results. This function will block the session waiting for results.

zip

create an archive that stores the task

clear_registry

clear the registry, usually used to re-use the task or to re-run the task

reload_registry

reload 'reg' registry under read-only or writable modes. If the task has been submitted, please load in read-only mode to avoid file corrects

remove

remove the whole task from the hard drive

Examples

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
task <- new_task2(function(...){
  list(...)
}, x = 1:3, y = 1:6)

# Not submitted
task

# save task name to same other places
task_name <- task$task_name

# submit the task
if(interactive()){

  ensure_server()

  task$submit()

  task

  # once finished
  kill_server()

}

# Remove the task. You can always restore with task_name

rm(task)

task_restored <- restore_task2(task_name = task_name)

if(task_restored$submitted && task_restored$resolved()){
  task_restored$collect()
}

# clean up, remove all task files
task_restored$remove()

dipterix/restbatch documentation built on Oct. 13, 2021, 8:58 a.m.