README-NOT.md

In ropenscilabs/jobstatus: Send Live Status, Progress and Other Information Between Functions and Processes

jobstatus

jobstatus lets you pass live progress, status, and other information between functions and processes in R, so that you can keep an eye on how complex and long-running jobs are progressing. jobstatus uses the future package so you can even get live progress information back from jobs running in parallel.

How it works

jobstatus passes status information between functions and processes in R so you can monitor what’s happening and how much progress the code has made. There are three main functions in jobstatus:

• jobstatus creates a jobstatus object, which lets you record your progress on a single task
• with_jobstatus runs a chunk of code that runs other jobs, and keeps track of the status information
• subjob_future is a wrapper function for future::future that lets you write code that can be executed either in sequence or in parallel.

with_jobstatus can also be used to display the progress information, for example with a progress bar.

A jobstatus set up might look something like this:

library (jobstatus)
library (progress)

# a function to run a single task
some_big_task <- function (iterations = 100) {

  # create a jobstatus object to track progress
  # do the work, incrementing the progress counter each time
  status <- jobstatus$new(iterations)

  result <- 0
  for (i in seq_len(iterations)) {
    result <- result + 1
    Sys.sleep(runif(1, 0, 0.2))
    status$tick()
  }

  # tidy up when we're done, and return the results
  status$finish()

  result

} 

# load the future package and set it to run the jobs in parallel
library (future)
plan(multiprocess)

# dispatch some jobs, tracking their status information and displaying multiple progress bars
with_jobstatus({

  # create some futures (possibly in parallel)
  f1 <- subjob_future(some_big_task())
  f2 <- subjob_future(some_big_task())
  f3 <- subjob_future(some_big_task())
  f4 <- subjob_future(some_big_task())

  # get their values
  v1 <- value(f1)
  v2 <- value(f2)
  v3 <- value(f3)
  v4 <- value(f4)

}, display = percentage)

Extensions

This is very much a work in progress and the current implementation is quite limited (and probably quite buggy). Ideally, we’d like it to support various different parallel backends and interfaces, handle other types of job status information, and provide different types of progress bars and displays.

This prototype was developed over two days at the 2018 rOpenSci unconference and the maintainers won’t have much time to extend and improve the package. We’d love to have help, so if you’re keen please let us know in the issues tracker!

Installation

You can install the development version from GitHub with:

# install.packages("devtools")
devtools::install_github("ropenscilabs/jobstatus")

ropenscilabs/jobstatus documentation built on May 11, 2022, 9:10 a.m.