In benjaminguinaudeau/executoR:

knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>",
  fig.path = "man/figures/README-",
  out.width = "100%"
)
pacman::p_load(tidyverse)
devtools::load_all()
# devtools::document()
# devtools::install()

ExecutoR

Executor orchestrates scripts: it can run, schedule and restart the execution of scripts. Scripts are executed in the background (similarly to rstudio jobs). Executor (unlike rstudio jobs) can be used in any environment, even if rstudio is not running.

To create an instance, you simply need to specify a folder, where all skript logs will be saved and an executor_id. If you don't specify an id, a random one will be automatically attributed.

# Random ID
exec_model <- executor$new(folder = "exec_test")
exec_model$ex_id

# Specified id
exec_model <- executor$new(folder = "exec_test", exec_id = "xg15")
exec_model$ex_id

Workflow

To execute a task, you need to:

1. Register the task in the executor
2. Start the task

Adding tasks

A task is basically a script and a name. Within one executof two tasks cannot have the same name.

To add a task to an existing executor, you need to specify at least two arguments:

• a name: general description of the task, that will be used as an id internally
• a script path: a path to the script that should be executed

You can eventually specify further arguments

• wd: the working directory for the script
• env: environment variables that should be provided to the script. This can be used to parametrize scripts.
• infinite_loop: when True (default), the task will be restarted each times an errors occurs
• period: a string specifying how often the task should executed. "1 hour" to execute the task every hour ; "1 day" for every day ; etc..
• start: a timestamp specifying when the interval should start. To execute a task once a day at 13:00, use period = "1 day" and start = lubridate::as_datetime("2021-02-27 13:00:00")

## Adding one task
exec_model$add_task(name = "LOL", script = "jobs/test.R", wd = getwd(), env = c("SYMBOL" = "LOL"))
exec_model$add_task(name = "LOL_Schedule", script = "jobs/test.R", wd = getwd(), env = c("SYMBOL" = "LOL_schedule"), infinite_loop = F, period = "2 min")
exec_model$add_task(name = "LOL_Schedule_day", script = "jobs/test.R", wd = getwd(), env = c("SYMBOL" = "LOL_schedule"), infinite_loop = F, period = "day", start = lubridate::as_datetime("2021-02-27 12:50:00", tz = "EST"))
## Adding a serie of task
c("LMFAO", "LOL") %>%
  purrr::walk(~{
    exec_model$add_task(name = .x, script = "jobs/test.R", wd = getwd(), env = c("SYMBOL" = .x))
  })

Start/Stop tasks

Once a task is added, it can be start/stop. Remember that the name provided to the task it the internal id, hence it should be unique.

# Start task
exec_model$start_task(name = "LOL")
# Start all exising tasks
exec_model$start_all()

# Stop task
exec_model$stop_task(name = "LOL")
# Stop all tasks
exec_model$stop_all()

Running the executor

To make sure, the scheduled script are correctly executed or breaking scripts are restarted, the executor needs to be runned. This will basically check every 60 seconds, if a script that should be running is running. In doing so, it will run script that were scheduled for the past 60 seconds and restart scripts that are forever loops.

sleep <- 60 # in seconds
exec_model$keep_restarting(sleep = sleep) # forever loop, you'll need to stop this, once this is started

An overall log is stored in a log vector

exec_model$log

Retrieving information

Meta information

To know the existing tasks in an executor, you can take a look at exec_model$task

exec_model$tasks

If you're interested in which scripts are running, you can use the following function:

exec_model$list_running_task()

exec_model$list_running_task(next_run = T)

Individual script information

Each script produces one output, saved in the log_folder specified when creating the executor.

If you navigate to "test", you will see that each script has one output file LOL.txt. If the script breaks and is restarted, the previous output is archived with a timestamp, so that we can know which error happened. So LOL_111111111.txt is the output of a previous run of the task LOL. Using this output, you can communicate with what's happening in the process.

exec_model$read_out(name = "LOL", n_tail = 50) %>% glimpse # Read the last 50 lines of the stream output of the task LOL

benjaminguinaudeau/executoR documentation built on June 11, 2022, 12:33 p.m.