minionWorker: A function that watches a message queue and starts working a...

In PieceMaker/RMinions: A Package for Easy R Processing Scalability Via Message Queues

Description

minionWorker is a blocking function that will completely block the R instance running it. It immediately performs some housekeeping by gathering logging information and connecting to the message queue. Once connected, it performs a blocking pop on the jobsQueue and waits until it receives a bundled job message. Once a message is received, it pulls the specified function for the specified package and runs it with the bundled parameters and stores any returned results in the bundled response key.

Usage

minionWorker(host, port = 6379, jobsQueue = "jobsQueue",
  logLevel = "DEBUG", logFileDir = "stdout", useJSON = F,
  whitelist = NULL)

Arguments

host	The name or ip address of the redis server.
port	The port the redis server is running on. Defaults to 6379.
jobsQueue	A string giving the name of the queue where jobs will be placed. Defaults to jobsQueue.
logLevel	A string, required. 'TRACE', 'DEBUG', 'INFO', 'WARN', 'ERROR', 'FATAL'. Level threshold required to trigger log write. You can change the level on an existing log.
logFileDir	A string giving the directory to store worker log files if logging is enabled. Defaults to stdout which outputs logs to standard out. Specify directory if you want logs to be written to a file. If a directory is specified, then logs will be saved in that directory to a file with name <host>-worker-<pid>.log.
useJSON	Flag specifying whether jobs and results will be sent in JSON format. Defaults to false so R-specific objects are preserved in transit. If sending jobs from languages other than R, set useJSON to true and make sure jobs are defined in the JSON format and the function being executed does not require any R-specific objects.
whitelist	String of list. If a list, it defines the whitelist using the same format as the blacklist. If a string, then it gives the path to a JSON file that defines the whitelist. If a whitelist is defined, then a function will only be allowed if it is in the specified list. Note that the whitelist takes precedent over the blacklist.

Details

minionWorker is the core of the rminions package and multiple R processes running this function should be spawned. The goal is to spawn just enough workers that the full cpu resources of the system running them are maxed out when under full jobs load. A good rule of thumb is that you should have number of cores + 2 workers running on a system.

The minion worker is constructed to work with nearly all tasks. In order to accomplish this, job messages need to be of a specific format. Job message must be lists with the following keys: package, func, parameters, resultsQueue, and optionally, errorQueue.

package is the name of the package containing the function that will be executed.

func is the name of the function, contained in the specified package, that you wish to execute with this request. If useJSON is false, func can take any type of parameter. If useJSON is true, then func should only take R data as parameters that can be converted to and from JSON, i.e. strings, numbers, simple vectors, etc.

parameters is a named list of all the parameters to be passed to func. The names must match those of the parameters func expects.

resultsQueue is required and will be a string with the name of the redis queue to store any returned results in. If not provided, an error will be returned in the queue "missingResultsQueueErrors".

errorQueue will be a string with the name of the redis queue to store any errors thrown while running the job. If not specified, it will default to resultsQueue.

PieceMaker/RMinions documentation built on Oct. 21, 2019, 8:01 p.m.