setGetTask: setGetTask

In cloudcell/doRedis111stable: Foreach parallel adapter for the rredis package

Description

Define a function used by workers to pull tasks.

Usage

setGetTask(fn = default_getTask)

Arguments

fn	A task pulling function, see details below.

Details

The doRedis package organizes work into a collections of tasks called a job. One job may contain several uniquely identified task collections, and jobs themselves are uniquely identified. The task collections are labeled by the taskLabel function.

A doRedis task is specifically a list of two elements: task_id and args, that specify the unique ID of the task collection, and the foreach loop expression arguments, respectively.

As of version 1.1.0 of the doRedis package, task collections are placed in a Redis hash table identified by the job ID. The getTask function is used by the R worker processes to pull tasks from this hash table. The getTask function must take at least two arguments, queue and job_id that specify the job queue and job ID, respectively. The function should work with Redis to obtain and return a task collection, or return NULL if no tasks are available.

The getTask function almost always defines a short Lua scipt run on the Redis server.

The default getTask function removes tasks in the order that they appear in the hash table. Custom getTask functions can be defined often in association with custom taskLabel functions. The custom functions can instruct Redis to perferentially dole out tasks based on network distance to data, for example.

Value

Nothing is returned.

Note

All doRedis functions require network access to a Redis server (not included with the doRedis package).

Author(s)

B. W. Lewis <[e-mail]>

Examples

## Not run: 
require('doRedis')

# The default getTask function defines a Redis Lua script that returns the first
# listed task or NULL:sk <-  function(queue, job_id, ...)

getTask <- function(queue, job_id, ...)
{
  key <- sprintf("
  redisEval("local x=redis.call('hkeys',KEYS[1])[1];
             if x==nil then return nil end;
             local ans=redis.call('hget',KEYS[1],x);
             redis.call('hdel',KEYS[1],x);i
             return ans",key)
}
setGetTask(getTask)

## End(Not run)

cloudcell/doRedis111stable documentation built on May 13, 2019, 8:02 p.m.