mantaJob.launch: The interface from which compute Jobs are launched.

Description Usage Arguments Details Value See Also Examples

View source: R/mantaJob.launch.R

Description

Submits R format Manta Job specification, runs Job, sends inputs if specified, closes inputs, polls Job status, returns Job status JSON.

Usage

1
2
mantaJob.launch(inputs, job, batchsize = 500, watch = TRUE, sleep = 15,
  watchtimeout = 600, silent = FALSE, verbose = FALSE)

Arguments

inputs

vector of character optional. List of inputs as a vector of character, each containing valid paths to Manta objects as the intended job input files. You may use output from mantaFind or mantaLs.paths here. If you have no inputs, your initial Job task must be a mantaReduce step.

job

required. The R job structure as created with mantaJob.setup and Map and Reduce job tasks as defined therein by one or more mantaMap and/or mantaReduce steps. More information and parameters are explained in the help for these three functions.

batchsize

numeric. Maximum number of input object paths to upload in one batch to the running job. This function sends inputs in batches until they are all sent. Default is 500.

watch

logical. When TRUE calls mantaJob.done in polling mode, after job is initiated, sleeping for for sleep seconds up to the duration of the watchtimeout value in seconds. This causes the function to wait until the job is done to return, or until timed out. Timeout does not imply job success or failure.

sleep

integer. Number of seconds to wait between status requests in polling mode when watch is TRUE. Default is 15 seconds.

watchtimeout

integer. Number of seconds after which polling ends. Passed to mantaJob.done when watch is set to TRUE. Default is 10 minutes (600 seconds).. If watchtimeout is exceeded, it means the job is still running or queued on Manta. mantaJob.done(poll = TRUE) or mantaJob.status can be called afterward for more monitoring.

silent

logical. Supress console messages, does not affect verbose setting.

verbose

logical optional. Passed to RCURL to reveal HTTP communication.

Details

Job is created by mantaJob.setup and tasks as defined therein by mantaMap, and/or mantaReduce functions. Note that Manta tasks are UNIX shell commands, not native R commands.

Value

Returns a Manta status JSON structure. The Manta Job identifier is the "id": field - like this "70c30bab-873b-66da-ebc8-ced12bd35ac4". This value is the jobid parameter to be used used by other mantaJob functions for inputs, status, errors and outputs as Job lookup key. This key can also be used by Node.js Manta command-line mjob commands.

See Also

Other mantaJobs: mantaJob.cancel; mantaJob.done; mantaJob.errors.stderr; mantaJob.errors; mantaJob.failures; mantaJob.inputs; mantaJob.outputs.cat; mantaJob.outputs; mantaJob.setup; mantaJob.status; mantaJobs.running; mantaJobs.tail; mantaJobs; mantaMap; mantaReduce

Examples

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
## Not run: 
## Example - Map/Reduce Unix Word Count

## Part 1.
## Job to download all of Shakespeare's plays to your account
plays <-
c("1kinghenryiv.txt", "1kinghenryvi.txt", "2kinghenryiv.txt",
"2kinghenryvi.txt", "3kinghenryvi.txt", "allswellthatendswell.txt",
"antonyandcleopatra.txt", "asyoulikeit.txt", "comedyoferrors.txt",
"coriolanus.txt", "cymbeline.txt", "hamlet.txt", "juliuscaesar.txt",
"kinghenryv.txt", "kinghenryviii.txt", "kingjohn.txt", "kinglear.txt",
"kingrichardii.txt", "kingrichardiii.txt", "loverscomplaint.txt",
"loveslabourslost.txt", "macbeth.txt", "measureforemeasure.txt",
"merchantofvenice.txt", "merrywivesofwindsor.txt",
"midsummersnightsdream.txt",
"muchadoaboutnothing.txt", "othello.txt", "periclesprinceoftyre.txt",
"rapeoflucrece.txt", "romeoandjuliet.txt", "sonnets.txt",
"tamingoftheshrew.txt",
"tempest.txt", "timonofathens.txt", "titusandronicus.txt",
"troilusandcressida.txt",
"twelfthnight.txt", "twogentlemenofverona.txt", "various.txt",
"venusandadonis.txt", "winterstale.txt")

file <- file("plays_list.txt", "wb")
# Important: This forces Windows to use /n instead of /r/n on write()
write(plays, file)
close(file)
rm(file)

mantaSetwd.stor()
mantaPut("plays_list.txt")

inputlist <- mantaLs.paths(grepfor = "plays_list.txt")

mantaMkdir("shakespeare")
mantaSetwd("shakespeare")
fileslocation <-
"https://us-east.manta.joyent.com/cwvhogue/public/shakespeare/"
destination <- mantaGetwd()
mantaJob.setup("Get Plays",
 mantaMap(paste("xargs -I {} sh -c 'curl -ksL ",
                 fileslocation,
                "{} | mput ",
                destination,
                "/{}'",
                sep=""))) -> moveplays

## Launch the first job to download the plays:
mantaJob.launch(inputlist, moveplays)

## See if they arrived.
mantaLs()
mantaLs.n()
mantaLs.du()

## Copy all the plays to your local drive?
# mantaGet(mantaFind())

## Speedread all of Shakespeare?
# mantaCat(mantaFind())

## Part 2.
## Map/Reduce Count all the words with wc and awk

inputs <- mantaFind()
job <- mantaJob.setup(
            name = "Word Count",
            mantaMap("wc"),
            mantaReduce("awk '{ l += $1; w += $2; c += $3 } END { print l, w, c }'")
        )

mantaJob.launch(inputs, job) -> status
## Getting Job Results:
## These functions find the last Job run if no jobid provided.
# mantaJob.status()  ## check to see if job is complete, as JSON information
# mantaJob.done()    ## returns logical job done (TRUE/FALSE)
# mantaJob.inputs()  ## returns list of inputs
mantaJob.outputs() ## retrieve list of paths to Manta output objects
# mantaJob.errors()  ## retrieve JSON formatted job error information
mantaJob.outputs.cat()   ## Print job output (text files) to console
# mantaJob.errors.stderr() ## Print any stderr messages to console

## End(Not run)

joyent/mantaRSDK documentation built on May 19, 2019, 10:43 p.m.