batchtools: Tools for Computation on Batch Systems

As a successor of the packages 'BatchJobs' and 'BatchExperiments', this package provides a parallel implementation of the Map function for high performance computing systems managed by schedulers 'IBM Spectrum LSF' (<http://www-03.ibm.com/systems/spectrum-computing/products/lsf/>), 'OpenLava' (<http://www.openlava.org/>), 'Univa Grid Engine'/'Oracle Grid Engine' (<http://www.univa.com/>), 'Slurm' (<http://slurm.schedmd.com/>), 'TORQUE/PBS' (<http://www.adaptivecomputing.com/products/open-source/torque/>), or 'Docker Swarm' (<https://docs.docker.com/swarm/>). A multicore and socket mode allow the parallelization on a local machines, and multiple machines can be hooked up via SSH to create a makeshift cluster. Moreover, the package provides an abstraction mechanism to define large-scale computer experiments in a well-organized and reproducible way.

AuthorMichel Lang [cre, aut], Bernd Bischl [aut], Dirk Surmann [ctb]
Date of publication2017-02-20 17:06:34
MaintainerMichel Lang <michellang@gmail.com>
LicenseLGPL-3
Version0.9.2
https://github.com/mllg/batchtools

View on CRAN

Man pages

addAlgorithm: Define Algorithms for Experiments

addExperiments: Add Experiments to the Registry

addProblem: Define Problems for Experiments

batchExport: Export Objects to the Slaves

batchMap: Map Operation for Batch Systems

batchMapResults: Map Over Results to Create New Jobs

batchReduce: Reduce Operation for Batch Systems

batchtools-deprecated: Deprecated function in the batchtools package

batchtools-package: batchtools: Tools for Computation on Batch Systems

btlapply: Synchronous Apply Functions

cfBrewTemplate: Cluster Functions Helper to Write Job Description Files

cfHandleUnknownSubmitError: Cluster Functions Helper to Handle Unknown Errors

cfKillJob: Cluster Functions Helper to Kill Batch Jobs

cfReadBrewTemplate: Cluster Functions Helper to Parse a Brew Template

chunk: Chunk Jobs for Sequential Execution

chunkIds: Chunk Jobs for Sequential Execution

clearRegistry: Remove All Jobs

doJobCollection: Execute Jobs of a JobCollection

estimateRuntimes: Estimate Remaining Runtimes

execJob: Execute a Single Jobs

findJobs: Find and Filter Jobs

getDefaultRegistry: Get and Set the Default Registry

getErrorMessages: Retrieve Error Messages

getJobTable: Query Job Information

getStatus: Summarize the Computational Status

grepLogs: Grep Log Files for a Pattern

JobCollection: JobCollection Constructor

JobExperiment: Jobs and Experiments

JoinTables: Inner, Left, Right, Outer, Semi and Anti Join for Data Tables

killJobs: Kill Jobs

loadRegistry: Load a Registry from the File System

loadResult: Load the Result of a Single Job

makeClusterFunctions: ClusterFunctions Constructor

makeClusterFunctionsDocker: ClusterFunctions for Docker

makeClusterFunctionsInteractive: ClusterFunctions for Sequential Execution in the Running R...

makeClusterFunctionsLSF: ClusterFunctions for LSF Systems

makeClusterFunctionsMulticore: ClusterFunctions for Parallel Multicore Execution

makeClusterFunctionsOpenLava: ClusterFunctions for OpenLava

makeClusterFunctionsSGE: ClusterFunctions for SGE Systems

makeClusterFunctionsSlurm: ClusterFunctions for Slurm Systems

makeClusterFunctionsSocket: ClusterFunctions for Parallel Socket Execution

makeClusterFunctionsSSH: ClusterFunctions for Remote SSH Execution

makeClusterFunctionsTORQUE: ClusterFunctions for OpenPBS/TORQUE Systems

makeExperimentRegistry: ExperimentRegistry Constructor

makeRegistry: Registry Constructor

makeSubmitJobResult: Create a SubmitJobResult

reduceResults: Reduce Results

reduceResultsList: Apply Functions on Results

removeExperiments: Remove Experiments

removeRegistry: Remove a Registry from the File System

resetJobs: Reset the Computational State of Jobs

runHook: Trigger Evaluation of Custom Function

runOSCommand: Run OS Commands on Local or Remote Machines

saveRegistry: Store the Registy to the File System

showLog: Inspect Log Files

submitJobs: Submit Jobs to the Batch Systems

summarizeExperiments: Quick Summary over Experiments

sweepRegistry: Check Consistency and Remove Obsolete Information

syncRegistry: Syncronize the Registry

Tags: Add or Remove Job Tags

testJob: Run Jobs Interactively

waitForJobs: Wait for Termination of Jobs

Worker: Create a Linux-Worker

Functions

addAlgorithm Man page
addExperiments Man page
addJobTags Man page
addProblem Man page
ajoin Man page
Algorithm Man page
batchExport Man page
batchMap Man page
batchMapResults Man page
batchReduce Man page
batchtools Man page
batchtools-deprecated Man page
batchtools-package Man page
binpack Man page
btlapply Man page
btmapply Man page
cfBrewTemplate Man page
cfHandleUnknownSubmitError Man page
cfKillJob Man page
cfReadBrewTemplate Man page
chunk Man page
chunkIds Man page
clearRegistry Man page
ClusterFunctions Man page
doJobCollection Man page
estimateRuntimes Man page
execJob Man page
Experiment Man page
ExperimentRegistry Man page
findDone Man page
findErrors Man page
findExperiments Man page
findExpired Man page
findJobs Man page
findNotDone Man page
findNotStarted Man page
findNotSubmitted Man page
findOnSystem Man page
findQueued Man page
findRunning Man page
findStarted Man page
findSubmitted Man page
findTagged Man page
getAlgorithmIds Man page
getDefaultRegistry Man page
getErrorMessages Man page
getJobPars Man page
getJobResources Man page
getJobStatus Man page
getJobTable Man page
getJobTags Man page
getLog Man page
getProblemIds Man page
getStatus Man page
getUsedJobTags Man page
grepLogs Man page
Hook Man page
Hooks Man page
ijoin Man page
Job Man page
JobCollection Man page
JoinTables Man page
killJobs Man page
ljoin Man page
loadRegistry Man page
loadResult Man page
lpt Man page
makeClusterFunctions Man page
makeClusterFunctionsDocker Man page
makeClusterFunctionsInteractive Man page
makeClusterFunctionsLSF Man page
makeClusterFunctionsMulticore Man page
makeClusterFunctionsOpenLava Man page
makeClusterFunctionsSGE Man page
makeClusterFunctionsSlurm Man page
makeClusterFunctionsSocket Man page
makeClusterFunctionsSSH Man page
makeClusterFunctionsTORQUE Man page
makeExperimentRegistry Man page
makeJob Man page
makeJobCollection Man page
makeRegistry Man page
makeSubmitJobResult Man page
ojoin Man page
print.RuntimeEstimate Man page
Problem Man page
reduceResults Man page
reduceResultsDataTable Man page
reduceResultsList Man page
Registry Man page
removeAlgorithms Man page
removeExperiments Man page
removeJobTags Man page
removeProblems Man page
removeRegistry Man page
resetJobs Man page
rjoin Man page
runHook Man page
runOSCommand Man page
saveRegistry Man page
setDefaultRegistry Man page
showLog Man page
sjoin Man page
SubmitJobResult Man page
submitJobs Man page
summarizeExperiments Man page
sweepRegistry Man page
syncRegistry Man page
Tags Man page
testJob Man page
ujoin Man page
waitForJobs Man page
Worker Man page

Files

batchtools
batchtools/inst
batchtools/inst/CITATION
batchtools/inst/bin
batchtools/inst/bin/linux-helper
batchtools/inst/templates
batchtools/inst/templates/torque_lido.tmpl
batchtools/inst/templates/slurm_simple.tmpl
batchtools/inst/templates/lsf_simple.tmpl
batchtools/inst/templates/openlava_simple.tmpl
batchtools/inst/templates/sge_simple.tmpl
batchtools/inst/templates/slurm_dortmund.tmpl
batchtools/inst/templates/testJob.tmpl
batchtools/inst/doc
batchtools/inst/doc/v20_ErrorHandling.Rmd
batchtools/inst/doc/v01_Migration.R
batchtools/inst/doc/v11_ExampleExperiment.html
batchtools/inst/doc/v00_Setup.Rmd
batchtools/inst/doc/v11_ExampleExperiment.Rmd
batchtools/inst/doc/v11_ExampleExperiment.R
batchtools/inst/doc/v20_ErrorHandling.html
batchtools/inst/doc/v10_ExamplePiSim.Rmd
batchtools/inst/doc/v01_Migration.Rmd
batchtools/inst/doc/v00_Setup.html
batchtools/inst/doc/v20_ErrorHandling.R
batchtools/inst/doc/v01_Migration.html
batchtools/inst/doc/v10_ExamplePiSim.html
batchtools/inst/doc/v10_ExamplePiSim.R
batchtools/inst/doc/v00_Setup.R
batchtools/tests
batchtools/tests/testthat.R
batchtools/tests/testthat
batchtools/tests/testthat/test_manual.R
batchtools/tests/testthat/test_sweepRegistry.R
batchtools/tests/testthat/test_seed.R
batchtools/tests/testthat/test_ClusterFunctionsSSH.R
batchtools/tests/testthat/test_export.R
batchtools/tests/testthat/test_estimateRuntimes.R
batchtools/tests/testthat/test_doJobCollection.R
batchtools/tests/testthat/test_JobCollection.R
batchtools/tests/testthat/test_summarizeExperiments.R
batchtools/tests/testthat/test_runOSCommand.R
batchtools/tests/testthat/test_testJob.R
batchtools/tests/testthat/test_Results.R
batchtools/tests/testthat/test_ClusterFunctionsSocket.R
batchtools/tests/testthat/test_ExperimentRegistry.R
batchtools/tests/testthat/test_submitJobs.R
batchtools/tests/testthat/test_resetJobs.R
batchtools/tests/testthat/test_addProblem.R
batchtools/tests/testthat/test_waitForJobs.R
batchtools/tests/testthat/test_addExperiments.R
batchtools/tests/testthat/test_btlapply.R
batchtools/tests/testthat/test_Registry.R
batchtools/tests/testthat/test_getJobTable.R
batchtools/tests/testthat/test_chunkIds.R
batchtools/tests/testthat/test_count.R
batchtools/tests/testthat/test_killJobs.R
batchtools/tests/testthat/test_mergeRegistries.R
batchtools/tests/testthat/test_findConfFile.R
batchtools/tests/testthat/test_addAlgorithm.R
batchtools/tests/testthat/test_ClusterFunctionsMulticore.R
batchtools/tests/testthat/test_getStatus.R
batchtools/tests/testthat/test_batchMap.R
batchtools/tests/testthat/test_findJobs.R
batchtools/tests/testthat/test_getErrorMessages.R
batchtools/tests/testthat/test_batchReduce.R
batchtools/tests/testthat/test_chunk.R
batchtools/tests/testthat/test_Job.R
batchtools/tests/testthat/test_removeExperiments.R
batchtools/tests/testthat/helper.R
batchtools/tests/testthat/test_removeRegistry.R
batchtools/tests/testthat/test_convertIds.R
batchtools/tests/testthat/test_memory.R
batchtools/tests/testthat/test_ClusterFunctions.R
batchtools/tests/testthat/test_hooks.R
batchtools/tests/testthat/test_tags.R
batchtools/tests/testthat/test_parallelMap.R
batchtools/tests/testthat/test_joins.R
batchtools/tests/testthat/test_showLog.R
batchtools/tests/testthat/test_grepLogs.R
batchtools/src
batchtools/src/count_not_missing.c
batchtools/src/fill_gaps.c
batchtools/src/lpt.c
batchtools/src/init.c
batchtools/src/binpack.c
batchtools/NAMESPACE
batchtools/NEWS.md
batchtools/R
batchtools/R/clusterFunctionsSlurm.R batchtools/R/clusterFunctionsInteractive.R batchtools/R/clusterFunctions.R batchtools/R/btlapply.R batchtools/R/Registry.R batchtools/R/reduceResults.R batchtools/R/killJobs.R batchtools/R/sleep.R batchtools/R/Logs.R batchtools/R/helpers.R batchtools/R/JobTables.R batchtools/R/clusterFunctionsOpenLava.R batchtools/R/Worker.R batchtools/R/Job.R batchtools/R/Algorithm.R batchtools/R/getStatus.R batchtools/R/estimateRuntimes.R batchtools/R/saveRegistry.R batchtools/R/filenames.R batchtools/R/Problem.R batchtools/R/getErrorMessages.R batchtools/R/chunkIds.R batchtools/R/execJob.R batchtools/R/Tags.R batchtools/R/Joins.R batchtools/R/runOSCommand.R batchtools/R/submitJobs.R batchtools/R/clusterFunctionsMulticore.R batchtools/R/syncRegistry.R batchtools/R/ExperimentRegistry.R batchtools/R/Export.R batchtools/R/clearRegistry.R batchtools/R/clusterFunctionsSocket.R batchtools/R/getDefaultRegistry.R batchtools/R/clusterFunctionsTORQUE.R batchtools/R/JobCollection.R batchtools/R/config.R batchtools/R/updateRegisty.R batchtools/R/clusterFunctionsSGE.R batchtools/R/loadResult.R batchtools/R/testJob.R batchtools/R/waitForJobs.R batchtools/R/loadRegistry.R batchtools/R/clusterFunctionsDocker.R batchtools/R/clusterFunctionsSSH.R batchtools/R/batchReduce.R batchtools/R/summarizeExperiments.R batchtools/R/clusterFunctionsLSF.R batchtools/R/addExperiments.R batchtools/R/removeExperiments.R batchtools/R/removeRegistry.R batchtools/R/waitForFiles.R batchtools/R/resetJobs.R batchtools/R/sweepRegistry.R batchtools/R/batchMapResults.R batchtools/R/findJobs.R batchtools/R/doJobCollection.R batchtools/R/ids.R batchtools/R/batchMap.R batchtools/R/zzz.R batchtools/R/Hooks.R batchtools/R/mergeRegistries.R
batchtools/vignettes
batchtools/vignettes/v20_ErrorHandling.Rmd
batchtools/vignettes/v00_Setup.Rmd
batchtools/vignettes/v11_ExampleExperiment.Rmd
batchtools/vignettes/tikz_prob_algo_simple.png
batchtools/vignettes/v10_ExamplePiSim.Rmd
batchtools/vignettes/v01_Migration.Rmd
batchtools/vignettes/tikz_prob_algo_simple.tex
batchtools/README.md
batchtools/MD5
batchtools/build
batchtools/build/vignette.rds
batchtools/DESCRIPTION
batchtools/man
batchtools/man/reduceResultsList.Rd batchtools/man/makeClusterFunctionsLSF.Rd batchtools/man/getJobTable.Rd batchtools/man/getDefaultRegistry.Rd batchtools/man/Tags.Rd batchtools/man/addAlgorithm.Rd batchtools/man/makeClusterFunctionsMulticore.Rd batchtools/man/killJobs.Rd batchtools/man/makeRegistry.Rd batchtools/man/getErrorMessages.Rd batchtools/man/batchtools-deprecated.Rd batchtools/man/makeClusterFunctionsSGE.Rd batchtools/man/clearRegistry.Rd batchtools/man/batchExport.Rd batchtools/man/btlapply.Rd batchtools/man/batchMapResults.Rd batchtools/man/cfHandleUnknownSubmitError.Rd batchtools/man/addExperiments.Rd batchtools/man/loadResult.Rd batchtools/man/batchMap.Rd batchtools/man/runHook.Rd batchtools/man/waitForJobs.Rd batchtools/man/execJob.Rd batchtools/man/makeSubmitJobResult.Rd batchtools/man/batchtools-package.Rd batchtools/man/Worker.Rd batchtools/man/saveRegistry.Rd batchtools/man/summarizeExperiments.Rd batchtools/man/loadRegistry.Rd batchtools/man/reduceResults.Rd batchtools/man/JobCollection.Rd batchtools/man/getStatus.Rd batchtools/man/addProblem.Rd batchtools/man/makeExperimentRegistry.Rd batchtools/man/grepLogs.Rd batchtools/man/makeClusterFunctions.Rd batchtools/man/submitJobs.Rd batchtools/man/JobExperiment.Rd batchtools/man/makeClusterFunctionsInteractive.Rd batchtools/man/estimateRuntimes.Rd batchtools/man/makeClusterFunctionsSocket.Rd batchtools/man/removeExperiments.Rd batchtools/man/makeClusterFunctionsSlurm.Rd batchtools/man/makeClusterFunctionsDocker.Rd batchtools/man/chunk.Rd batchtools/man/findJobs.Rd batchtools/man/testJob.Rd batchtools/man/syncRegistry.Rd batchtools/man/makeClusterFunctionsOpenLava.Rd batchtools/man/showLog.Rd batchtools/man/chunkIds.Rd batchtools/man/makeClusterFunctionsTORQUE.Rd batchtools/man/cfKillJob.Rd batchtools/man/doJobCollection.Rd batchtools/man/batchReduce.Rd batchtools/man/runOSCommand.Rd batchtools/man/sweepRegistry.Rd batchtools/man/makeClusterFunctionsSSH.Rd batchtools/man/resetJobs.Rd batchtools/man/cfBrewTemplate.Rd batchtools/man/JoinTables.Rd batchtools/man/cfReadBrewTemplate.Rd batchtools/man/removeRegistry.Rd

Questions? Problems? Suggestions? or email at ian@mutexlabs.com.

Please suggest features or report bugs with the GitHub issue tracker.

All documentation is copyright its authors; we didn't write any of that.