batchtools: Tools for Computation on Batch Systems

As a successor of the packages 'BatchJobs' and 'BatchExperiments', this package provides a parallel implementation of the Map function for high performance computing systems managed by schedulers 'IBM Spectrum LSF' (<http://www-03.ibm.com/systems/spectrum-computing/products/lsf/>), 'OpenLava' (<http://www.openlava.org/>), 'Univia Grid Engine'/'Oracle Grid Engine' (<http://www.univa.com/>), 'Slurm' (<http://slurm.schedmd.com/>), 'Torque/PBS' (<http://www.adaptivecomputing.com/products/open-source/torque/>), or 'Docker Swarm' (<https://docs.docker.com/swarm/>). A multicore and socket mode allow the parallelization on a local machines, and multiple machines can be hooked up via SSH to create a makeshift cluster. Moreover, the package provides an abstraction mechanism to define large-scale computer experiments in a well-organized and reproducible way.

AuthorMichel Lang [cre, aut], Bernd Bischl [aut], Dirk Surmann [ctb]
Date of publication2016-11-08 00:59:52
MaintainerMichel Lang <michellang@gmail.com>
LicenseLGPL-3
Version0.9.0
https://github.com/mllg/batchtools

View on CRAN

Man pages

addExperiments: Add Experiments to the Registry

batchExport: Export Objects to the Slaves

batchMap: Map Operation for Batch Systems

batchMapResults: Map Over Results to Create New Jobs

batchtools-package: Tools for Computation on Batch Systems

btlapply: Synchronous Apply Functions

cfBrewTemplate: Cluster Functions Helper to Write Job Description Files

cfHandleUnknownSubmitError: Cluster Functions Helper to Handle Unknown Errors

cfKillJob: Cluster Functions Helper to Kill Batch Jobs

cfReadBrewTemplate: Cluster Functions Helper to Parse a Brew Template

chunkIds: Chunk Jobs for Sequential Execution

doJobCollection: Execute Jobs of a JobCollection

execJob: Execute a Single Jobs

ExperimentRegistry: ExperimentRegistry Constructor

findJobs: Find and Filter Jobs

getErrorMessages: Retrieve Error Messages

getJobTable: Query Job Information

getStatus: Summarize the Computational Status

grepLogs: Grep Log Files for a Pattern

JobCollection: JobCollection Constructor

JobExperiment: Jobs and Experiments

JoinTables: Inner, Left, Right, Outer, Semi and Anti Join for Data Tables

killJobs: Kill Jobs

loadResult: Load the Result of a Single Job

makeClusterFunctions: ClusterFunctions Constructor

makeClusterFunctionsDocker: ClusterFunctions for Docker

makeClusterFunctionsInteractive: ClusterFunctions for Sequential Execution in the Running R...

makeClusterFunctionsLSF: ClusterFunctions for LSF Systems

makeClusterFunctionsMulticore: ClusterFunctions for Parallel Multicore Execution

makeClusterFunctionsOpenLava: ClusterFunctions for OpenLava

makeClusterFunctionsSGE: ClusterFunctions for SGE Systems

makeClusterFunctionsSlurm: ClusterFunctions for Slurm Systems

makeClusterFunctionsSocket: ClusterFunctions for Parallel Socket Execution

makeClusterFunctionsSSH: ClusterFunctions for Remote SSH Execution

makeClusterFunctionsTorque: ClusterFunctions for OpenPBS/Torque Systems

makeSubmitJobResult: Create a SubmitJobResult

ProblemAlgorithm: Define Problems and Algorithms for Experiments

reduceResults: Reduce Results

reduceResultsList: Apply Functions on Results

Registry: Registry Constructor

removeExperiments: Remove Experiments

resetJobs: Reset the Computational State of Jobs

runHook: Trigger Evaluation of Custom Function

runOSCommand: Run OS Commands on Local or Remote Machines

showLog: Inspect Log Files

submitJobs: Submit Jobs to the Batch Systems

summarizeExperiments: Quick Summary over Experiments

Tags: Add or Remove Job Tags

testJob: Run Jobs Interactively

waitForJobs: Wait for Termination of Jobs

Worker: Create a Linux-Worker

Files in this package

batchtools
batchtools/inst
batchtools/inst/CITATION
batchtools/inst/bin
batchtools/inst/bin/linux-helper
batchtools/inst/templates
batchtools/inst/templates/torque_lido.tmpl
batchtools/inst/templates/slurm_simple.tmpl
batchtools/inst/templates/lsf_simple.tmpl
batchtools/inst/templates/openlava_simple.tmpl
batchtools/inst/templates/sge_simple.tmpl
batchtools/inst/templates/slurm_dortmund.tmpl
batchtools/inst/templates/testJob.tmpl
batchtools/inst/doc
batchtools/inst/doc/ExamplePiSim.Rmd
batchtools/inst/doc/Setup.R
batchtools/inst/doc/ExampleExperiment.html
batchtools/inst/doc/ErrorHandling.html
batchtools/inst/doc/ExampleExperiment.Rmd
batchtools/inst/doc/Setup.html
batchtools/inst/doc/Setup.Rmd
batchtools/inst/doc/ExamplePiSim.R
batchtools/inst/doc/ErrorHandling.Rmd
batchtools/inst/doc/ErrorHandling.R
batchtools/inst/doc/ExamplePiSim.html
batchtools/inst/doc/ExampleExperiment.R
batchtools/tests
batchtools/tests/testthat.R
batchtools/tests/testthat
batchtools/tests/testthat/test_manual.R
batchtools/tests/testthat/test_seed.R
batchtools/tests/testthat/test_ClusterFunctionsSSH.R
batchtools/tests/testthat/test_export.R
batchtools/tests/testthat/test_doJobCollection.R
batchtools/tests/testthat/test_JobCollection.R
batchtools/tests/testthat/test_summarizeExperiments.R
batchtools/tests/testthat/test_runOSCommand.R
batchtools/tests/testthat/test_testJob.R
batchtools/tests/testthat/test_Results.R
batchtools/tests/testthat/test_ClusterFunctionsSocket.R
batchtools/tests/testthat/test_ExperimentRegistry.R
batchtools/tests/testthat/test_submitJobs.R
batchtools/tests/testthat/test_resetJobs.R
batchtools/tests/testthat/test_addProblem.R
batchtools/tests/testthat/test_waitForJobs.R
batchtools/tests/testthat/test_addExperiments.R
batchtools/tests/testthat/test_btlapply.R
batchtools/tests/testthat/test_Registry.R
batchtools/tests/testthat/test_getJobTable.R
batchtools/tests/testthat/test_chunkIds.R
batchtools/tests/testthat/test_count.R
batchtools/tests/testthat/test_killJobs.R
batchtools/tests/testthat/test_findConfFile.R
batchtools/tests/testthat/test_addAlgorithm.R
batchtools/tests/testthat/test_ClusterFunctionsMulticore.R
batchtools/tests/testthat/test_getStatus.R
batchtools/tests/testthat/test_batchMap.R
batchtools/tests/testthat/test_findJobs.R
batchtools/tests/testthat/test_getErrorMessages.R
batchtools/tests/testthat/test_Job.R
batchtools/tests/testthat/test_removeExperiments.R
batchtools/tests/testthat/helper.R
batchtools/tests/testthat/test_convertIds.R
batchtools/tests/testthat/test_memory.R
batchtools/tests/testthat/test_ClusterFunctions.R
batchtools/tests/testthat/test_hooks.R
batchtools/tests/testthat/test_tags.R
batchtools/tests/testthat/test_parallelMap.R
batchtools/tests/testthat/test_joins.R
batchtools/tests/testthat/test_showLog.R
batchtools/tests/testthat/test_grepLogs.R
batchtools/src
batchtools/src/count_not_missing.c
batchtools/src/fill_gaps.c
batchtools/NAMESPACE
batchtools/NEWS.md
batchtools/R
batchtools/R/clusterFunctionsSlurm.R batchtools/R/clusterFunctionsInteractive.R batchtools/R/clusterFunctions.R batchtools/R/btlapply.R batchtools/R/Registry.R batchtools/R/reduceResults.R batchtools/R/killJobs.R batchtools/R/Logs.R batchtools/R/helpers.R batchtools/R/JobTables.R batchtools/R/clusterFunctionsOpenLava.R batchtools/R/Worker.R batchtools/R/Job.R batchtools/R/ProblemAlgorithm.R batchtools/R/getStatus.R batchtools/R/getErrorMessages.R batchtools/R/chunkIds.R batchtools/R/execJob.R batchtools/R/Tags.R batchtools/R/Joins.R batchtools/R/submitJobs.R batchtools/R/clusterFunctionsMulticore.R batchtools/R/ExperimentRegistry.R batchtools/R/Export.R batchtools/R/clusterFunctionsSocket.R batchtools/R/JobCollection.R batchtools/R/updateRegisty.R batchtools/R/clusterFunctionsSGE.R batchtools/R/loadResult.R batchtools/R/testJob.R batchtools/R/waitForJobs.R batchtools/R/clusterFunctionsDocker.R batchtools/R/clusterFunctionsSSH.R batchtools/R/summarizeExperiments.R batchtools/R/clusterFunctionsLSF.R batchtools/R/addExperiments.R batchtools/R/removeExperiments.R batchtools/R/clusterFunctionsTorque.R batchtools/R/resetJobs.R batchtools/R/batchMapResults.R batchtools/R/findJobs.R batchtools/R/doJobCollection.R batchtools/R/ids.R batchtools/R/batchMap.R batchtools/R/zzz.R batchtools/R/Hooks.R
batchtools/vignettes
batchtools/vignettes/ExamplePiSim.Rmd
batchtools/vignettes/ExampleExperiment.Rmd
batchtools/vignettes/Setup.Rmd
batchtools/vignettes/ErrorHandling.Rmd
batchtools/README.md
batchtools/MD5
batchtools/build
batchtools/build/vignette.rds
batchtools/DESCRIPTION
batchtools/man
batchtools/man/reduceResultsList.Rd batchtools/man/makeClusterFunctionsLSF.Rd batchtools/man/getJobTable.Rd batchtools/man/ProblemAlgorithm.Rd batchtools/man/Tags.Rd batchtools/man/Registry.Rd batchtools/man/makeClusterFunctionsMulticore.Rd batchtools/man/killJobs.Rd batchtools/man/getErrorMessages.Rd batchtools/man/makeClusterFunctionsSGE.Rd batchtools/man/batchExport.Rd batchtools/man/btlapply.Rd batchtools/man/batchMapResults.Rd batchtools/man/cfHandleUnknownSubmitError.Rd batchtools/man/addExperiments.Rd batchtools/man/loadResult.Rd batchtools/man/batchMap.Rd batchtools/man/runHook.Rd batchtools/man/waitForJobs.Rd batchtools/man/execJob.Rd batchtools/man/makeSubmitJobResult.Rd batchtools/man/batchtools-package.Rd batchtools/man/Worker.Rd batchtools/man/summarizeExperiments.Rd batchtools/man/reduceResults.Rd batchtools/man/JobCollection.Rd batchtools/man/getStatus.Rd batchtools/man/grepLogs.Rd batchtools/man/makeClusterFunctions.Rd batchtools/man/submitJobs.Rd batchtools/man/JobExperiment.Rd batchtools/man/makeClusterFunctionsInteractive.Rd batchtools/man/makeClusterFunctionsSocket.Rd batchtools/man/removeExperiments.Rd batchtools/man/makeClusterFunctionsSlurm.Rd batchtools/man/makeClusterFunctionsDocker.Rd batchtools/man/findJobs.Rd batchtools/man/testJob.Rd batchtools/man/makeClusterFunctionsOpenLava.Rd batchtools/man/showLog.Rd batchtools/man/chunkIds.Rd batchtools/man/cfKillJob.Rd batchtools/man/doJobCollection.Rd batchtools/man/makeClusterFunctionsTorque.Rd batchtools/man/runOSCommand.Rd batchtools/man/ExperimentRegistry.Rd batchtools/man/makeClusterFunctionsSSH.Rd batchtools/man/resetJobs.Rd batchtools/man/cfBrewTemplate.Rd batchtools/man/JoinTables.Rd batchtools/man/cfReadBrewTemplate.Rd

Questions? Problems? Suggestions? or email at ian@mutexlabs.com.

All documentation is copyright its authors; we didn't write any of that.