BatchJobs: Batch Computing with R

Provides Map, Reduce and Filter variants to generate jobs on batch computing systems like PBS/Torque, LSF, SLURM and Sun Grid Engine. Multicore and SSH systems are also supported. For further details see the project web page.

AuthorBernd Bischl <bernd_bischl@gmx.net>, Michel Lang <michellang@gmail.com>, Henrik Bengtsson <henrikb@braju.com>
Date of publication2015-03-18 18:01:21
MaintainerBernd Bischl <bernd_bischl@gmx.net>
LicenseBSD_2_clause + file LICENSE
Version1.6
https://github.com/tudo-r/BatchJobs

View on CRAN

Man pages

addRegistryPackages: Add packages to registry.

addRegistrySourceDirs: Add source dirs to registry.

addRegistrySourceFiles: Add source files to registry.

applyJobFunction: applyJobFunction ONLY FOR INTERNAL USAGE.

batchExpandGrid: Map function over all combinations.

batchExport: Export R object to be available on the slaves.

BatchJobs: The BatchJobs package

batchMap: Maps a function over lists or vectors, adding jobs to a...

batchMapQuick: Combination of makeRegistry, batchMap and submitJobs.

batchMapResults: Maps a function over the results of a registry by using...

batchReduce: Reduces via a binary function over a list adding jobs to a...

batchReduceResults: Reduces results via a binary function and adds jobs for this...

batchUnexport: Unload exported R objects.

callFunctionOnSSHWorkers: Call an arbitrary function on specified SSH workers.

cfBrewTemplate: Cluster functions helper: Brew your template into a job...

cfHandleUnknownSubmitError: Cluster functions helper: Handle an unknown error during job...

cfKillBatchJob: Cluster functions helper: Kill a batch job via OS command

cfReadBrewTemplate: Cluster functions helper: Read in your brew template file.

configuration: BatchJobs configuration.

copyRequiredJobFiles: ONLY FOR INTERNAL USAGE.

dbCreateJobDefTable: ONLY FOR INTERNAL USAGE.

dbGetJobs: ONLY FOR INTERNAL USAGE.

debugMulticore: Helper function to debug multicore mode.

debugSSH: Helper function to debug SSH mode.

filterResults: Find all results where a specific condition is true.

findJobs: Finds ids of jobs that match a query.

findState: Find jobs depending on computional state.

getConfig: Returns a list of BatchJobs configuration settings

getErrorMessages: Get error messages of jobs.

getJob: Get job from registry by id.

getJobIds: Get ids of jobs in registry.

getJobInfo: Get computational information of jobs.

getJobNr: Get number of jobs in registry.

getJobParamDf: Returns parameters for all jobs as the rows of a data.frame.

getJobResources: Function to get the resources that were submitted for some...

getJobs: Get jobs from registry by id.

getLogFiles: Get log file paths for jobs.

getResources: Function to get job resources in job function.

getSSHWorkersInfo: Print and return R installation and other information for SSH...

grepLogs: Grep log files for a pattern.

installPackagesOnSSHWorkers: Install packages on SSH workers.

killJobs: Kill some jobs on the batch system.

loadConfig: Load a specific configuration file.

loadExports: Load exported R data objects.

loadRegistry: Load a previously saved registry.

loadResult: Loads a specific result file.

loadResults: Loads result files for id vector.

makeClusterFunctions: Create a ClusterFuntions object.

makeClusterFunctionsInteractive: Create cluster functions for sequential execution in same...

makeClusterFunctionsLocal: Create cluster functions for sequential execution on local...

makeClusterFunctionsLSF: Create cluster functions for LSF systems.

makeClusterFunctionsMulticore: Use multiple cores on local Linux machine to spawn parallel...

makeClusterFunctionsSGE: Create cluster functions for Sun Grid Engine systems.

makeClusterFunctionsSLURM: Create cluster functions for SLURM-based systems.

makeClusterFunctionsSSH: Create an SSH cluster to execute jobs.

makeClusterFunctionsTorque: Create cluster functions for torque-based systems.

makeJob: Creates a job description.

makeRegistry: Construct a registry object.

makeSSHWorker: Create SSH worker for SSH cluster functions.

makeSubmitJobResult: Create a SubmitJobResult object.

reduceResults: Reduce results from result directory.

removeRegistry: Remove a registry object.

removeRegistryPackages: Remove packages from registry.

removeRegistrySourceDirs: Remove packages from registry.

removeRegistrySourceFiles: Remove source files from registry.

resetJobs: Reset computational state of jobs.

sanitizePath: Sanitize a path

setConfig: Set and overwrite configuration settings

setJobFunction: Sets the job function for already existing jobs.

setJobNames: Set job names.

showClusterStatus: Show information about available computational resources on...

showLog: Display the contents of a log file.

showStatus: Retrieve or show status information about jobs.

sourceRegistryFiles: Source registry files

submitJobs: Submit jobs or chunks of jobs to batch system via cluster...

sweepRegistry: Sweep obsolete files from the file system.

testJob: Tests a job by running it with Rscript in a new process.

updateRegistry: ONLY FOR INTERNAL USAGE.

waitForJobs: Wait for termination of jobs on the batch system.

Functions

addRegistryPackages Man page
addRegistrySourceDirs Man page
addRegistrySourceFiles Man page
applyJobFunction Man page
batchExpandGrid Man page
batchExport Man page
BatchJobs Man page
BatchJobs-package Man page
.BatchJobs.R Man page
batchMap Man page
batchMapQuick Man page
batchMapResults Man page
batchReduce Man page
batchReduceResults Man page
batchUnexport Man page
callFunctionOnSSHWorkers Man page
cfBrewTemplate Man page
cfHandleUnknownSubmitError Man page
cfKillBatchJob Man page
cfReadBrewTemplate Man page
ClusterFunctions Man page
configuration Man page
copyRequiredJobFiles Man page
dbCreateJobDefTable Man page
dbGetJobs Man page
debugMulticore Man page
debugSSH Man page
filterResults Man page
findDisappeared Man page
findDone Man page
findErrors Man page
findExpired Man page
findJobs Man page
findMissingResults Man page
findNotDone Man page
findNotErrors Man page
findNotOnSystem Man page
findNotRunning Man page
findNotStarted Man page
findNotSubmitted Man page
findNotTerminated Man page
findOnSystem Man page
findRunning Man page
findStarted Man page
findSubmitted Man page
findTerminated Man page
getConfig Man page
getErrorMessages Man page
getJob Man page
getJobIds Man page
getJobInfo Man page
getJobNr Man page
getJobParamDf Man page
getJobResources Man page
getJobs Man page
getLogFiles Man page
getResources Man page
getSSHWorkersInfo Man page
getStatus Man page
grepLogs Man page
installPackagesOnSSHWorkers Man page
Job Man page
killJobs Man page
loadConfig Man page
loadExports Man page
loadRegistry Man page
loadResult Man page
loadResults Man page
makeClusterFunctions Man page
makeClusterFunctionsInteractive Man page
makeClusterFunctionsLocal Man page
makeClusterFunctionsLSF Man page
makeClusterFunctionsMulticore Man page
makeClusterFunctionsSGE Man page
makeClusterFunctionsSLURM Man page
makeClusterFunctionsSSH Man page
makeClusterFunctionsTorque Man page
makeJob Man page
makeRegistry Man page
makeSSHWorker Man page
makeSubmitJobResult Man page
reduceResults Man page
reduceResultsDataFrame Man page
reduceResultsList Man page
reduceResultsMatrix Man page
reduceResultsVector Man page
Registry Man page
removeRegistry Man page
removeRegistryPackages Man page
removeRegistrySourceDirs Man page
removeRegistrySourceFiles Man page
resetJobs Man page
sanitizePath Man page
setConfig Man page
setJobFunction Man page
setJobNames Man page
showClusterStatus Man page
showLog Man page
showStatus Man page
sourceRegistryFiles Man page
SSHWorker Man page
SubmitJobResult Man page
submitJobs Man page
sweepRegistry Man page
testJob Man page
updateRegistry Man page
waitForJobs Man page

Files

BatchJobs
BatchJobs/inst
BatchJobs/inst/CITATION
BatchJobs/inst/bin
BatchJobs/inst/bin/linux-helper
BatchJobs/inst/etc
BatchJobs/inst/etc/BatchJobs_global_config.R
BatchJobs/tests
BatchJobs/tests/run-all.R
BatchJobs/tests/testthat
BatchJobs/tests/testthat/test_sweepRegistry.R
BatchJobs/tests/testthat/test_export.R
BatchJobs/tests/testthat/helpers.R
BatchJobs/tests/testthat/test_reduceResults.R
BatchJobs/tests/testthat/test_getJobInfo.R
BatchJobs/tests/testthat/test_sanitizePath.R
BatchJobs/tests/testthat/test_testJob.R
BatchJobs/tests/testthat/test_filterResults.R
BatchJobs/tests/testthat/test_getJobParamDf.R
BatchJobs/tests/testthat/test_batchMapQuick.R
BatchJobs/tests/testthat/test_loadResults.R
BatchJobs/tests/testthat/test_submitJobs.R
BatchJobs/tests/testthat/test_resources.R
BatchJobs/tests/testthat/test_resetJobs.R
BatchJobs/tests/testthat/test_getLogFiles.R
BatchJobs/tests/testthat/test_waitForJobs.R
BatchJobs/tests/testthat/test_batchMap.R
BatchJobs/tests/testthat/test_setJobFunction.R
BatchJobs/tests/testthat/test_findJobs.R
BatchJobs/tests/testthat/test_showStatus.R
BatchJobs/tests/testthat/test_batchReduce.R
BatchJobs/tests/testthat/test_findStatus.R
BatchJobs/tests/testthat/test_batchMapResults.R
BatchJobs/tests/testthat/test_batchReduceResults.R
BatchJobs/tests/testthat/test_sourceRegistryFiles.R
BatchJobs/tests/testthat/test_zzz_cleanup.R
BatchJobs/tests/testthat/test_getJob.R
BatchJobs/tests/testthat/test_database.R
BatchJobs/tests/testthat/test_sqlquotes.R
BatchJobs/tests/testthat/test_batchExpandGrid.R
BatchJobs/tests/testthat/test_packages.R
BatchJobs/tests/testthat/test_makeRegistry.R
BatchJobs/tests/testthat/test_grepLogs.R
BatchJobs/tests/testthat/test_doJob.R
BatchJobs/NAMESPACE
BatchJobs/NEWS
BatchJobs/R
BatchJobs/R/getJobInfo.R BatchJobs/R/buffer.R BatchJobs/R/clusterFunctionsInteractive.R BatchJobs/R/clusterFunctions.R BatchJobs/R/Registry.R BatchJobs/R/filterResults.R BatchJobs/R/reduceResults.R BatchJobs/R/killJobs.R BatchJobs/R/database.R BatchJobs/R/writeRscripts.R BatchJobs/R/callFunctionOnSSHWorkers.R BatchJobs/R/getJobParamDf.R BatchJobs/R/helpers.R BatchJobs/R/Worker.R BatchJobs/R/Job.R BatchJobs/R/conf.R BatchJobs/R/debugMulticore.R BatchJobs/R/clusterFunctionsHelpers.R BatchJobs/R/showLog.R BatchJobs/R/debugSSH.R BatchJobs/R/batchApply.R BatchJobs/R/batchExpandGrid.R BatchJobs/R/filenames.R BatchJobs/R/Packages.R BatchJobs/R/getErrorMessages.R BatchJobs/R/loadResults.R BatchJobs/R/setJobNames.R BatchJobs/R/sanitizePath.R BatchJobs/R/findStatus.R BatchJobs/R/submitJobs.R BatchJobs/R/batchReduceResults.R BatchJobs/R/clusterFunctionsMulticore.R BatchJobs/R/Exports.R BatchJobs/R/syncRegistry.R BatchJobs/R/getLogFiles.R BatchJobs/R/sourceRegistryFiles.R BatchJobs/R/showStatus.R BatchJobs/R/sendMail.R BatchJobs/R/clusterFunctionsSLURM.R BatchJobs/R/installPackagesOnSSHWorkers.R BatchJobs/R/getJob.R BatchJobs/R/clusterFunctionsSGE.R BatchJobs/R/loadResult.R BatchJobs/R/doJob.R BatchJobs/R/testJob.R BatchJobs/R/waitForJobs.R BatchJobs/R/setJobFunction.R BatchJobs/R/clusterFunctionsSSH.R BatchJobs/R/checkIdValid.R BatchJobs/R/showClusterStatus.R BatchJobs/R/batchReduce.R BatchJobs/R/batchMapQuick.R BatchJobs/R/clusterFunctionsLSF.R BatchJobs/R/Resources.R BatchJobs/R/clusterFunctionsTorque.R BatchJobs/R/resetJobs.R BatchJobs/R/sweepRegistry.R BatchJobs/R/clusterFunctionsLocal.R BatchJobs/R/updateRegistry.R BatchJobs/R/batchMapResults.R BatchJobs/R/findJobs.R BatchJobs/R/getSSHWorkersInfo.R BatchJobs/R/grepLogs.R BatchJobs/R/batchMap.R BatchJobs/R/zzz.R BatchJobs/R/WorkerLinux.R BatchJobs/R/scheduleWorkerJobs.R BatchJobs/R/applyJobFunction.R BatchJobs/R/addJob.R
BatchJobs/MD5
BatchJobs/DESCRIPTION
BatchJobs/man
BatchJobs/man/makeClusterFunctionsLSF.Rd BatchJobs/man/loadResults.Rd BatchJobs/man/addRegistrySourceDirs.Rd BatchJobs/man/updateRegistry.Rd BatchJobs/man/dbCreateJobDefTable.Rd BatchJobs/man/getJobs.Rd BatchJobs/man/setJobNames.Rd BatchJobs/man/applyJobFunction.Rd BatchJobs/man/installPackagesOnSSHWorkers.Rd BatchJobs/man/makeClusterFunctionsMulticore.Rd BatchJobs/man/killJobs.Rd BatchJobs/man/makeRegistry.Rd BatchJobs/man/getErrorMessages.Rd BatchJobs/man/configuration.Rd BatchJobs/man/makeJob.Rd BatchJobs/man/makeClusterFunctionsSGE.Rd BatchJobs/man/batchExport.Rd BatchJobs/man/makeClusterFunctionsSLURM.Rd BatchJobs/man/batchMapResults.Rd BatchJobs/man/cfHandleUnknownSubmitError.Rd BatchJobs/man/loadResult.Rd BatchJobs/man/batchMap.Rd BatchJobs/man/showClusterStatus.Rd BatchJobs/man/sanitizePath.Rd BatchJobs/man/setConfig.Rd BatchJobs/man/waitForJobs.Rd BatchJobs/man/addRegistrySourceFiles.Rd BatchJobs/man/copyRequiredJobFiles.Rd BatchJobs/man/makeSubmitJobResult.Rd BatchJobs/man/showStatus.Rd BatchJobs/man/batchMapQuick.Rd BatchJobs/man/loadRegistry.Rd BatchJobs/man/reduceResults.Rd BatchJobs/man/batchReduceResults.Rd BatchJobs/man/filterResults.Rd BatchJobs/man/loadConfig.Rd BatchJobs/man/getJobInfo.Rd BatchJobs/man/grepLogs.Rd BatchJobs/man/makeClusterFunctions.Rd BatchJobs/man/submitJobs.Rd BatchJobs/man/makeClusterFunctionsInteractive.Rd BatchJobs/man/findState.Rd BatchJobs/man/cfKillBatchJob.Rd BatchJobs/man/findJobs.Rd BatchJobs/man/testJob.Rd BatchJobs/man/showLog.Rd BatchJobs/man/debugSSH.Rd BatchJobs/man/removeRegistrySourceDirs.Rd BatchJobs/man/getJobIds.Rd BatchJobs/man/getResources.Rd BatchJobs/man/batchReduce.Rd BatchJobs/man/makeClusterFunctionsTorque.Rd BatchJobs/man/addRegistryPackages.Rd BatchJobs/man/getLogFiles.Rd BatchJobs/man/debugMulticore.Rd BatchJobs/man/removeRegistryPackages.Rd BatchJobs/man/getJobParamDf.Rd BatchJobs/man/getJobResources.Rd BatchJobs/man/batchUnexport.Rd BatchJobs/man/getSSHWorkersInfo.Rd BatchJobs/man/sweepRegistry.Rd BatchJobs/man/batchExpandGrid.Rd BatchJobs/man/BatchJobs.Rd BatchJobs/man/getJobNr.Rd BatchJobs/man/makeClusterFunctionsSSH.Rd BatchJobs/man/getJob.Rd BatchJobs/man/resetJobs.Rd BatchJobs/man/cfBrewTemplate.Rd BatchJobs/man/sourceRegistryFiles.Rd BatchJobs/man/loadExports.Rd BatchJobs/man/setJobFunction.Rd BatchJobs/man/callFunctionOnSSHWorkers.Rd BatchJobs/man/dbGetJobs.Rd BatchJobs/man/getConfig.Rd BatchJobs/man/makeSSHWorker.Rd BatchJobs/man/removeRegistrySourceFiles.Rd BatchJobs/man/cfReadBrewTemplate.Rd BatchJobs/man/removeRegistry.Rd BatchJobs/man/makeClusterFunctionsLocal.Rd
BatchJobs/LICENSE

Questions? Problems? Suggestions? or email at ian@mutexlabs.com.

Please suggest features or report bugs with the GitHub issue tracker.

All documentation is copyright its authors; we didn't write any of that.