BatchJobs: Batch Computing with R

Share:

Provides Map, Reduce and Filter variants to generate jobs on batch computing systems like PBS/Torque, LSF, SLURM and Sun Grid Engine. Multicore and SSH systems are also supported. For further details see the project web page.

Author
Bernd Bischl <bernd_bischl@gmx.net>, Michel Lang <michellang@gmail.com>, Henrik Bengtsson <henrikb@braju.com>
Date of publication
2015-03-18 18:01:21
Maintainer
Bernd Bischl <bernd_bischl@gmx.net>
License
BSD_2_clause + file LICENSE
Version
1.6
URLs

View on CRAN

Man pages

addRegistryPackages
Add packages to registry.
addRegistrySourceDirs
Add source dirs to registry.
addRegistrySourceFiles
Add source files to registry.
applyJobFunction
applyJobFunction ONLY FOR INTERNAL USAGE.
batchExpandGrid
Map function over all combinations.
batchExport
Export R object to be available on the slaves.
BatchJobs
The BatchJobs package
batchMap
Maps a function over lists or vectors, adding jobs to a...
batchMapQuick
Combination of makeRegistry, batchMap and submitJobs.
batchMapResults
Maps a function over the results of a registry by using...
batchReduce
Reduces via a binary function over a list adding jobs to a...
batchReduceResults
Reduces results via a binary function and adds jobs for this...
batchUnexport
Unload exported R objects.
callFunctionOnSSHWorkers
Call an arbitrary function on specified SSH workers.
cfBrewTemplate
Cluster functions helper: Brew your template into a job...
cfHandleUnknownSubmitError
Cluster functions helper: Handle an unknown error during job...
cfKillBatchJob
Cluster functions helper: Kill a batch job via OS command
cfReadBrewTemplate
Cluster functions helper: Read in your brew template file.
configuration
BatchJobs configuration.
copyRequiredJobFiles
ONLY FOR INTERNAL USAGE.
dbCreateJobDefTable
ONLY FOR INTERNAL USAGE.
dbGetJobs
ONLY FOR INTERNAL USAGE.
debugMulticore
Helper function to debug multicore mode.
debugSSH
Helper function to debug SSH mode.
filterResults
Find all results where a specific condition is true.
findJobs
Finds ids of jobs that match a query.
findState
Find jobs depending on computional state.
getConfig
Returns a list of BatchJobs configuration settings
getErrorMessages
Get error messages of jobs.
getJob
Get job from registry by id.
getJobIds
Get ids of jobs in registry.
getJobInfo
Get computational information of jobs.
getJobNr
Get number of jobs in registry.
getJobParamDf
Returns parameters for all jobs as the rows of a data.frame.
getJobResources
Function to get the resources that were submitted for some...
getJobs
Get jobs from registry by id.
getLogFiles
Get log file paths for jobs.
getResources
Function to get job resources in job function.
getSSHWorkersInfo
Print and return R installation and other information for SSH...
grepLogs
Grep log files for a pattern.
installPackagesOnSSHWorkers
Install packages on SSH workers.
killJobs
Kill some jobs on the batch system.
loadConfig
Load a specific configuration file.
loadExports
Load exported R data objects.
loadRegistry
Load a previously saved registry.
loadResult
Loads a specific result file.
loadResults
Loads result files for id vector.
makeClusterFunctions
Create a ClusterFuntions object.
makeClusterFunctionsInteractive
Create cluster functions for sequential execution in same...
makeClusterFunctionsLocal
Create cluster functions for sequential execution on local...
makeClusterFunctionsLSF
Create cluster functions for LSF systems.
makeClusterFunctionsMulticore
Use multiple cores on local Linux machine to spawn parallel...
makeClusterFunctionsSGE
Create cluster functions for Sun Grid Engine systems.
makeClusterFunctionsSLURM
Create cluster functions for SLURM-based systems.
makeClusterFunctionsSSH
Create an SSH cluster to execute jobs.
makeClusterFunctionsTorque
Create cluster functions for torque-based systems.
makeJob
Creates a job description.
makeRegistry
Construct a registry object.
makeSSHWorker
Create SSH worker for SSH cluster functions.
makeSubmitJobResult
Create a SubmitJobResult object.
reduceResults
Reduce results from result directory.
removeRegistry
Remove a registry object.
removeRegistryPackages
Remove packages from registry.
removeRegistrySourceDirs
Remove packages from registry.
removeRegistrySourceFiles
Remove source files from registry.
resetJobs
Reset computational state of jobs.
sanitizePath
Sanitize a path
setConfig
Set and overwrite configuration settings
setJobFunction
Sets the job function for already existing jobs.
setJobNames
Set job names.
showClusterStatus
Show information about available computational resources on...
showLog
Display the contents of a log file.
showStatus
Retrieve or show status information about jobs.
sourceRegistryFiles
Source registry files
submitJobs
Submit jobs or chunks of jobs to batch system via cluster...
sweepRegistry
Sweep obsolete files from the file system.
testJob
Tests a job by running it with Rscript in a new process.
updateRegistry
ONLY FOR INTERNAL USAGE.
waitForJobs
Wait for termination of jobs on the batch system.

Files in this package

BatchJobs
BatchJobs/inst
BatchJobs/inst/CITATION
BatchJobs/inst/bin
BatchJobs/inst/bin/linux-helper
BatchJobs/inst/etc
BatchJobs/inst/etc/BatchJobs_global_config.R
BatchJobs/tests
BatchJobs/tests/run-all.R
BatchJobs/tests/testthat
BatchJobs/tests/testthat/test_sweepRegistry.R
BatchJobs/tests/testthat/test_export.R
BatchJobs/tests/testthat/helpers.R
BatchJobs/tests/testthat/test_reduceResults.R
BatchJobs/tests/testthat/test_getJobInfo.R
BatchJobs/tests/testthat/test_sanitizePath.R
BatchJobs/tests/testthat/test_testJob.R
BatchJobs/tests/testthat/test_filterResults.R
BatchJobs/tests/testthat/test_getJobParamDf.R
BatchJobs/tests/testthat/test_batchMapQuick.R
BatchJobs/tests/testthat/test_loadResults.R
BatchJobs/tests/testthat/test_submitJobs.R
BatchJobs/tests/testthat/test_resources.R
BatchJobs/tests/testthat/test_resetJobs.R
BatchJobs/tests/testthat/test_getLogFiles.R
BatchJobs/tests/testthat/test_waitForJobs.R
BatchJobs/tests/testthat/test_batchMap.R
BatchJobs/tests/testthat/test_setJobFunction.R
BatchJobs/tests/testthat/test_findJobs.R
BatchJobs/tests/testthat/test_showStatus.R
BatchJobs/tests/testthat/test_batchReduce.R
BatchJobs/tests/testthat/test_findStatus.R
BatchJobs/tests/testthat/test_batchMapResults.R
BatchJobs/tests/testthat/test_batchReduceResults.R
BatchJobs/tests/testthat/test_sourceRegistryFiles.R
BatchJobs/tests/testthat/test_zzz_cleanup.R
BatchJobs/tests/testthat/test_getJob.R
BatchJobs/tests/testthat/test_database.R
BatchJobs/tests/testthat/test_sqlquotes.R
BatchJobs/tests/testthat/test_batchExpandGrid.R
BatchJobs/tests/testthat/test_packages.R
BatchJobs/tests/testthat/test_makeRegistry.R
BatchJobs/tests/testthat/test_grepLogs.R
BatchJobs/tests/testthat/test_doJob.R
BatchJobs/NAMESPACE
BatchJobs/NEWS
BatchJobs/R
BatchJobs/R/getJobInfo.R
BatchJobs/R/buffer.R
BatchJobs/R/clusterFunctionsInteractive.R
BatchJobs/R/clusterFunctions.R
BatchJobs/R/Registry.R
BatchJobs/R/filterResults.R
BatchJobs/R/reduceResults.R
BatchJobs/R/killJobs.R
BatchJobs/R/database.R
BatchJobs/R/writeRscripts.R
BatchJobs/R/callFunctionOnSSHWorkers.R
BatchJobs/R/getJobParamDf.R
BatchJobs/R/helpers.R
BatchJobs/R/Worker.R
BatchJobs/R/Job.R
BatchJobs/R/conf.R
BatchJobs/R/debugMulticore.R
BatchJobs/R/clusterFunctionsHelpers.R
BatchJobs/R/showLog.R
BatchJobs/R/debugSSH.R
BatchJobs/R/batchApply.R
BatchJobs/R/batchExpandGrid.R
BatchJobs/R/filenames.R
BatchJobs/R/Packages.R
BatchJobs/R/getErrorMessages.R
BatchJobs/R/loadResults.R
BatchJobs/R/setJobNames.R
BatchJobs/R/sanitizePath.R
BatchJobs/R/findStatus.R
BatchJobs/R/submitJobs.R
BatchJobs/R/batchReduceResults.R
BatchJobs/R/clusterFunctionsMulticore.R
BatchJobs/R/Exports.R
BatchJobs/R/syncRegistry.R
BatchJobs/R/getLogFiles.R
BatchJobs/R/sourceRegistryFiles.R
BatchJobs/R/showStatus.R
BatchJobs/R/sendMail.R
BatchJobs/R/clusterFunctionsSLURM.R
BatchJobs/R/installPackagesOnSSHWorkers.R
BatchJobs/R/getJob.R
BatchJobs/R/clusterFunctionsSGE.R
BatchJobs/R/loadResult.R
BatchJobs/R/doJob.R
BatchJobs/R/testJob.R
BatchJobs/R/waitForJobs.R
BatchJobs/R/setJobFunction.R
BatchJobs/R/clusterFunctionsSSH.R
BatchJobs/R/checkIdValid.R
BatchJobs/R/showClusterStatus.R
BatchJobs/R/batchReduce.R
BatchJobs/R/batchMapQuick.R
BatchJobs/R/clusterFunctionsLSF.R
BatchJobs/R/Resources.R
BatchJobs/R/clusterFunctionsTorque.R
BatchJobs/R/resetJobs.R
BatchJobs/R/sweepRegistry.R
BatchJobs/R/clusterFunctionsLocal.R
BatchJobs/R/updateRegistry.R
BatchJobs/R/batchMapResults.R
BatchJobs/R/findJobs.R
BatchJobs/R/getSSHWorkersInfo.R
BatchJobs/R/grepLogs.R
BatchJobs/R/batchMap.R
BatchJobs/R/zzz.R
BatchJobs/R/WorkerLinux.R
BatchJobs/R/scheduleWorkerJobs.R
BatchJobs/R/applyJobFunction.R
BatchJobs/R/addJob.R
BatchJobs/MD5
BatchJobs/DESCRIPTION
BatchJobs/man
BatchJobs/man/makeClusterFunctionsLSF.Rd
BatchJobs/man/loadResults.Rd
BatchJobs/man/addRegistrySourceDirs.Rd
BatchJobs/man/updateRegistry.Rd
BatchJobs/man/dbCreateJobDefTable.Rd
BatchJobs/man/getJobs.Rd
BatchJobs/man/setJobNames.Rd
BatchJobs/man/applyJobFunction.Rd
BatchJobs/man/installPackagesOnSSHWorkers.Rd
BatchJobs/man/makeClusterFunctionsMulticore.Rd
BatchJobs/man/killJobs.Rd
BatchJobs/man/makeRegistry.Rd
BatchJobs/man/getErrorMessages.Rd
BatchJobs/man/configuration.Rd
BatchJobs/man/makeJob.Rd
BatchJobs/man/makeClusterFunctionsSGE.Rd
BatchJobs/man/batchExport.Rd
BatchJobs/man/makeClusterFunctionsSLURM.Rd
BatchJobs/man/batchMapResults.Rd
BatchJobs/man/cfHandleUnknownSubmitError.Rd
BatchJobs/man/loadResult.Rd
BatchJobs/man/batchMap.Rd
BatchJobs/man/showClusterStatus.Rd
BatchJobs/man/sanitizePath.Rd
BatchJobs/man/setConfig.Rd
BatchJobs/man/waitForJobs.Rd
BatchJobs/man/addRegistrySourceFiles.Rd
BatchJobs/man/copyRequiredJobFiles.Rd
BatchJobs/man/makeSubmitJobResult.Rd
BatchJobs/man/showStatus.Rd
BatchJobs/man/batchMapQuick.Rd
BatchJobs/man/loadRegistry.Rd
BatchJobs/man/reduceResults.Rd
BatchJobs/man/batchReduceResults.Rd
BatchJobs/man/filterResults.Rd
BatchJobs/man/loadConfig.Rd
BatchJobs/man/getJobInfo.Rd
BatchJobs/man/grepLogs.Rd
BatchJobs/man/makeClusterFunctions.Rd
BatchJobs/man/submitJobs.Rd
BatchJobs/man/makeClusterFunctionsInteractive.Rd
BatchJobs/man/findState.Rd
BatchJobs/man/cfKillBatchJob.Rd
BatchJobs/man/findJobs.Rd
BatchJobs/man/testJob.Rd
BatchJobs/man/showLog.Rd
BatchJobs/man/debugSSH.Rd
BatchJobs/man/removeRegistrySourceDirs.Rd
BatchJobs/man/getJobIds.Rd
BatchJobs/man/getResources.Rd
BatchJobs/man/batchReduce.Rd
BatchJobs/man/makeClusterFunctionsTorque.Rd
BatchJobs/man/addRegistryPackages.Rd
BatchJobs/man/getLogFiles.Rd
BatchJobs/man/debugMulticore.Rd
BatchJobs/man/removeRegistryPackages.Rd
BatchJobs/man/getJobParamDf.Rd
BatchJobs/man/getJobResources.Rd
BatchJobs/man/batchUnexport.Rd
BatchJobs/man/getSSHWorkersInfo.Rd
BatchJobs/man/sweepRegistry.Rd
BatchJobs/man/batchExpandGrid.Rd
BatchJobs/man/BatchJobs.Rd
BatchJobs/man/getJobNr.Rd
BatchJobs/man/makeClusterFunctionsSSH.Rd
BatchJobs/man/getJob.Rd
BatchJobs/man/resetJobs.Rd
BatchJobs/man/cfBrewTemplate.Rd
BatchJobs/man/sourceRegistryFiles.Rd
BatchJobs/man/loadExports.Rd
BatchJobs/man/setJobFunction.Rd
BatchJobs/man/callFunctionOnSSHWorkers.Rd
BatchJobs/man/dbGetJobs.Rd
BatchJobs/man/getConfig.Rd
BatchJobs/man/makeSSHWorker.Rd
BatchJobs/man/removeRegistrySourceFiles.Rd
BatchJobs/man/cfReadBrewTemplate.Rd
BatchJobs/man/removeRegistry.Rd
BatchJobs/man/makeClusterFunctionsLocal.Rd
BatchJobs/LICENSE