Manage parallel Azure connections
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
Other arguments passed on to functions in the parallel package. See below.
AzureRMR provides the ability to parallelise communicating with Azure by utilizing a pool of R processes in the background. This often leads to major speedups in scenarios like downloading large numbers of small files, or working with a cluster of virtual machines. This functionality is intended for use by packages that extend AzureRMR (and was originally implemented as part of the AzureStor package), but can also be called directly by the end-user.
A small API consisting of the following functions is currently provided for managing the pool. They pass their arguments down to the corresponding functions in the parallel package.
init_pool initialises the pool, creating it if necessary. The pool is created by calling
parallel::makeCluster with the pool size and any additional arguments. If
init_pool is called and the current pool is smaller than
size, it is resized.
delete_pool shuts down the background processes and deletes the pool.
pool_exists checks for the existence of the pool, returning a TRUE/FALSE value.
pool_size returns the size of the pool, or zero if the pool does not exist.
pool_export exports variables to the pool nodes. It calls
parallel::clusterExport with the given arguments.
pool_map carry out work on the pool. They call
parallel::clusterMap with the given arguments.
pool_evalq execute code on the pool nodes. They call
parallel::clusterEvalQ with the given arguments.
The pool is persistent for the session or until terminated by
delete_pool. You should initialise the pool by calling
init_pool before running any code on it. This restores the original state of the pool nodes by removing any objects that may be in memory, and resetting the working directory to the master working directory.
parallel::makeCluster, parallel::clusterCall, parallel::parLapply
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.