Parallel: Multi-core Processing

Description Warning References Examples

Description

From version 4.0 secr uses multi-threading in C++ (package RcppParallel, Allaire et al. 2021) to speed likelihood evaluation and hence model fitting in secr.fit. Detection histories are distributed over threads. Setting ncores = NULL in functions with multi-threading uses the existing value from the environment variable RCPP_PARALLEL_NUM_THREADS (see setNumThreads).

These functions use multi-threading and call setNumThreads internally:

autoini
confint.secr
derived.secr
derivedSystematic
esa
fxi.secr and related functions
pdot
region.N
score.test
secr.fit

These functions use multi-threading without calling setNumThreads:

LLsurface.secr
mask.check
expected.n
secr.test
sim.secr

Other functions may use multithreading indirectly through a call to one of these functions. Examples are suggest.buffer (autoini), esa.plot (pdot), and bias.D (pdot).

NOTE: The mechanism for setting the number of threads changed between versions 4.1.0 and 4.2.0. The default number of cores is now capped at 2 to meet CRAN requirements. Setting ncores = NULL previously specified one less than the maximum number of cores.

Earlier versions of secr made more limited use of multiple cores (CPUs) through the package parallel. This mechanism is still available in the functions listed here, but the speed gains are often small or even negative. Set ncores > 1 in the function call to use multiple cores.

Function Unit Benefit Notes
ip.secr replicate large
par.secr.fit model none
par.derived fitted model none
par.region.N fitted model none

‘Unit’ refers to the unit of work sent to each worker process. As a guide, a ‘large’ benefit means >60% reduction in process time with 4 CPUs.

parallel offers several different mechanisms, bringing together the functionality of multicore and snow. The mechanism used by secr is the simplest available, and is expected to work across all operating systems. Technically, it relies on Rscript and communication between the master and worker processes is via sockets. As stated in the parallel documentation "Users of Windows and Mac OS X may expect pop-up dialog boxes from the firewall asking if an R process should accept incoming connections". You may possibly get warnings from R about closing unused connections. These can safely be ignored.

Use parallel::detectCores() to get an idea of how many cores are available on your machine; this may (in Windows) include virtual cores over and above the number of physical cores. See RShowDoc("parallel", package = "parallel") in core R for explanation.

In secr.fit the output component ‘proctime’ misrepresents the elapsed processing time when multiple cores are used.

Warning

It appears that multicore operations in secr using parallel may fail if the packages snow and snowfall are also loaded. The error message is obscure:

“Error in UseMethod("sendData") : no applicable method for 'sendData' applied to an object of class "SOCK0node"”

References

Allaire, J. J., Francois, R., Ushey, K., Vandenbrouck, G., Geelnard, M. and Intel (2021) RcppParallel: Parallel Programming Tools for 'Rcpp'. R package version 5.1.2. https://CRAN.R-project.org/package=RcppParallel.

Examples

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
## Not run: 

sessionInfo()
# R version 3.6.1 (2019-07-05)
# Platform: x86_64-w64-mingw32/x64 (64-bit)
# Running under: Windows 7 x64 (build 7601) Service Pack 1
# quad-core i7 CPU, 16 Gb RAM
# ...

## benefit from multi-threading in secr.fit

for (i in 1:8) 
     print(system.time(secr.fit(ovenCH, buffer = 400, trace = FALSE, ncores = i)))
     
#   user  system elapsed 
#  54.25    0.17   54.43 
#   user  system elapsed 
#  29.48    0.16   20.30 
#   user  system elapsed 
#  43.34    0.17   25.04 
#   user  system elapsed 
#  43.92    0.14   22.95 
#   user  system elapsed 
#  46.16    0.22   22.70 
#   user  system elapsed 
#  31.59    0.17   15.19 
#   user  system elapsed 
#  45.58    0.12   21.93 
#   user  system elapsed 
#  37.43    0.15   18.43 

## and for simulation...
 
for (i in 1:8)
     print(system.time(sim.secr(secrdemo.0, nsim = 20, tracelevel = 0, ncores = i)))
     
#   user  system elapsed 
# 160.68    0.78  161.93 
#   user  system elapsed 
# 158.48    0.56   89.42 
#   user  system elapsed 
# 158.54    0.43   65.05 
#   user  system elapsed 
# 165.00    0.39   55.06 
#   user  system elapsed 
# 171.38    0.53   48.03 
#   user  system elapsed 
# 191.03    0.47   48.14 
#   user  system elapsed 
# 184.46    0.52   43.42 
#   user  system elapsed 
# 193.07    0.56   42.34 

for (i in 1:8) 
    print(system.time(ip.secr (captdata, trace = FALSE, ncores = i)))

#   user  system elapsed 
# 121.88    0.08  122.27 
#   user  system elapsed 
#   0.54    0.42   72.85 
#   user  system elapsed 
#   0.55    0.76   55.91 
#   user  system elapsed 
#   0.91    0.77   47.65 
#   user  system elapsed 
#   1.21    0.81   44.83 
#   user  system elapsed 
#   1.42    1.23   43.21 
#   user  system elapsed 
#   1.18    1.98   42.46 
#   user  system elapsed 
#   1.81    1.81   42.54  
 
for (i in 1:8)
    print(system.time(LLsurface(secrdemo.0, ncores = i)))

# Evaluating log likelihood across grid of 121 points...
#    user  system elapsed 
#   26.59    0.14   26.80 
# Evaluating log likelihood across grid of 121 points...
#    user  system elapsed 
#   25.74    0.13   17.73 
# Evaluating log likelihood across grid of 121 points...
#    user  system elapsed 
#   26.08    0.18   15.12 
# Evaluating log likelihood across grid of 121 points...
#    user  system elapsed 
#   26.93    0.19   13.82 
# Evaluating log likelihood across grid of 121 points...
#    user  system elapsed 
#   28.32    0.06   13.52 
# Evaluating log likelihood across grid of 121 points...
#    user  system elapsed 
#   29.40    0.16   13.05 
# Evaluating log likelihood across grid of 121 points...
#    user  system elapsed 
#   29.76    0.22   12.92 
# Evaluating log likelihood across grid of 121 points...
#    user  system elapsed 
#   29.10    0.20   12.81 


## End(Not run)

secr documentation built on Oct. 18, 2021, 9:06 a.m.