benchmark: High Precise Measurement of R Expressions Execution Time

In benchr: High Precise Measurement of R Expressions Execution Time

Description

benchmark serves as a more accurate replacement of the often seen system.time(replicate(1000, expr)) expression. It tries hard to accurately measure only the time it takes to evaluate expr. To achieve this, the sub-millisecond (supposedly nanosecond) accurate timing functions most modern operating systems provide are used. Additionally all evaluations of the expressions are done in C++ code to minimize any measurment error.

Usage

benchmark(
  ...,
  times = 100L,
  order = c("random", "inorder", "block"),
  envir = parent.frame(),
  progress = TRUE,
  gcFirst = TRUE,
  gcDuring = FALSE
)

Arguments

...	Captures any number of unevaluated expressions passed to benchmark as named or unnamed arguments.
times	Integer. Number of times to evaluate each expression.
order	Character. The order in which the expressions are evaluated.
envir	The environment in which the expressions will be evaluated.
progress	Logical. Show progress bar during expressions evaluation.
gcFirst	Logical. Should a garbage collection be performed immediately before the timing?
gcDuring	Logical. Should a garbage collection be performed immediately before each iteration, as produced by times? (very slow)

Details

Before evaluating each expression times times, the overhead of calling the timing functions and the C++ function call overhead are estimated. This estimated overhead is subtracted from each measured evaluation time. Should the resulting timing be negative, a warning is thrown and the respective value is replaced by 0. If the timing is zero, a warning is also raised. Should all evaluations result in one of the two error conditions described above, an error is raised.

Value

Object of class benchmark, which is a data.frame with a number of additional attributes and contains the following columns:

expr	The deparsed expression as passed to benchmark or the name of the argument if the expression was passed as a named argument.
time	The measured execution time of the expression in seconds. The order of the observations in the data frame is the order in which they were executed.

An object of class benchmark also contains the following attributes:

precision	Timer precision in seconds.
error	Timer error (overhead) in seconds.
units	Units for time intervals (by default, "s" – seconds).
times	Number of repeats for each measurement.
order	Execution regime.
gc	Whether garbage collection took place before each execution.

The order in which the expressions are evaluated

“random”

(the default) randomizes the execution order

“inorder”

executes each expression in order

“block”

executes all repetitions of each expression as one block.

Author(s)

Artem Klevtsov [e-mail]

See Also

summary.benchmark(), mean.benchmark(), print.benchmark(), plot.benchmark(), boxplot.benchmark()

Examples

## Measure the time it takes to dispatch a simple function call
## compared to simply evaluating the constant NULL
f <- function() NULL
res <- benchmark(NULL, f(), times = 1000L)

## Print results:
print(res)

## Plot results
boxplot(res)

Example output

0%   10   20   30   40   50   60   70   80   90   100%
[----|----|----|----|----|----|----|----|----|----|
**************************************************|
Benchmark summary:
Time units : nanoseconds 
 expr n.eval min lw.qu median   mean up.qu     max   total relative
 NULL   1000   4     6      7    7.7     7     254    7700      1.0
  f()   1000 133   138    141 3610.0   182 3430000 3610000     20.1

benchr documentation built on March 13, 2020, 3:29 a.m.