atime_pkg: Asymptotic timing package tests

In atime: Asymptotic Timing

Asymptotic timing package tests

Description

R package performance testing, by computing time/memory usage of several R expressions of several different data sizes, for several package versions (base, HEAD, CRAN, merge-base, other historical references specified by user). atime_pkg_test_info returns an environment containing test code/calls (useful for running a single test), whereas atime_pkg runs all tests and saves results/plots to disk.

Usage

atime_pkg(pkg.path=".", tests.dir=NULL)
atime_pkg_test_info(pkg.path=".", tests.dir=NULL)

Arguments

pkg.path	path to git repository containing R package.
tests.dir	path to directory which contains atime/tests.R, relative to pkg.path (default NULL means first existing directory of "inst" or ".ci").

Details

There should be a tests.R code file which defines test.list, a list with names corresponding to different tests. Each element should be a list with at least three named elements: N, setup, expr, (possibly others such as pkg.edit.fun and version_name="1234567890abcdef") to be passed as named arguments to atime_versions, along with the following versions which are passed using the sha.vec argument: base ref comes from GITHUB_BASE_REF environment variable (default master), HEAD ref is the branch that you want to merge into base, CRAN is current published version (sha value ""), merge-base is most recent common ancestor commit between base and HEAD. For visualization, default colors are provided for versions with names: HEAD, base, CRAN, merge-base, Before, Regression, Slow, Fixed, Fast; other version names will be gray using the default colors. If tests.R defines a variable named version.colors, then it should be a character vector of colors to be used instead of the default (names for versions, values for colors). If tests.R defines a variable named pval.thresh, then it should be a positive numeric value between 0 and 1 (any smaller p-values will generate messages about slowdowns in HEAD_issues.md).

Value

atime_pkg_test_info returns an environment in which the code of tests.R was evaluated, including a variable test.call which is a list of un-evaluated atime_versions calls, one for each test (use with eval to run a single test). atime_pkg returns a named list of test results, names come from names of test.list, and values come from results of atime_versions. Side effect is that data/plot files are saved in atime directory, including tests.RData (test results which can be read into R if you want to make your own alternative plots/analyses), tests_all_facet.png (plot summarizing all test results), tests_preview_facet.png (plot summarizing only most significant results), and install_seconds.txt (total number of seconds used to install different package versions). Plots emphasize differences between HEAD and a comparison version, which is defined as the first of these versions which is available: merge-base, base, CRAN. Plots show test cases, first sorted by N.factor, which is the ratio of N values at the time limit, HEAD versus comparison version. If N.factor<1 then HEAD is much slower than comparison version (the smaller N value for HEAD went over the time limit). N.factor=1 means HEAD and comparison version had same value for largest N (so have nearly the same speed); ties are broken using P.value, un-paired two-sample T-test for HEAD time greater than comparison time at largest N.

Author(s)

Toby Dylan Hocking

See Also

atime_test for defining each test, atime_test_list for defining common arguments in each element of the test list.

Examples

if(FALSE){

  ## Clone binsegRcpp, and checkout a branch which has performance tests.
  tdir <- tempfile()
  dir.create(tdir)
  git2r::clone("https://github.com/tdhock/binsegRcpp", tdir)
  repo <- git2r::repository(tdir)
  git2r::checkout(repo, "another-branch")

  ## Run one test.
  info.env <- atime::atime_pkg_test_info(tdir)
  test_name <- "binseg(1:N,maxSegs=N/2) DIST=l1"
  (one.call <- info.env$test.call[[test_name]])
  one.result <- eval(one.call)
  plot(one.result) # constant difference for small N should be visible.

  ## Run all tests.
  result.list <- atime::atime_pkg(tdir)
  inst.atime <- file.path(tdir, "inst", "atime")
  dir(inst.atime)
  tests.RData <- file.path(inst.atime, "tests.RData")
  (objs <- load(tests.RData))
  plot(pkg.results[[test_name]]) # should look similar.

  atime::atime_versions_remove("binsegRcpp")

}

## https://github.com/tdhock/binsegRcpp/blob/atime-test-funs/.ci/atime/tests.R
## has another real example, see how to run it in tests/testthat/test-versions.R

atime documentation built on June 8, 2025, 12:46 p.m.