ScripTests-package: Support for running transcript-style tests

Description Details Note Author(s)


Support for running transcript-style tests


Transcript-style tests are text files containing R commands and output, as though copied verbatim from an interactive R session. The output in the transcript file must match the actual output from running the command for the test to pass (with some exceptions - see "Control over matching" below). This testing framework is identical in concept to the standard .R/ tests that are run by the R CMD check, but with some enhancements that are intended to make test development and maintanance faster, more convenient, and easier to automate:

To make it so that "R CMD check" will run transcript-style tests, do the following:

  1. Make sure the scriptests package is installed on your system

  2. Put a file named "runtests.R" in the "tests" directory in your own package with the following contents:

  3. add transcript files ending in .Rt in the "tests" directory in your own package, e.g.:

    > 1 + 2
    [1] 3
  4. add the line Suggests: scriptests to DESCRIPTION file. If there is an existing "Suggests:" line, just add scriptests to it. (It's better to use the Suggests: than the Depends: fields, because packages listed in the depends field are loaded when the package is loaded for normal use, and the scriptests package is usually not needed for normal use of a package – the scriptests package will only be needed for testing.

At the end of testing, the file test-summary.txt will be left in the tests directory. To be entirely sure that the tests were run, also check for the existence of test-summary.txt.

If any tests fail, the file (a copy of test-summary.txt) will also be left in the tests directory – the existence of this file can be used in a programmatic check for whether all tests passed.

Tests can be run interactively using the function runtests(). The function source.pkg() can be useful to quickly re-read the function definitions in a package during code development. See the section "Running tests" in the documentation for runtests() for further details.


Control over matching

Actual output is matched to desired output extracted from the transcript file in a line-by-line fashion. If text is wrapped differently over multiple lines, the tests will fail (unless ignore-linebreaks is used). Different output width can easily happen if options("width") was different in the session that generated the desired output. Before trying to match, scriptests converts all white-space to single white-space, unless a control line specifies otherwise.

The following control lines can be present in the transcript after a command and before its output:


Ignore the output of this particular command – a test with this control line will always pass (unless it causes an R error, and options(error=function() NULL) was not set.)

#@gsub(pattern, replacement, WHAT)

where WHAT is target, actual or both (without quotes). Make a global substitution of replacement text for pattern text (a regular expression) in the desired (target) output or the actual output.


> cat("The date is <", date(), ">\n", sep="")
#@gsub("<[^>]*>", "<a date>", both)
The date is <Sat Jul 10 16:20:01 2010>
#@warn-only: OPTIONAL-TEXT

A mismatch is treated as an "warning", not an error

#@info-only: OPTIONAL-TEXT

A mismatch is treated as an "info" event, not an error

#@diff-msg: OPTIONAL-TEXT

Output OPTIONAL-TEXT if the desired and actual output do not match


Leave the whitespace as-is in the desired and actual output


Target and actual will match even if wrapped differently over multiple lines

The tests directory can also contain a CONFIG file, which can specify the functions to call for testing. The defaults are equivalent to the following lines in the CONFIG file:

Depends: scriptests
Debug: FALSE
Rsuffix: R
StopOnError: FALSE
Initialize: scriptests:::initializeTests()
Diff: scriptests:::ScripDiff()
Finalize: scriptests:::summarizeTests() 

The CONFIG file is optional and is not needed in ordinary usage.


The standard Emacs ESS functions for writing out ".Rt" files will strip trailing white space, which can result in many unimportant mismatches when using ediff to compare ".Rt" and ".Rout" files (e.g., because an R transcript will have "> " for empty command lines). Also, ".Rt" files are read-only by default, and the return key is bound to a command to send the current line to an R interpreter. It is more convenient if all these special behaviors are turned off. Put the following in your .emacs file to tell ESS not mess with ".Rt" files prior to saving them:

(add-hook 'ess-transcript-mode-hook
          ;; According to the ess docs, ess-nuke-trailing-whitespace-p
          ;; is supposed to be nil by default (see the defvar in ess-utils.el).
          ;; But it gets set to t somewhere else, so disable it here for
          ;; .Rt files, and also make RET behave the regular way.
          (lambda ()
          (if (string-match ".[Rr]t$" (buffer-name))
                (make-variable-buffer-local 'ess-nuke-trailing-whitespace-p)
                (define-key ess-transcript-mode-map (kbd "RET") 'newline)
                (toggle-read-only 0)
                (setq ess-nuke-trailing-whitespace-p nil)))) t)


Tony Plate <>

scriptests documentation built on May 1, 2019, 6:51 p.m.