Support for running transcript-style tests
Transcript-style tests are text files containing R commands and output, as
though copied verbatim from an interactive R session. The output in the
transcript file must match the actual output from running the command
for the test to pass (with some exceptions - see "Control over matching"
below). This testing framework is identical in concept to the standard
.R/.Rout.save tests that are run by the
R CMD check, but
with some enhancements that are intended to make test development and
maintanance faster, more convenient, and easier to automate:
Only the output file is needed - the inputs are parsed from
the output file (i.e.,
.Rt file, which is analogous to an
Test output matching is more lenient on white space differences, and more flexible in that some test output can be transformed by regular expressions prior to matching, or ignored entirely
directives can specify whether a test-case output mismatch
should be noted as an informational message, a warning, or an error
(one or more errors results in
R CMD check stopping with an
indication of error after running all tests). Unlike the standard
R CMD check, output mismatch detected by
scriptests results in
R CMD check stopping with an error.
A concise summary of warnings and errors is given at the end
Testing can continue after errors and can report multiple errors at the end, rather than stopping at the first error.
To make it so that "R CMD check" will run transcript-style tests, do the following:
Make sure the
scriptests package is installed on your system
Put a file named "runtests.R" in the "tests" directory in your own package with the following contents:
add transcript files ending in .Rt in the "tests" directory in your own package, e.g.:
> 1 + 2  3 >
add the line
Suggests: scriptests to DESCRIPTION file.
If there is an existing "Suggests:" line, just add
to it. (It's better to use the
Suggests: than the
Depends: fields, because packages listed in the depends field
are loaded when the package is loaded for normal use, and the
scriptests package is usually not needed for normal use of a package –
scriptests package will only be needed for testing.
At the end of testing, the file
test-summary.txt will be left in
tests directory. To be entirely sure that the tests were
run, also check for the existence of
If any tests fail, the file
test-summary.fail (a copy of
test-summary.txt) will also be left in the
– the existence of this file can be used in a programmatic check for
whether all tests passed.
Tests can be run interactively using the function
runtests(). The function
be useful to quickly re-read the function definitions in a package
during code development. See the section "Running tests" in the
runtests() for further details.
All commands in the transcript file must be prefixed with command or continuation prompts, exactly as they appear in a transcript.
scriptests uses simple heuristics to identify commands, comments and output. If the transcript cannot be separated into comments, commands and output by these heuristics (e.g., if a command prints out a line starting with the command prompt "> "), things will not work properly.
When running tests in a package, scriptests uses a heuristic to
guess the package name and automatically include an appropriate
library(package-being-tested) command before the tests. If
this heuristic fails, the functions from the package being tested
may not be accessible. If this problem occurs, it can be worked
around by explicitly including a
library(package-being-tested) command at the beginning of
To have tests continue to run after encountering an error, put
options(error=function() NULL) at the beginning
of the transcript file. This will cause the non-interactive R
session that runs the commands in the scripts to continue after an
error, instead of stopping, which is the default behavior for
Control over matching
Actual output is matched to desired output extracted from the
transcript file in a line-by-line fashion. If text is wrapped
differently over multiple lines, the tests will fail (unless
ignore-linebreaks is used). Different output width can easily
options("width") was different in the session that
generated the desired output. Before trying to match, scriptests
converts all white-space to single white-space, unless a control
line specifies otherwise.
The following control lines can be present in the transcript after a command and before its output:
Ignore the output of this particular
command – a test with this control line will always pass
(unless it causes an R error, and
NULL) was not set.)
both (without quotes).
Make a global substitution of
replacement text for
pattern text (a regular expression) in the desired
(target) output or the actual output.
1 2 3 4
A mismatch is treated as an "warning", not an error
A mismatch is treated as an "info" event, not an error
Output OPTIONAL-TEXT if the desired and actual output do not match
Leave the whitespace as-is in the desired and actual output
Target and actual will match even if wrapped differently over multiple lines
tests directory can also contain a
CONFIG file, which can
specify the functions to call for testing. The defaults are equivalent
to the following lines in the
1 2 3 4 5 6 7
CONFIG file is optional and is not needed in ordinary usage.
The standard Emacs ESS functions for writing out
".Rt" files will strip
trailing white space, which can result in many unimportant mismatches
ediff to compare
".Rout" files (e.g.,
because an R transcript will have
"> " for empty command lines).
".Rt" files are read-only by default, and the return key is
bound to a command to send the current line to an R interpreter. It is more
convenient if all these special behaviors are turned off.
Put the following in your
.emacs file to tell ESS not mess with
".Rt" files prior to saving them:
1 2 3 4 5 6 7 8 9 10 11 12
(add-hook 'ess-transcript-mode-hook ;; According to the ess docs, ess-nuke-trailing-whitespace-p ;; is supposed to be nil by default (see the defvar in ess-utils.el). ;; But it gets set to t somewhere else, so disable it here for ;; .Rt files, and also make RET behave the regular way. (lambda () (if (string-match ".[Rr]t$" (buffer-name)) (progn (make-variable-buffer-local 'ess-nuke-trailing-whitespace-p) (define-key ess-transcript-mode-map (kbd "RET") 'newline) (toggle-read-only 0) (setq ess-nuke-trailing-whitespace-p nil)))) t)
Tony Plate <email@example.com>
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.