ScripTests-package: Support for running transcript-style tests
In scriptests: Transcript-Based Unit Tests that are Easy to Create and Maintain

Description Details Note Author(s)

Support for running transcript-style tests

Transcript-style tests are text files containing R commands and output, as though copied verbatim from an interactive R session. The output in the transcript file must match the actual output from running the command for the test to pass (with some exceptions - see "Control over matching" below). This testing framework is identical in concept to the standard .R/.Rout.save tests that are run by the R CMD check, but with some enhancements that are intended to make test development and maintanance faster, more convenient, and easier to automate:

Only the output file is needed - the inputs are parsed from the output file (i.e., .Rt file, which is analogous to an .Rout.save file)
Test output matching is more lenient on white space differences, and more flexible in that some test output can be transformed by regular expressions prior to matching, or ignored entirely
directives can specify whether a test-case output mismatch should be noted as an informational message, a warning, or an error (one or more errors results in R CMD check stopping with an indication of error after running all tests). Unlike the standard tests in R CMD check, output mismatch detected by scriptests results in R CMD check stopping with an error.
A concise summary of warnings and errors is given at the end
Testing can continue after errors and can report multiple errors at the end, rather than stopping at the first error.

To make it so that "R CMD check" will run transcript-style tests, do the following:

Make sure the scriptests package is installed on your system
Put a file named "runtests.R" in the "tests" directory in your own package with the following contents:
```
library(scriptests)
runScripTests() 
```
add transcript files ending in .Rt in the "tests" directory in your own package, e.g.:
```
> 1 + 2
[1] 3
>   
```
add the line Suggests: scriptests to DESCRIPTION file. If there is an existing "Suggests:" line, just add scriptests to it. (It's better to use the Suggests: than the Depends: fields, because packages listed in the depends field are loaded when the package is loaded for normal use, and the scriptests package is usually not needed for normal use of a package – the scriptests package will only be needed for testing.

At the end of testing, the file test-summary.txt will be left in the tests directory. To be entirely sure that the tests were run, also check for the existence of test-summary.txt.

If any tests fail, the file test-summary.fail (a copy of test-summary.txt) will also be left in the tests directory – the existence of this file can be used in a programmatic check for whether all tests passed.

Tests can be run interactively using the function runtests(). The function source.pkg() can be useful to quickly re-read the function definitions in a package during code development. See the section "Running tests" in the documentation for runtests() for further details.

Notes:

All commands in the transcript file must be prefixed with command or continuation prompts, exactly as they appear in a transcript.
scriptests uses simple heuristics to identify commands, comments and output. If the transcript cannot be separated into comments, commands and output by these heuristics (e.g., if a command prints out a line starting with the command prompt "> "), things will not work properly.
When running tests in a package, scriptests uses a heuristic to guess the package name and automatically include an appropriate library(package-being-tested) command before the tests. If this heuristic fails, the functions from the package being tested may not be accessible. If this problem occurs, it can be worked around by explicitly including a library(package-being-tested) command at the beginning of each .Rt file.
To have tests continue to run after encountering an error, put the command options(error=function() NULL) at the beginning of the transcript file. This will cause the non-interactive R session that runs the commands in the scripts to continue after an error, instead of stopping, which is the default behavior for non-interactive R.

Control over matching

Actual output is matched to desired output extracted from the transcript file in a line-by-line fashion. If text is wrapped differently over multiple lines, the tests will fail (unless ignore-linebreaks is used). Different output width can easily happen if options("width") was different in the session that generated the desired output. Before trying to match, scriptests converts all white-space to single white-space, unless a control line specifies otherwise.

The following control lines can be present in the transcript after a command and before its output:

#@ignore-output: Ignore the output of this particular command – a test with this control line will always pass (unless it causes an R error, and options(error=function() NULL) was not set.)
#@gsub(pattern, replacement, WHAT): where WHAT is target, actual or both (without quotes). Make a global substitution of replacement text for pattern text (a regular expression) in the desired (target) output or the actual output.

E.g.,

> cat("The date is <", date(), ">\n", sep="")
#@gsub("<[^>]*>", "<a date>", both)
The date is <Sat Jul 10 16:20:01 2010>
>

#@warn-only: OPTIONAL-TEXT: A mismatch is treated as an "warning", not an error
#@info-only: OPTIONAL-TEXT: A mismatch is treated as an "info" event, not an error
#@diff-msg: OPTIONAL-TEXT: Output OPTIONAL-TEXT if the desired and actual output do not match
#@keep-whitespace: Leave the whitespace as-is in the desired and actual output
#@ignore-linebreaks: Target and actual will match even if wrapped differently over multiple lines

The tests directory can also contain a CONFIG file, which can specify the functions to call for testing. The defaults are equivalent to the following lines in the CONFIG file:

Depends: scriptests
Debug: FALSE
Rsuffix: R
StopOnError: FALSE
Initialize: scriptests:::initializeTests()
Diff: scriptests:::ScripDiff()
Finalize: scriptests:::summarizeTests()

The CONFIG file is optional and is not needed in ordinary usage.

The standard Emacs ESS functions for writing out ".Rt" files will strip trailing white space, which can result in many unimportant mismatches when using ediff to compare ".Rt" and ".Rout" files (e.g., because an R transcript will have "> " for empty command lines). Also, ".Rt" files are read-only by default, and the return key is bound to a command to send the current line to an R interpreter. It is more convenient if all these special behaviors are turned off. Put the following in your .emacs file to tell ESS not mess with ".Rt" files prior to saving them:

(add-hook 'ess-transcript-mode-hook
          ;; According to the ess docs, ess-nuke-trailing-whitespace-p
          ;; is supposed to be nil by default (see the defvar in ess-utils.el).
          ;; But it gets set to t somewhere else, so disable it here for
          ;; .Rt files, and also make RET behave the regular way.
          (lambda ()
          (if (string-match ".[Rr]t$" (buffer-name))
             (progn
                (make-variable-buffer-local 'ess-nuke-trailing-whitespace-p)
                (define-key ess-transcript-mode-map (kbd "RET") 'newline)
                (toggle-read-only 0)
                (setq ess-nuke-trailing-whitespace-p nil)))) t)