Description Details Comparisons to previous scriptests designs Original scriptests design based on Makefiles Not used: A Makefile framework that modifies Makevars to create a goal, and also uses a dummy .R file A Makefile framework using .DEFAULT\_GOAL
Design considerations for package scriptests. This file is a poorly organized collection of notes regarding the design and evolution of the scriptests package.
"R CMD check" is (as of R-2.9.0) an R script
src/scripts/check.in
, which invokes R with the command
tools:::.runPackageTestsR($extra)
to run tests. If this R
session returns non-zero, the check process stops with an error,
outputting 13 lines from the first .fail file found (if one exists).
tools:::.runPackageTestsR($extra)
invokes R CMD BATCH
to
run each .R file. This is where the non-overrideable redirection of
stderr to .Rout happens.
New design for scriptests to work with R-2.9.0 in which "R CMD check" no longer uses Makefiles in tests directory.
All transcript files are stored in the tests
directory
with the file extension .Rt
.
There should be a file called runtests.Rin
, which
should contain the following two lines:
library(scriptests) runScripTests()
When runScripTests()
runs, it will create a .R
file with the commands extracted from each .Rt
file. It will
then run each .R
file in a separate R
session, save
the output in a .Rout
file, and compare the output with the
.Rt
file. runScripTests
will leave a summary in the
file test-summary.txt
. If there are errors, the summary will
be duplicated in the file test-summary.fail
(the presence of
a file ending in .fail
signals an error to "R CMD
check"
.)
The functionality that this design gives up over previous designs is
immediate checking of test output. Instead test output is checked
after all tests are run. This is because of the difficulty of
controlling the execution order of different test files – it's hard
to create pairs of test/check files that will run in the correct order
(order of execution depends on the order of files in the value of
dir
).
The functionality that this design retains from previous designs (and from the native testing framework in R) is that each file of R commands to be checked in run in a new R session. This means that commands in one test file cannot mess up another test file. It also means that the testing framework doesn't need to mess around with trying to capture output.
There are several possible methods that could be used to check output immediately after running each test:
prepend each .R
file with a function that checks output
of all previously run tests (skipping ones already done).
append to each .R
file a call to a function that
flushes stdout (flush(stdout())
) and checks tests output.
define a .Last()
function (will be called by
q()
) that checks test output.
Hangup with the above approach: it's not possible to have output from the runs go to stdout – because all the output from running a .R file is redirected to the corresponding .Rout file.
Alternate approach: a setup file calls my own version of .runPackageTests(), which then executes an R session for each .R file. Still have the problem of not getting any output to the console. Can solve this by by-passing R CMD BATCH, and just invoking R more directly.
An initial design changed the default goal by assigning .DEFAULT\_GOAL in tests/Makefile. This works with GNU make version 3.81, which is standard in Ubuntu Linux. However, the Rtools set of programs for Windows includes GNU make version 3.79, which does not appear to recognize the .DEFAULT\_GOAL special variable. Additionally, version 2.6.2 (2008-02-08) of "R Installation and Administration" specifically says that GNU make version 3.81 does not work to compile under Windows. Furthermore, Mac OS X version 10.4 (Tiger) includes GNU make version 3.80, which also does not appear to recognize the .DEFAULT\_GOAL special variable.
Consequently, a different approach to getting R CMD check to run additional tests is needed.
The approach implemented as of version 0.1-6 (March 2009) is to use a
"%" target in the makefile (which is always called), with an action in
the body of the rule that invokes a 'make' recursively with the desired
target (here all-Rt
).
http://www.gnu.org/software/automake/manual/make/Force-Targets.html describes 'force targets'.
Here's the relevant section from scriptests/inst/scripts/Makefile.sub:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | # Use 'force' to effectively create another target, by calling
# make recursively with the target 'all-Rt'.
# Based on code at
# http://www.gnu.org/software/automake/manual/make/Overriding-Makefiles.html
# but with more levels of protection to avoid calling make with
# the target 'all-Rt' more than once, because this makefile is
# read many times. Condition on DONEFORCE being not defined
# to avoid infinite recursion.
ifeq ($(strip $(DONEFORCE)),)
@(if [ ! -f forceonce ] ; then \
$(MAKE) -f $(R_SHARE_DIR)/make/$(RSHAREMAKEFILE) $(makevars) -f $(MAINTESTMAKE) DONEFORCE=TRUE all-Rt ; \
fi )
@touch forceonce
force: ;
endif
|
The various 'make' variables used here are defined in ...
This makefile does two things to try to create a goal: (1) it edits Makevars to insert an extra goal; and (2) it uses a specific rule for a allrt.Rin to allrt.R to run a sub-make (the rule is written to Makevars).
Both of these approaches should work, but the whole thing does not seem to work reliably. In any case, the sub-make can be called directly from the "force" block instead of using the force block to create a rule that calls the sub-make. This code is preserved here because it uses a number of techniques that might come in handy somewhere else.
This framework uses three files in the tests directory:
Makefile
, Makefile.win
and allrt.Rin
.
Makefile:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 | # Redefine the default goal (a GNU make feature) so that we
# have the Rt tests as subgoals.
# Note that the default goal can contain only one target.
# .DEFAULT_GOAL := all+Rt
ifeq ($(strip $(RSHAREMAKEFILE)),)
RSHAREMAKEFILE=tests.mk
endif
# Need MAINTESTMAKE so that when we edit Makevars we can set $makevars
# to include the same makefile as R CMD check uses (which, under Windows,
# is Makefile.win if it exists)
ifeq ($(strip $(MAINTESTMAKE)),)
MAINTESTMAKE=Makefile
endif
# Define ScripTestsErrorAction to be 'continue' or 'stop'.
# This controls what happens running under R CMD check
# when there are errors in one of the Rt tests.
ScripTestsErrorAction=stop
# Under Unix-likes we want to run R like this:
# @R_LIBS=$(R_LIBS) $(R) ... arguments ...
# while under Windows we want to run R like this:
# $(R) @R_LIBS=$(R_LIBS) ... arguments ...
# So, always run R like this: $(R_PRE) @R_LIBS=$(R_LIBS) $(R_POST)
# with appropriate definitions for R_PRE and R_POST.
# If we are running under Windows, we will have already executed
# Makefile.win, which will have defined R_PRE=$(R) .
ifeq ($(strip $(R_PRE)),)
# Unix-like
R_PRE=
R_POST=$(R)
endif
# If we do want to change the RDIFF command, should include
# 'changeRdiff' as the second dependency for all+Rt
all+Rt: createScripts all rt-tests ScripTests.summary
all-Rt: createScripts rt-tests ScripTests.summary
@echo doing all-Rt
allrt := $(wildcard *.Rt)
# allrtwant := $(allrt:.Rt=.Rout.want)
# allrtout := $(allrt:.Rt=.Rout)
allrtres := $(allrt:.Rt=.Rtres)
# allrtin := $(allrt:.Rt=.R)
@(if [ ! -f forceonce ] ; then \
echo Modifying Makevars for rttests ; \
sed -i 's/test-src-1 =/test-src-1 = all-Rt/' Makevars ; \
sed -i "s/makevars =/makevars = -f $(MAINTESTMAKE)/" Makevars ; \
fi )
@(if [ ! -f forceonce ] ; then echo 'allrt.R:
echo ' @echo Using special rule $$@ to run all tests ... ' ; \
echo ' $$(MAKE) -f $$(R_SHARE_DIR)/make/tests.mk $$(makevars) all-Rt' ; \
echo ' @echo 1+1 > $$@' ; fi ) >> Makevars
@touch forceonce
force: ;
# Don't need this unless we want standard .R/.Rout.save tests
# to use the modified RDIFF (not the ones run here - they use
# the commands in the .Rout.Rtres rule)
changeRdiff:
@echo Changing Rdiff in Makevars
echo RDIFF = echo hehehe >> Makevars
# Create scripts we will need (do this here to minimize the
# number of files needed to support scriptests in a package.)
createScripts:
@( echo '## Run a script in the scriptests/scripts (from scriptests/inst/scripts)' ; \
echo '## This file does not have .R suffix because if it did, R CMD check would want to run it as a test' ; \
echo 'library(package="scriptests", char=TRUE)' ; \
echo 'args <- commandArgs(TRUE)' ; \
echo 'debug <- is.element("--debug", args)' ; \
echo 'if (length(args)>0) {' ; \
echo ' script.path <- system.file(package="scriptests", "scripts")' ; \
echo ' if (debug) {' ; \
echo ' cat("RtTestScript called with ", length(args), " args: ", paste("\"", args, "\"", collapse=", ", sep=""), "\n", sep="")' ; \
echo ' cat("Looking for scripts in \"", script.path, "\"\n", sep="")' ; \
echo ' }' ; \
echo ' source(file.path(script.path, args[1]))' ; \
echo '}' ) > RunScripTestsScript
# Don't need the following because we can run and check the tests
# with out needing a sub-make. If we do use a sub-make, then we
# need to put rules used in 'Makevars'.
# echo .SUFFIXES: .R .Rin .Rout .Rt >> Makevars
# echo .Rt.R: >> Makevars
# echo ' echo Creating $$@ from $$<' >> Makevars
# echo ' sed -n "s/^[>+] //p" $$< > $$@' >> Makevars
# echo ' cat $$< > $$@out.save' >> Makevars
# This suffix list and rule for .Rt.R needs to go in Makevars
.SUFFIXES: .R .Rin .Rout .Rt .Rtres
# Files:
# .Rt: a transcript, with both commands and desired output, & possibly directives
# .R: R commands generated from .Rt
# .Rout.want: desired transcript output generated from .Rt (maybe don't need this)
# .Rt.save: the processed Rt script as an R object
# .Rout: the output from R when given the commands in .R
# .Rtres: summary of results from comparing .Rout and .Rout.want
.Rt.R:
@echo '**' Creating $@ and $@out.want from $<
$(R_PRE) @R_LIBS=$(R_LIBS) $(R_POST) $(R_OPTS) --vanilla --slave --args prepin.R $< $@ $@out.want $<.save < RunScripTestsScript
# Preserve intermediate .R and .Rout files - the user might want to inspect them
# (the .R file to see exactly what the R commands were, and the .Rout file to
# see what output was actually produced)
.PRECIOUS:
# Diff the output
# Need to have R_LIBS & R round the other way for Windows
.Rout.Rtres:
@echo '**' Creating $@ from $< and $<.want
$(R_PRE) @R_LIBS=$(R_LIBS) $(R_POST) $(R_OPTS) --vanilla --slave --args diffout.R $< $(@:Rtres=Rt.save) $@ < RunScripTestsScript
rt-tests: $(allrtres)
# Method for using a 'make' sub process to run tests (will use a chained rule
# for X.Rout: X.Rt -> X.R -> X.Rout if we put the rule .Rt.R in Makevars
# (needs to go there because this Makefile is not read by the sub-make).
rt-tests2:
echo Making target rt-tests
(out=`echo *.Rt | sed 's/\.Rt\( \|$\)/.Rout /g'`; \
if test -n "$${out}"; then \
$(MAKE) -f $(R_SHARE_DIR)/make/$(RSHAREMAKEFILE) $(makevars) $${out}; \
fi)
ScripTests.summary: $(allrtres)
@echo '**' Creating summary of tests in $@
$(R_PRE) @R_LIBS=$(R_LIBS) $(R_POST) $(R_OPTS) --vanilla --slave --args summary.R $@ $(allrtres) < RunScripTestsScript
@(if [ "$(ScripTestsErrorAction)" = stop -a -f ScripTests.haserrors ] ; then \
echo "Stopping because tests had errors and ScripTestsErrorAction=stop in Makefile" ; \
exit 1 ; \
fi)
|
Makefile.win:
1 2 3 4 5 6 7 8 9 | RSHAREMAKEFILE=wintests.mk
# R_SHARE_DIR is defined when running under Linux, but not Windows...
R_SHARE_DIR=$(R_HOME)/share
MAINTESTMAKE=Makefile.win
# Under Windows we want to run R like this: $(R) R_LIBS=$(R_LIBS)
# Seems that don't need this anymore, at least as of R 2.6.1
# R_PRE=$(R)
# R_POST=
include Makefile
|
allrt.Rin:
1 |
This framework requires two files in the tests directory:
Makefile
and Makefile.win
. Note that some of the code in
here is unnecessary (e.g., the R\_PRE and R\_POST stuff – this was used
to put VAR=VALUE before or after the name of the R program on a command
line depending on whether Windows or Unix was being used.
Makefile:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 | # Redefine the default goal (a GNU make feature) so that we
# have the Rt tests as subgoals.
# Note that the default goal can contain only one target.
.DEFAULT_GOAL := all+Rt
ifeq ($(strip $(RSHAREMAKEFILE)),)
RSHAREMAKEFILE=tests.mk
endif
# Define ScripTestsErrorAction to be 'continue' or 'stop'.
# This controls what happens running under R CMD check
# when there are errors in one of the Rt tests.
ScripTestsErrorAction=stop
# Under Unix-likes we want to run R like this:
# @R_LIBS=$(R_LIBS) $(R) ... arguments ...
# while under Windows we want to run R like this:
# $(R) @R_LIBS=$(R_LIBS) ... arguments ...
# So, always run R like this: $(R_PRE) @R_LIBS=$(R_LIBS) $(R_POST)
# with appropriate definitions for R_PRE and R_POST.
# If we are running under Windows, we will have already executed
# Makefile.win, which will have defined R_PRE=$(R) .
ifeq ($(strip $(R_PRE)),)
# Unix-like
R_PRE=
R_POST=$(R)
endif
# If we do want to change the RDIFF command, should include
# 'changeRdiff' as the second dependency for all+Rt
all+Rt: createScripts all rt-tests ScripTests.summary
allrt := $(wildcard *.Rt)
allrtwant := $(allrt:.Rt=.Rout.want)
allrtout := $(allrt:.Rt=.Rout)
allrtres := $(allrt:.Rt=.Rtres)
# allrtin := $(allrt:.Rt=.R)
# Don't need this unless we want standard .R/.Rout.save tests
# to use the modified RDIFF (not the ones run here - they use
# the commands in the .Rout.Rtres rule)
changeRdiff:
@echo Changing Rdiff in Makevars
echo RDIFF = echo hehehe >> Makevars
# Create scripts we will need (do this here to minimize the
# number of files needed to support scriptests in a package.)
createScripts:
@( echo '## Run a script in the scriptests/scripts (from scriptests/inst/scripts)' ; \
echo '## This file does not have .R suffix because if it did, R CMD check would want to run it as a test' ; \
echo 'library(package="scriptests", char=TRUE)' ; \
echo 'args <- commandArgs(TRUE)' ; \
echo 'debug <- is.element("--debug", args)' ; \
echo 'if (length(args)>0) {' ; \
echo ' script.path <- system.file(package="scriptests", "scripts")' ; \
echo ' if (debug) {' ; \
echo ' cat("RtTestScript called with ", length(args), " args: ", paste("\"", args, "\"", collapse=", ", sep=""), "\n", sep="")' ; \
echo ' cat("Looking for scripts in \"", script.path, "\"\n", sep="")' ; \
echo ' }' ; \
echo ' source(file.path(script.path, args[1]))' ; \
echo '}' ) > RunScripTestsScript
# Don't need the following because we can run and check the tests
# with out needing a sub-make. If we do use a sub-make, then we
# need to put rules used in 'Makevars'.
# echo .SUFFIXES: .R .Rin .Rout .Rt >> Makevars
# echo .Rt.R: >> Makevars
# echo ' echo Creating $$@ from $$<' >> Makevars
# echo ' sed -n "s/^[>+] //p" $$< > $$@' >> Makevars
# echo ' cat $$< > $$@out.save' >> Makevars
# This suffix list and rule for .Rt.R needs to go in Makevars
.SUFFIXES: .R .Rin .Rout .Rt .Rtres
# Files:
# .Rt: a transcript, with both commands and desired output, & possibly directives
# .R: R commands generated from .Rt
# .Rout.want: desired transcript output generated from .Rt (maybe don't need this)
# .Rt.save: the processed Rt script as an R object
# .Rout: the output from R when given the commands in .R
# .Rtres: summary of results from comparing .Rout and .Rout.want
.Rt.R:
@echo '**' Creating $@ and $@out.want from $<
$(R_PRE) @R_LIBS=$(R_LIBS) $(R_POST) $(R_OPTS) --vanilla --slave --args prepin.R $< $@ $@out.want $<.save < RunScripTestsScript
# Preserve intermediate .R and .Rout files - the user might want to inspect them
# (the .R file to see exactly what the R commands were, and the .Rout file to
# see what output was actually produced)
.PRECIOUS:
# Diff the output
# Need to have R_LIBS & R round the other way for Windows
.Rout.Rtres:
@echo '**' Creating $@ from $< and $<.want
$(R_PRE) @R_LIBS=$(R_LIBS) $(R_POST) $(R_OPTS) --vanilla --slave --args diffout.R $< $(@:Rtres=Rt.save) $@ < RunScripTestsScript
rt-tests: $(allrtres)
# Method for using a 'make' sub process to run tests (will use a chained rule
# for X.Rout: X.Rt -> X.R -> X.Rout if we put the rule .Rt.R in Makevars
# (needs to go there because this Makefile is not read by the sub-make).
rt-tests2:
echo Making target rt-tests
(out=`echo *.Rt | sed 's/\.Rt\( \|$\)/.Rout /g'`; \
if test -n "$${out}"; then \
$(MAKE) -f $(R_SHARE_DIR)/make/$(RSHAREMAKEFILE) $(makevars) $${out}; \
fi)
ScripTests.summary: $(allrtres)
@echo '**' Creating summary of tests in $@
$(R_PRE) @R_LIBS=$(R_LIBS) $(R_POST) $(R_OPTS) --vanilla --slave --args summary.R $@ $(allrtres) < RunScripTestsScript
@(if [ "$(ScripTestsErrorAction)" = stop -a -f ScripTests.haserrors ] ; then \
echo "Stopping because tests had errors and ScripTests ErrorAction=stop in Makefile" ; \
exit 1 ; \
fi)
|
Makefile.win:
1 2 3 4 5 | RSHAREMAKEFILE=wintests.mk
# Under Windows we want to run R like this: $(R) @R_LIBS=$(R_LIBS)
R_PRE=$(R)
R_POST=
include Makefile
|
Add the following code to your website.
For more information on customizing the embed code, read Embedding Snippets.