STF Testing Frameworks
In test.assessr: Assessing Package Test Reliability and Quality

knitr::opts_chunk$set(echo = TRUE)

library(test.assessr)

The test.assessr package checks package structure and test drivers to detect and classify testing configurations.

If it detects a standard testing framework (STF) (i.e. testthat or testit), it runs a standard testing workflow involving a number of steps:

Step 1

run callr::r_safe which calls covr::package_coverage.
The r_safe function has the parameter error = "error".
This means if package_coverage encounters an error, then r_safe returns an error to the main testing process.
If Step 1 is successful, then a unit test data coverage object like this is produced:

Step 1 unit test data coverage object

| R object | field | element 1 | element 2 | value | | -------- | ----- | --------- | --------- | ------| | test_package |pkg_name | | | "test.package.0001" | | | pkg_ver | | | "0.1.0" | | | date_time | | | "2026-03-09 14:27:36.673915" | | | executor | | | "xxxxxxx" | | | sysname | | | "Linux" | | | version | | | "#1 SMP PREEMPT_DYNAMIC Mon Dec 1 05:36:50 UTC 2025" | | | release | | | "6.1.158-180.294.amzn2023.x86_64" | | | machine | | | "x86_64" | | | r_version | | |"4.4.1" | | | test_framework_type | | | standard testing framework | | | total_cov | | | 1 | | | res_cov | name| |"test.package.0001" | | | | coverage | filecoverage | R/myscript.R 100 | | | | | totalcoverage | 100 | | | errors | | |NA | | | notes | | |NA |

Step 2

run setup_covr_env which
- loads helper files from tests/testthat,
- loads package namespace objects into cov_env,
- resolves the project root,
- loads package datasets into cov_env,
- loads Base R datasets and exports into cov_env,
- optionally loads nycflights13 datasets if available,
- loads dependencies into cov_env, and
- unloads the package namespace afterward to reduce conflicts.
run create fail reporter to find failing tests
run create test reporter to create a reporter for all tests
map exported functions to test files
create a filter for test status i.e. tests that have "FAILED", "ERROR", "SKIPPED"
identify tests to be skipped based on status
comment out tests to be skipped
run covr::environment_coverage on remaining test files.
If Step 2 is successful, then a unit test data coverage object like this is produced:

Step 2 unit test data coverage object

| R object | field | element 1 | element 2 | value | | -------- | ----- | --------- | --------- | ------| | test_package |pkg_name | | | "test.package.0003" | | | pkg_ver | | | "0.1.0" | | | date_time | | | "2026-04-09 10:17:27.71564" | | | executor | | | "xxxxxxx" | | | sysname | | | "Linux" | | | version | | | "#1 SMP PREEMPT_DYNAMIC Mon Dec 1 05:36:50 UTC 2025" | | | release | | | "6.1.158-180.294.amzn2023.x86_64" | | | machine | | | "x86_64" | | | r_version | | |"4.4.1" | | | test_framework_type | | | standard testing framework | | | total_cov | | | 0 | | | res_cov | name| |"test.package.0003" | | | | coverage | filecoverage | num [1(1d)] 0 | | | | | attr(*, "dimnames")=List of 1 chr "myscript.R" | | | | | totalcoverage | 0 | | | errors | | |NA | | | notes | | |NA | | long_summary | file | | | "test-myscript.R#L2_L2" | | | code_script | | | "myscript" | | | context | | | "this works" | | | status | | | "FAILED" | | | n | | | 1 | | | time | | | 0.291 | | test_skip | file | | | "test-myscript.R" | | | code_script | | | "myscript" | | | context | | | "this works" | | | status | | | "FAILED" | | | n | | | 1 | | | time | | | 0.291 | | | expectation: | | | "expect_equal" | | | line1 | | | 2 | | | line2 | | | 2 |

This data object differs from Step 1 data object in that it gives information about which tests/test blocks were skipped and the status tells the user why they were skipped.

Step 3

run withr::with_dir which calls covr::package_coverage.
covr::package_coverage has the parameters type = "tests", quiet = TRUE
If Step 3 is successful, then a unit test data coverage object like this is produced:

Step 3 unit test data coverage object

| R object | field | element 1 | element 2 | value | | -------- | ----- | --------- | --------- | ------| | test_package |pkg_name | | | "dplyr" | | | pkg_ver | | | "1.1.4" | | | date_time | | | "2026-04-09 10:17:27.71564" | | | executor | | | "xxxxxxx" | | | sysname | | | "Linux" | | | version | | | "#1 SMP PREEMPT_DYNAMIC Mon Dec 1 05:36:50 UTC 2025" | | | release | | | "6.1.158-180.294.amzn2023.x86_64" | | | machine | | | "x86_64" | | | r_version | | |"4.4.1" | | | test_framework_type | | | standard testing framework | | | total_cov | | | 0 | | | res_cov | name| |"dplyr" | | | | coverage | filecoverage | num [1:109(1d)] 98.3 76.7 97.5 100 100 ... | | | | | attr(*, "dimnames")=List of 1 [1:109] "across.R" "all-equal.R" "arrange.R" "bind-cols.R" ... | | | | | totalcoverage | 0.907 | | | errors | | |NA | | | notes | | |NA | | long_summary | file | | | [1:3432] "test-across.R#L7_L7" "test-across.R#L14_L14" "test-across.R#L20_L20" "test-across.R#L24_L24" ... | | | code_script | | | [1:3432] "across" "across" "across" "across" ... | | | context | | | [1:3432] "across() works on one column data.frame" "across() does not select grouping variables" "across() corre" | | | status | | | [1:3432] "PASS" "PASS" "PASS" "PASS" ... | | | n | | | [1:3432] 1 1 1 1 1 1 1 1 1 1 ... | | | time | | | [1:3432] 0.041 0.014 0.077 0.077 0.077 ...| | test_skip | file | | | [1:448] "test-across.R" "test-across.R" "test-across.R" "test-across.R" ... | | | code_script | | | [1:448] "across" "across" "across" "across" ... | | | context | | | "across() gives meaningful messages" "across() gives meaningful messages" "across() gives meaningful mes" | | | status | | | [1:448] "FAILED" "FAILED" "FAILED" "FAILED" ... | | | n | | | [1:448] 1 1 1 1 1 1 1 1 1 1 ... | | | time | | | [1:448] 2.03 2.03 2.03 2.03 2.03 ... | | | expectation| | | [1:448] "expect_snapshot" "expect_error" "expect_error" "expect_error" ... | | | line1 | | | [1:448] 209 212 216 222 226 230 234 238 243 244 ... | | | line2 | | | [1:448] 280 215 219 225 229 233 237 241 243 244 ... |

Both Step 2 unit test data coverage object and Step 3 unit test data coverage object give the user details about the status of the tests FAILED, PASSED and SKIPPED.

They also contain the test block start line (i.e. line1 and line2) and the expectation type in the expectation column (e.g. expect_snapshot expect_error).