README.md

In cloudcell/fuzztest: Fuzz/Stress Testing Environment

fuzztest

(Please read a supplementary description here: https://gtdir.wordpress.com/2016/03/02/fuzz-testing/)

Purpose

To isolate errors through identifying groups of parameter combinations that cause them.

What the Package Does

Tests all possible combinations of argument values for any function to produce test statistics and graphs to help identify and fix defects faster than by using more traditional approaches. The key feature of the package is its ability to present multiple test results in a very compact form.

Alternative Use

Every run automatically creates data for unit testing. Simply compare saved "reference" results against the ongoing development results. For now, one can check them by loading data files with test results and checking whether exit status is identical for all combinations of input parameters. (Timestamped data is saved in a current work directory after every test.)

Roadmap

1. To arrange axes on a parallel plot automatically based on a relative influence of a particular parameter to make the visual analysis faster.
2. To enable automatic comparison and visualization of a current test against saved reference data.
3. To report "incremental" "PASS/FAIL" i.e. only differences between the current test and previously recorded reference data.

Demo Visual Output

Below are a several demo test graphs showing parameter settings leading to function failures (in color). Axes are used only to order and group tests by cardinal_numbers of argument parameters.

PASS Results Only

FAIL Results Only

cloudcell/fuzztest documentation built on May 13, 2019, 8:02 p.m.