[Nobug] coverage testing, are all failures covered?

[Christian Thaeter]

Just a bit brainstorming for developers:

NoBug has the fault-injection features, we don't use it yet for Lumiera
testing, I am thinking about making it little more useful/automatic.

The Plan:
 - load an (initially empty) log which contains backtraces of
   already tested fault-injection points
 - at an fault injection point, check if this point was already tested
   - if not, then log it and inject a fault
   - if yes, do nothing
 - rinse repeat with the new log .. towards disovering every possibly
   way leading to a fault injection point


Details/Thoughts:
logging backtraces can be done in a raw way using the glibc bactrace
function. Just a CSV list of addresses would suffice.

Parsing log in builds a tree for each discovered code path.

Probably this needs an second pseudo log which defines an exclusion list
of bactraces you know which need no error checking and can not fail.

Its not as trivial as thought on a second view, because when we trigger
a fault the error handling might visit more fault injection points
(nasty errors while handling errors ..). Code  flow is a DAG, not
linear. Luckily this can be handled by the test harness (some script to
be written) maybe bactraces above need some kind of tag if they are
sufficiently disovered.

Possible even moderate testcases can already visit many many
fault-injection points, since we only care about unique paths this
should reduce the number of test runs significantly but anyways this is
rather heavy testing which may iterate a lot times on each test.

Parameters passed into functions are not considered when injecting a
fault, this may produce a lot false positives, see pseudo-log for
visited paths above, maybe we need a facility to log exclusions dynamically.

In presence of threads faults can occur asynchronously, this is at least
a minor problem, can this be a big problem for testing things?



	Christian