[Nobug] Lumiera Trac #645: planned test.sh improvments

[Lumiera Trac]

#645: planned test.sh improvments
---------------------+------------------------------------------------------
 Reporter:  ct       |       Owner:           
     Type:  planned  |      Status:  new      
 Priority:  normal   |   Milestone:           
Component:  lumiera  |    Keywords:  test bash
 Blocking:           |   Blockedby:           
---------------------+------------------------------------------------------
 After some discussion on IRC I introduce some needed/planned improvements
 for tests.sh here. Implementing these needs moderate bash skills, anyone
 who want to help the project may try it, otherwise I'll do it when I have
 some spare time for it.

 Refactor and add more flags for output specifiers:

 Currently we have hardcoded output specifiers out, out-lit, err and err-
 lit. This should be rewritten so that the first word gives the stdio
 fdname (out or err) and any following word delimited by - acts as a flag.
 Then some shell variables should provide defaults and overrideable in
 tests.conf and testsuites themself. Defaults should stay as it currently
 is (*-rx-trig) for backwards compatibility.

 Following flags have to be supported:

  * -lit the following specification string has to be taken literally
  * -rx the following specification string is a regex

 Note that these are mutally exclusive and either one has to be provided
 (by defaults)

  * -trig triggering behaviour, after one fail the next regex is tried, if
 that fails again then the test fails.
  * -once the specification must match exactly one time
 This 2 are mutally exclusive too same as above.

  * -not inverses the match

 (do we need any flags for the 'in' fd? be prepared at least)


 Diverting output based on regex matches:

 Currently test.sh expects outputs in very well defined order, this is a
 little problem when testing asynchronous multithreaded programs, so we
 introduce a new syntax to split output specifications into seraveral
 'channels'. This is based on a parentized regex just before the delimiting
 ': ', the old syntax without this parens then takes any outputs not
 diverted into this channels. I expect that implemeneting this feature is a
 bit challenging.

 Some examples:
 {{{
 out-rx(thread_1:): Matches this on any line also containing thread_1:
 err-not: error
 }}}



 Include some stdout context into the testlog in case of an fail

 Currently only stderr is included in testlog (completely) but data on
 stdout is left out. In case of an fail we want some last lines from stdout
 as well, since stdout is expected to be volumious and usually relatively
 boring this should be limited to some context around the detected error
 (+/-10 lines)


 Better explanation and reports

 Fails don't produce much insight on why they failed yet, we should keep a
 log with more details (what actually matched and what failed) and include
 the last few things in the testlog in case of an failure.


 Hashing tests

 currently tests are solely identified by their textual descriptor. This is
 used by the FAST and FIRSTFAIL testmodes. This is error prone in case of
 non-unique descriptions. Better hash the whole test-call and stuff that
 into a logfile which is then grepped for previous test results.


 better abort on timeout

 There are some glitches on the currently quite hacked code aborting a test
 after a timeout, figure that out and fix it.

-- 
Ticket URL: <http://issues.lumiera.org/ticket/645>
Lumiera Issues Trac <http://lumiera.org>
Lumiera -- the new emerging NLE for GNU/Linux