[SystemSafety] Fwd: Re: CbyC and unit testing
Olwen Morgan
olwen at phaedsys.com
Sun Jul 5 12:00:19 CEST 2020
On 04/07/2020 22:10, Martyn Thomas wrote:
>
> "What do you do when your system *fails* a UT?".
>
I check for errors in the code *then in the tooling system, then
backwards through the process following the traceability trail (assuming
it hasn't been eaten by the creatures in the forest)*.
My reason for this is that part of my professional background is in
compiler testing for which I have held registration as a UKAS (successor
to NAMAS) assessor. Under the old NAMAS standard an assessor is required
to check test systems for influence factors (a term used in metrology)
that could affect the results of testing or measurement. Among these
factors are defects in the testing and measurement systems used.
Owing to this concern, I have produced designs for compiler test control
systems that run self-check routines after each run of a program in a
compiler validation suite. Among other things, such self-tests serve the
purpose of guarding against the test control system, among other things:
(1) failing to satisfy assumptions upon which the integrity of testing
relies,
(2) falling into a state such that the integrity of testing is compromised,
(3) being unable to re-establish for rerun, the test conditions under
which a test may have failed,
(4) failing adequately to control other influence factors within its
control that might, if uncontrolled, render the results of testing
untrustworthy.
I am not aware of any current compiler validation suite that comes with
a test control system that seeks to achieve this degree of control. Few
compiler test suite vendors seem to take much notice of the relevant
international standard, ISO/IEC 17025:2017 General requirements for the
competence of testing and calibration laboratories (which UKAS now uses
instead of the old NAMAS standard of similar title). Indeed, I have
actually wanted to abandon some attempts at compiler validation because
it proved impossible to set up an adequately self-checking test control
system.
What price CbyC reliance on compilers that may not have been constructed
using CbyC and may not have been tested under adequately controlled
conditions?
(Takes deep breath and waits for Derek Jones, who AFAI am aware has
never been a UKAS assessor, to hint, not for the first time, that I'm
anally obsessive about compiler validation.)
Counter question:
If you are using CbyC and a system unit fails under test at a stage of
testing later than UT, what do you do?
Olwen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/pipermail/systemsafety/attachments/20200705/fda1e353/attachment.html>
More information about the systemsafety
mailing list