<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<p><br>
</p>
<div class="moz-cite-prefix">On 04/07/2020 22:10, Martyn Thomas
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:cd9f5d1b-102a-a875-2bf6-c14a669b0146@thomas-associates.co.uk">
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<p>"What do you do when your system <b>fails</b> a UT?".</p>
</blockquote>
<p>I check for errors in the code *then in the tooling system, then
backwards through the process following the traceability trail
(assuming it hasn't been eaten by the creatures in the forest)*.<br>
</p>
<p>My reason for this is that part of my professional background is
in compiler testing for which I have held registration as a UKAS
(successor to NAMAS) assessor. Under the old NAMAS standard an
assessor is required to check test systems for influence factors
(a term used in metrology) that could affect the results of
testing or measurement. Among these factors are defects in the
testing and measurement systems used.</p>
<p>Owing to this concern, I have produced designs for compiler test
control systems that run self-check routines after each run of a
program in a compiler validation suite. Among other things, such
self-tests serve the purpose of guarding against the test control
system, among other things:</p>
<p>(1) failing to satisfy assumptions upon which the integrity of
testing relies,<br>
</p>
<p>(2) falling into a state such that the integrity of testing is
compromised,</p>
<p>(3) being unable to re-establish for rerun, the test conditions
under which a test may have failed,<br>
</p>
<p>(4) failing adequately to control other influence factors within
its control that might, if uncontrolled, render the results of
testing untrustworthy.</p>
<p>I am not aware of any current compiler validation suite that
comes with a test control system that seeks to achieve this degree
of control. Few compiler test suite vendors seem to take much
notice of the relevant international standard, ISO/IEC 17025:2017
General requirements for the competence of testing and calibration
laboratories (which UKAS now uses instead of the old NAMAS
standard of similar title). Indeed, I have actually wanted to
abandon some attempts at compiler validation because it proved
impossible to set up an adequately self-checking test control
system.</p>
<p>What price CbyC reliance on compilers that may not have been
constructed using CbyC and may not have been tested under
adequately controlled conditions?<br>
</p>
<p>(Takes deep breath and waits for Derek Jones, who AFAI am aware
has never been a UKAS assessor, to hint, not for the first time,
that I'm anally obsessive about compiler validation.)<br>
</p>
<p><br>
</p>
<p>Counter question:</p>
<p>If you are using CbyC and a system unit fails under test at a
stage of testing later than UT, what do you do?</p>
<p><br>
</p>
<p>Olwen</p>
<p><br>
</p>
<p><br>
</p>
<p><br>
</p>
<br>
</body>
</html>