[SystemSafety] a public beta phase ???

[jean-louis Boulanger]

ISO 26262 is nor required and in fact for electronics no regulation exist
...
just for motor control, the EURO NCAP introduced some requests.
in another hand some regulation appear for specific equipment such parking
break (see VDA documents).

but the legacy is not clear .... and it's different from china, europe,
.... and US market


2016-07-19 18:12 GMT+02:00 David Ward <david.ward at horiba-mira.com>:

> Mike
>
> Re point 3, ISO 26262 is not a European regulatory requirement, while Regs
> 13 and 79 include a requirement for "complex electronic systems" which is
> in effect a high level demonstration of the functional safety of a function
> like ACC to the type approval authority, the regulation does not require
> ISO 26262 be followed (we are aware some but not all type approval
> authorities may require it, but the regulation doesn't).
>
> David
> ________________________________________
> From: systemsafety [systemsafety-bounces at lists.techfak.uni-bielefeld.de]
> on behalf of Mike Ellims [michael.ellims at tesco.net]
> Sent: 19 July 2016 16:09
> To: 'Les Chambers'; 'Peter Bernard Ladkin';
> systemsafety at lists.techfak.uni-bielefeld.de
> Subject: Re: [SystemSafety] a public beta phase ???
>
> Hi Les,
>
> First a correction it's been  I used the word " connivance" rather than "
> convenience". Oh poop.
>
> A couple of inconstancies here,
>
> 1. the Daily Mail article states that " Brown narrowly avoided a very
> similar smash earlier this year, when his car did not notice a white truck
> turning in on him on the freeway. Pictured, the truck cutting him up in
> April". However brown posted this video on facebook and crowed about how
> the
> Tesla did avoid the accident. He didn't have to intervene.
>
> 2. The photo at
> http://www.autospies.com/images/users/Agent009/main/1naias-17.jpg proves
> nothing; if you use google search you find it takes you to sites such as
>
> http://windsorstar.com/business/electric-tesla-model-x-to-compete-with-premi
> um-suvs-minivans?__lsa=4182-2da6 where the car is stationary and it shows
> it's clearly a publicity shot taken on a showroom floor. The question is
> whether the car allows you to do this while moving. Tesla say no and in the
> US I believe it would be illegal for it to do so and Tesla should have
> known
> that as their former head a regulatory compliance came from an established
> player.
>
> 3. The timeline given doesn't match with other's I've see which state the
> accident occurred 7 May and Tesla provided NHTSA with data logs in the 8th.
> I'm not sure who is right, the NTHSA letter doesn't provide the necessary
> information but does confirm that Tesla instigated the investigation i.e.
> "The May 7, 2016 fatal crash that was reported to ODI by Tesla".
>
> However trying to track that down I came across the following:
>
> http://www.digitaltrends.com/cars/tesla-autopilot-examined-dutch-regulators/
>
> Which suggests that Tesla obtained type approval in the EU which would
> usually imply conformance to ISO 26262. If that is the case then there
> should be a set of associated documents as you suggested i.e. requirements,
> hazard analysis, test plans, validation reviews etc. Note however just as
> with the FAA there will (or should be) too much material to audit
> completely and what the regulator (or any auditor) will often do is check
> the process and do deep dives where they think necessary.
>
> I'm not sure about Mary Cummings' comments - we don't have any visibility
> of
> what Tesla, or anyone else are up to and in America neither do the
> regulators, in the US automotive companies self certify against regulations
> and the regulators only get involved if there is a problem. I'm sure prof,
> Cummings' has no more visibility than we do. As a general set of concerns
> she is however correct. However as stated above the Dutch certification
> authority should have had the visibility.
>
>
> -----Original Message-----
> From: Les Chambers [mailto:les at chambers.com.au]
> Sent: 19 July 2016 13:24
> To: 'Les Chambers'; 'Mike Ellims'; 'Peter Bernard Ladkin';
> systemsafety at lists.techfak.uni-bielefeld.de
> Subject: RE: [SystemSafety] a public beta phase ???
>
> More reading on the Tesla issue. It speaks for itself.
>
>
> http://www.teslaosterone.com/uncategorized/teslas-fatal-crash-6-unanswered-q
> uestions/
>
> Duke University robotics professor Mary Cummings testified at a U.S. Senate
> hearing in March that the self-driving car community is “woefully
> deficient” in its testing programs, “at least in the dissemination of
> their test plans and data.”
>
> In particular, she said she’s concerned about the lack of “principled,
> evidenced-based tests and evaluations.”
> -----------------
>
> http://www.dailymail.co.uk/news/article-3671265/PICTURED-Ex-Navy-SEAL-died-s
> elf-driving-car-crashed-truck-seen-showing-no-hands-autopilot.html
>
> Joshua Brown, 40, was behind wheel of self-driving Tesla Model S car. He
> was
> killed when the vehicle collided with a trailer truck in Florida in May.
> Trucker claims Brown was watching a Harry Potter movie when he crashed but
> Tesla says it is impossible to watch movies on its touchscreen.
> My Comment: Is it really ? Refer:
> http://www.autospies.com/images/users/Agent009/main/1naias-17.jpg
>
> Elon Musk, the billionaire CEO of Tesla Motors, tweeted: 'Our condolences
> for the tragic loss.' [of Joshua Brown] My comment: is that all Brown's
> death rated? A tweet? I hope not.
>
> -----------------
>
> http://www.eetimes.com/document.asp?doc_id=1329257
> Why Autonomous Cars Are ‘Absolutely Not Ready’
> 7 limitations in tech, tests & evaluations
>
> The crux of the issue, according to Cummings, is twofold: There is a lack
> of
> “principled, evidenced-based tests and evaluations” for autonomous cars,
> and little leadership from federal regulators to create “a clear
> certification process” for self-driving cars.
>
> To make matters worse, little information about automakers’ test plans and
> data is available today for experts to measure the performance of
> self-driving cars. Put more bluntly, today’s self-driving community is
> substituting demonstrations for rigorous testing, in Cummings’ opinion.
>
> In her testimony, she laid out seven “limitations” in current self-driving
> car technologies 1. Lack of evidence-based tests 2. Bad weather conditions
> 3. Vulnerability to malevolent or prankster intent 4. Gaming the
> self-driving car (For example, “A $60 laser device can trick self-driving
> cars  into seeing  objects  that  aren’t  there,” she said.) 5. What about
> privacy and control of personal data?
> 6.  Regulatory agencies’ roles
> (Cummings pointed out that “The U.S. government cannot and has not
> maintained sufficient staffing in the number of people it needs who can
> understand, much less manage, complex systems such as self‐driving cars.”)
> 7. Accumulating miles is no assurance for safety
>
> What matters isn’t exactly the number of hours and miles the designed
> system logged in in-house testing. It’s how the tests are designed, who
> validates the results, whether procedures are peer-reviewed, and if a clear
> certification process exists.
>
> My comment:
> My sense is that Mary Cummings' heart is in the right place but she is
> missing quite a bit here (I'm assuming this is all she said, I may be
> wrong).
> Testing is definitely important but if you've got a hacked up system no
> amount of testing will make it safe.
> I would add:
> 1. Who wrote the system requirements and where they complete and correct as
> they could be (did anyone write a requirement spec???) 2. Where the
> requirements validated (were they validatable?) 3. What formalisms were
> used
> to design and build these systems. What architectural patterns were used,
> what formal models were used?
> 4. How observable was the software behaviour for the purposes of testing?
> 5. Is the code instrumented. How thorough is the error logging?
> 6. Who wrote the code? How experienced were they? How much code review
> occurred? How experienced were the code reviewers?
> 7. What was the defect density in first-round unit, integration and system
> testing?
>
> Elon Musk is doing a great job in making artificial intelligence open. He
> has donated 10,000,000 dollars to the open AI foundation. He is also
> opening
> his Tesla designs to the public. The code is not open for very good
> security
> reasons. So the gaming starts. Refer:
>
> http://www.digitaltrends.com/cars/syscan-announces-10000-prize-hacking-tesla
> /
> $10,000 bounty on Model S hacks entices tinkerers, aggravates Tesla
>
> Interesting times!
>
> Les
>
>
>
>
> -----Original Message-----
> From: systemsafety [mailto:
> systemsafety-bounces at lists.techfak.uni-bielefeld.
> de] On Behalf Of Les Chambers
> Sent: Tuesday, July 19, 2016 12:51 PM
> To: 'Mike Ellims'; 'Peter Bernard Ladkin';
> systemsafety at lists.techfak.uni-bielefeld.de
> Subject: Re: [SystemSafety] a public beta phase ???
>
> Mike
>
> >Do you any evidence to back up that claim?
> There are many videos on the web depicting drivers grabbing the wheel at
> the
> last moment when the lane marker sensors fail. There was a video depicting
> a
> man sitting in the back seat of his car. I can't find it though.
> My point is that all automation systems have a mission. If they can't
> perform that mission there must be a smooth hand off to manual control. Any
> automation that requires split-second hand off (or you die) is a dangerous
> object and should not be in the hands of untrained operators. Formula one
> drivers maybe.
>
> > But we'd never know because a) the data hasn't been collected on a
> > routine
> basis
>
> There will never be enough data. If we wait around for it to appear more
> people will die. What we need to do is apply judgement. The judgement the
> engineering profession has accumulated over the last 50 years of computer
> automation design. What we need is lead indicators. Accident data is a lag
> indicator. Dead bodies are lag indicators. They are bad indicators.
> Judgement is a lead indicator. When we see an organisation violating the
> fundamental rules of safe control system design we speak out VERY LOUDLY.
>
> We can't predict the future so because where human we make up narratives to
> convince ourselves everything will be okay. That's what this journalist has
> done. He says it's okay 33,000 people are killed every year. That's normal
> don't worry. He is contributing to what has become a popular narrative,
> community belief, pathological groupthink.
> My narrative is: It's NOT okay if one single death is a result of bad
> design. It's NOT okay if someone scratches their finger due to bad design.
> This is the engineering discipline we admire. This is what sets us apart
> from the un-engineer. Tesla's operational concept of "be careful but ... Oh
> quick grab the wheel" is bad design. We've known for years it's bad design.
> It needs to stop. OK!
>
> Re: software upgrades. I don't care if it's one line or 10,000 lines. You
> can introduce a bug with one line that can bring a system down. One of my
> engineers once nearly destroyed 10,000,000 dollar chemical reactor with a
> single bad configuration word in memory. Proven in use arguments are not
> relevant to software. It is just too fluid. We should stop propagating this
> BS. Another bad unsafe narrative.
>
> And as for maintaining the rage, I'm sorry that's the only way human beings
> will change course - through some emotional connection to a better idea,
> often triggered by horrendous events like the unnecessary death of a human
> being.
>
> Blow, winds, and crack your cheeks! rage! blow!
> You cataracts and hurricanoes, spout
> Till you have drench'd our steeples, drown'd the cocks!
> You sulphurous and thought-executing fires, Vaunt-couriers to oak-cleaving
> thunderbolts, Singe my white head! And thou, all-shaking thunder, Smite
> flat
> the thick rotundity o' the world!
> Crack nature's moulds, an germens spill at once, That make ingrateful man!
>
> It took rage over the death of 30 children in the Armagh rail disaster to
> mobilise a government to enforce safety rules in the rail industry. We
> should know better these days. How to manage technology churn that is. What
> we don't need is engineers contributing to the wrong narrative.
>
> ONE DEATH IS NOT OKAY. DO YOU HEAR ME SON!
>
> Please adjust your narrative accordingly.
>
> Les
>
> -----Original Message-----
> From: Mike Ellims [mailto:michael.ellims at tesco.net]
> Sent: Tuesday, July 19, 2016 12:32 AM
> To: 'Les Chambers'; 'Peter Bernard Ladkin';
> systemsafety at lists.techfak.uni-bielefeld.de
> Subject: RE: [SystemSafety] a public beta phase ???
>
> Good afternoon Les,
>
> > The argument that 33,000 people are killed in accidents every year, so
> > why
> should we care, is also
> > drivel. None of these fatalities occurred because a driver trusted a
> system that couldn't be trusted.
>
> Do you any evidence to back up that claim? For example can you show that no
> one was harmed because a lane keep feature failed to keep a vehicle in lane
> or that no one was killed because they became over reliant of features such
> as emergency brake assist or any one of the dozen or so driver assist
> systems that the current crop of vehicles have (see video below)? I know
> it's nearly impossible to prove a negative but I'm also pretty sure that if
> we had access to all the relevant data we'd probably find someone,
> somewhere
> who rolled their vehicle because of overconfidence in their ESC system. But
> we'd never know because a) the data hasn't been collected on a routine
> basis
> and b) vehicle manufactures don't in general have an mechanism to collect
> such data. At least in this regard Tesla seems to be ahead of the game and
> I
> suspect that if Tesla hadn't asked the NHTSA to investigate it's possible
> we
> may never had found out about this either.
>
> The problem here is that absence of evidence isn't the same evidence of
> absence. Current systems may have issues we (or anyone) know about and I
> know of at least one failure mechanism for ABS that would fool most
> implementations from 10 years back (it may not now - I don't know) but ABS
> (a driver aid) is mandatory in Europe as is ESC because statistically it's
> believed to save lives. Neither provides a guarantee of either safety or
> that the system will work all the time; but these days I wouldn't buy a car
> without them.
>
> The following video demonstrates some of the benefits of standard systems
> (ABS, ESC,TC) https://www.youtube.com/watch?v=wR1SSxpKitE
>
> And I found this which looks at AEB (autonomous emergency braking) and some
> success and some failures.
> https://www.youtube.com/watch?v=E_ZNG8cmnlw
>
> Note the driver involved in the tests was British Touring Car champion and
> has held an F1 super licence so he actually does know how to drive.
>
>
> > The media needs to maintain the rage and keep reporting self driving
> > car
> fatalities.
>
> Rage is the wrong concept. Rather the media needs to track the issue and
> report it in some sort of sane balanced manner, or at least as sane and
> balanced as the media seem to be able to muster these days. We should do
> likewise.
>
>
> > Every time you receive a software upgrade in your garage the safety
> > claims
> made on your current version
> > minus one are null and void. The game starts over again. You drive out
> > the
> gate. You roll the dice.
> > Thousands of lines of code have been changed, the potential for
> > screwups
> is high, exacerbated by the massive
> > complexity of these AI fuelled applications.
>
> Only the first and last sentences here are strictly correct, the rest to
> some extent is exaggerated.
>
> First for the driver is not necessarily the case the game starts completely
> from zero. Tesla have stated that before release they have a lab test
> program and they test the software in their own test fleet. I have no idea
> what this comprises in detail or how much testing is done but at the level
> we have visible information that is reasonably comparable with normal
> practice in automotive.
>
> Likewise to state that 1000's lines have been changed is hyperbole - we
> have
> no visibility of what was changed. It might be 1000's of lines, it might be
> one (same effect) or it may be zero, for example if it's an update to a
> neural network then is could be zero code lines.
>
>
> > exacerbated by the massive complexity of these AI fuelled applications.
>
> To my mind that is the one point really on the money - how do you cope with
> the complexity of AI based systems? You could potentially formally prove
> the
> inference engines and perhaps much of the rest of the system but has been
> noted elsewhere the specification of say a neural network or other
> stochastically based system is a combination of the size and completeness
> of
> the training AND test sets.
>
> Questions that need to be addressed are
> - how big is big enough for either set?
> - how do you quantify diversity in either set?
> - can you improve on a simple division of training/tests or are there ways
> to use one test set in both ways, e.g. develop the test set from the
> training set by changing vehicle colours/background etc.
>
> > And furthermore, it takes years for an organisation to develop an
> effective safety culture, matter cannot move
> > faster than the speed of light nor can Tesla develop a culture that
> > would
> rival that of NASA or the aircraft
> > industries in the short time they've been in business.
>
> Neither of the two examples has an unblemished record as regards safety,
> NASA have overseen the loss of several man carrying spacecraft i.e. Apollo
> 1
> and two shuttles. Recently the aviation industry has given us battery fires
> in the 777 and what could only be described as an imaginary safety case on
> behalf of the Nimrod. Experience does not necessarily equate to safety it's
> something that has to be continually worked on. Whether or not Tesla have
> it
> I'm not sure we can say at this point in time.
>
> Some of the things that do annoy me about reporting and discussion of this
> event are:
>
> 1. The Tesla in question appears to not have noticed it was involved in a
> crash and to have continued down the road without its roof until it ran off
> into a field. That doesn't seem to be an appropriate reaction to an
> accident
> and potentially it's worse than the original accident.
>
> 2. As is usual in the USA (but not Europe) the truck was not fitted with
> side impact bars so a) to the Tesla's radar the road appeared clear (it
> apparently classified the side of the vehicle as a road sign) and b) if the
> vehicle had run into the bars the impact would possibly have been less bad.
> It has been commented elsewhere that that fitting of under-run bars would
> prevent 250 deaths a year in the US but after decades of trying it's still
> not mandatory.
>
>
>
> -----Original Message-----
> From: systemsafety
> [mailto:systemsafety-bounces at lists.techfak.uni-bielefeld.de] On Behalf Of
> Les Chambers
> Sent: 18 July 2016 12:15
> To: 'Peter Bernard Ladkin'; systemsafety at lists.techfak.uni-bielefeld.de
> Subject: Re: [SystemSafety] a public beta phase ???
>
> PBL
> John Naughton's article in the Guardian is not sensible, it is uninformed,
> illogical and flat out wrong. Obviously written by a person has never had
> the experience of writing code that could kill someone; never had to take a
> community of untrained operators and put a highly automated system in their
> hands from a starting position of total ignorance.
> Telling a driver to behave responsibly and keep their hands on the wheel is
> a bit like telling a gambler to gamble responsibly. If a car can drive
> itself, untrained drivers will take advantage of this feature and put too
> much trust in what is currently an untrustworthy system. Drivers put
> through
> focused training will take these warnings seriously. Your average bunny who
> can afford a Tesla but has had no training will not.
> The argument that 33,000 people are killed in accidents every year, so why
> should we care, is also drivel. None of these fatalities occurred because a
> driver trusted a system that couldn't be trusted.
> And lastly RE: Naughton's comment that "... mainstream media will have to
> change the way they report self driving cars. Every time a Tesla or a
> Google
> car is involved in a crash, by all means report it. But also report all the
> human error crashes that occurred on the same day." Not so. The media needs
> to maintain the rage and keep reporting self driving car fatalities. This
> is
> probably the only way we will get the message through to the general public
> that if you buy one of these cars you are taking a substantial risk. Every
> time you receive a software upgrade in your garage the safety claims made
> on
> your current version minus one are null and void. The game starts over
> again. You drive out the gate. You roll the dice. Thousands of lines of
> code
> have been changed, the potential for screwups is high, exacerbated by the
> massive complexity of these AI fuelled applications. This is the new
> normal,
> we are now beta testing safety critical systems on the public. PBL you
> might
> as well put a clause in 61508 two okay this behaviour.
> And furthermore, it takes years for an organisation to develop an effective
> safety culture, matter cannot move faster than the speed of light nor can
> Tesla develop a culture that would rival that of NASA or the aircraft
> industries in the short time they've been in business. System safety has
> one
> source: motivated people and it takes years to develop that motivation.
> They and we will get there eventually but in the meantime the public has a
> right to be made aware of the risks they are taking with these vehicles.
>
> Les
>
> -----Original Message-----
> From: systemsafety
> [mailto:systemsafety-bounces at lists.techfak.uni-bielefeld.de] On Behalf Of
> Peter Bernard Ladkin
> Sent: Sunday, July 17, 2016 7:29 PM
> To: systemsafety at lists.techfak.uni-bielefeld.de
> Subject: Re: [SystemSafety] a public beta phase ???
>
> A very sensible comment from John Naughton today in The Observer
>
>
> https://www.theguardian.com/commentisfree/2016/jul/17/self-driving-car-crash
> -proves-nothing-tesla-autopilot
>
> PBL
>
> Prof. Peter Bernard Ladkin, Bielefeld, Germany MoreInCommon Je suis Charlie
> Tel+msg +49 (0)521 880 7319  www.rvs-bi.de
>
>
>
>
>
>
>
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
>
>
> ---
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus
>
>
>
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
>
>
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
>
> --
> HORIBA MIRA Ltd
>
> Watling Street, Nuneaton, Warwickshire, CV10 0TU, England
> Registered in England and Wales No. 9626352
> VAT Registration  GB 100 1464 84
>
> This email and any files transmitted with it are confidential and intended
> solely for the use of the individual or entity to whom they are addressed.
> If you are not the named addressee you should not disseminate, distribute
> or copy this e-mail. Please notify the sender immediately by e-mail if you
> have received this e-mail by mistake and delete this e-mail from your
> system. If you are not the intended recipient you are notified that
> disclosing, copying, distributing or taking any action in reliance on the
> contents of this information is strictly prohibited.
>
> _______________________________________________
> The System Safety Mailing List
> systemsafety at TechFak.Uni-Bielefeld.DE
>



-- 
Mr Jean-louis Boulanger
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20160719/c9374f2d/attachment-0001.html>