[SystemSafety] Elephants, dinosaurs and integrating the VLA model

Fri Aug 4 04:37:56 CEST 2023

For what it’s worth, OWASP (Open Worldwide Application Security Project, www.owasp.org) released their “OWASP Top 10 for LLM”. You can find the document here:

<https://owasp.org/www-project-top-10-for-large-language-model-applications/assets/PDF/OWASP-Top-10-for-LLMs-2023-v1_0.pdf>
[preview.png]
OWASP-Top-10-for-LLMs-2023-v1_0<https://owasp.org/www-project-top-10-for-large-language-model-applications/assets/PDF/OWASP-Top-10-for-LLMs-2023-v1_0.pdf>
PDF Document · 33.6 MB<https://owasp.org/www-project-top-10-for-large-language-model-applications/assets/PDF/OWASP-Top-10-for-LLMs-2023-v1_0.pdf>

Quote (page 2):

“The frenzy of interest of Large Language Models (LLMs) following of mass-market pre- trained chatbots in late 2022 has been remarkable. Businesses, eager to harness the potential of LLMs, are rapidly integrating them into their operations and client facing offerings. Yet, the breakneck speed at which LLMs are being adopted has outpaced the establishment of comprehensive security protocols, leaving many applications vulnerable to high-risk issues.

The absence of a unified resource addressing these security concerns in LLMs was evident. Developers, unfamiliar with the specific risks associated with LLMs, were left scattered resources and OWASP’s mission seemed a perfect fit to help drive safer adoption of this technology.”

Later on that same page:

“Over the course of a month, we brainstormed and proposed potential vulnerabilities, with team members writing up 43 distinct threats. Through multiple rounds of voting, we refined these proposals down to a concise list of the ten most critical vulnerabilities.”

I, personally, wish they hadn’t stopped at just the top 10. I would surely like to know what the other 33 vulnerabillities are all about. But at least it seems people are taking this seriously.

Cheers,

— steve

On Aug 3, 2023, at 7:48 PM, Les Chambers <les at chambers.com.au> wrote:

Peter
Your comment:
"Martyn already observed on 2023-06-27 that there are legal requirements which
constrain deployment of safety-related systems. That legal requirement in the
UK and Australia is 77 years old. Your question seems to be suggesting that you
somehow think it, and other constraints, might no longer apply. Well, they do.
As Martyn said "AI doesn't change that.
In the UK or Australia, developer and deployer must reduce risks ALARP."

. is righteous . that is, if decreed by a king ruling by fiat (Latin for "let
it be done").

Legal requirements are one thing, usually coming into play to the right of
"bang"; keeping the public safe in the first place is another more important
issue.
The interesting question is, how does one PROVE (to an auditor or a judge) that
one has reduced risks ALARP if one's delivered system's behaviour is initiated
from a neural network? A dataset that cannot be interpreted, verified or
validated thoroughly in process, and that changes after delivery. AI
aficionados admit they don't understand why NNs can work so well or fail so
unpredictably.
Witness: https://dawnproject.com/

Case study: Elaine Herzberg, killed by a self-driving Uber in Tempe, Arizona in
2018. The system did not classify her as a pedestrian because she was crossing
without a crosswalk; the neural net did not include consideration for
jaywalking pedestrians.

These systems are famous for not knowing what they don't know and imposing
their ignorance on the real world. Hannah Arendt was prescient: "It's not so
much that our models are false, but that they might become true"

Imagine yourself as an expert witness supporting Tesla in a similar situation.
What section, subsection or footnote of IEC 61508 or ISO 26262 - or other
standard - would you cite to prove Elon had applied best practice in his
development life cycle?

Or, if you cannot pony up, would you agree that these standards are no longer
fit for purpose in regulating the development of AI-integrated Safety-Critical
systems?

And furthermore, please explain the purpose of these standards, if they cannot
be instrumental in stopping the murder for money currently occurring on US
roads?

Les

PS: I note that Tesla's full self-driving (FSD) feature is available in the UK
as well as the US. It is not available in Australia or Germany.

---------------------------
On 2023-08-03 02:32 , Les Chambers wrote:

Can anyone on this list refer me to where in the standards one can obtain
guidance on how to engineer such a system safely?

That seems to be a question with a completely obvious answer.

Martyn already observed on 2023-06-27 that there are legal requirements which
constrain deployment
of safety-related systems. That legal requirement in the UK and Australia is
77 years old. Your
question seems to be suggesting that you somehow think it, and other
constraints, might no longer
apply. Well, they do. As Martyn said "AI doesn't change that."

In the UK or Australia, developer and deployer must reduce risks ALARP.

How do you go about engineering any system such that risks are reduced ALARP,
say in the UK? You
follow sector-specific functional safety standards if there are some, as well
as the engineering
functional safety standard for E/E/PE systems, which is IEC 61508. This
approach is regarded by the
regulator, at least in the UK, as appropriate to fulfill the ALARP
requirement (although of course
the courts are the final arbiters of that).

PBL

Prof. i.R. Dr. Peter Bernard Ladkin, Bielefeld, Germany
Tel+msg +49 (0)521 880 7319  www.rvs-bi.de

--
Les Chambers
les at chambers.com.au

https://www.chambers.com.au
https://www.systemsengineeringblog.com

+61 (0)412 648 992

_______________________________________________
The System Safety Mailing List
systemsafety at TechFak.Uni-Bielefeld.DE
Manage your subscription: https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety

_______________________________________________
The System Safety Mailing List
systemsafety at TechFak.Uni-Bielefeld.DE
Manage your subscription: https://lists.techfak.uni-bielefeld.de/mailman/listinfo/systemsafety

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/pipermail/systemsafety/attachments/20230804/b281d7f1/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: preview.png
Type: image/png
Size: 68111 bytes
Desc: preview.png
URL: <https://lists.techfak.uni-bielefeld.de/pipermail/systemsafety/attachments/20230804/b281d7f1/attachment-0001.png>