[SystemSafety] Bursting the anti formal methods bubble

Thu Oct 26 20:25:58 CEST 2017

Here are some additional examples of pretty significant software failures I came across when researching for my new book:

“The outage was caused by a software coding error in the Colorado facility, and resulted in a loss of 911 service for more than 11 million people for up to six hours. … Although, fortunately, it appears that no one died as a result, the incident – and the flaws it revealed – is simply unacceptable.”
See http://transition.fcc.gov/Daily_Releases/Daily_Business/2014/db1017/DOC-330012A1.pdf

“An American Airlines spokesperson confirmed the issue, … ‘Some flights are experiencing an issue with a software application on pilot iPads …’”
See http://www.theguardian.com/technology/2015/apr/29/apple-ipad-fail-grounds-few-dozen-american-airline-flights

“Federal regulators will order operators of Boeing787 Dreamliners to shut down the plane’s electrical power periodically after Boeing discovered a software error that could result in a total loss of power.”
See http://www.nytimes.com/2015/05/01/business/faa-orders-fix-for-possible-power-loss-in-boeing-787.html?_r=0

“The causes of the National Air Traffic Services (NATS) flight control centre system failure in December 2014 that affected 65,000 passengers directly and up to 230,000 indirectly have been revealed in a recently published report … How could an error not tolerated in undergraduate-level programming homework enter software developed by professionals over a decade at a cost approaching a billion pounds?”
See https://theconversation.com/air-traffic-control-failure-shows-we-need-a-better-approach-to-programming-42496

“… give the attacker wireless control, via the Internet, to any of thousands of vehicles. Their code is an automaker’s nightmare: software that lets hackers send commands through the Jeep’s entertainment system to its dashboard functions, steering, brakes, and transmission, all from a laptop that may be across the country.”
See http://www.wired.com/2015/07/hackers-remotely-kill-jeep-highway/

“The US car company General Motors is recalling more than four million vehicles worldwide due to a software defect linked to at least one death . . . the defect concerns the sensing and diagnostic module. In rare cases it can go into test mode, meaning airbags will not inflate in a crash.”
See http://www.bbc.com/news/world-us-canada-37321361

“A flaw found in a calculator tool used by doctors at GP surgeries has potentially led to a number of patients being erroneously prescribed or denied statins across England. … Due to the unidentified error in the code, it's possible that the risk of CVD was overstated … This could—in turn—have led to mistakes in prescriptions for statins”.
See http://arstechnica.co.uk/security/2016/05/bug-in-gp-heart-risk-calculator-tool-tpp/

“… vulnerability in Hospira’s Symbiq Infusion System … could allow an attacker to remotely control the operation of the device, potentially impacting prescribed therapy and patient safety.”
See https://ics-cert.us-cert.gov/advisories/ICSA-15-174-01

“… Apple Pay users with Bank of America accounts have reported that Apple’s new tap-to-pay solution has become a huge headache by charging their accounts twice for a single purchase. Bank of America has confirmed … that it is issuing refunds for duplicate Apple Pay charges.”
See http://www.cultofmac.com/300574/apple-pay-customers-getting-charged-twice-purchases/

“… a currency exchange-rate error in 3rd party software supplied to United (Airlines) affected several thousand bookings on United's Denmark-facing website. Specifically, this error temporarily caused flights originating in the United Kingdom and denominated in Danish Kroners (DKK) to be presented at only a fraction of their intended prices. While United filed fares correctly, this software error caused amounts charged to be significantly lower than prices offered through all other distribution channels or available in any other currency.”
see http://www.united.com/web/en-US/content/travel/exchange-rate-error.aspx?v_ctrk=HHLN$0-202-7697-1-5798

“A programming blunder in its reporting software has led to Citigroup being fined $7m (£5m) … When the system was introduced in the mid-1990s, the program code filtered out any transactions that were given three-digit branch codes from 089 to 100 and used those prefixes for testing purposes. But in 1998, the company started using alphanumeric branch codes as it expanded its business. Among them were the codes 10B, 10C and so on, which the system treated as being within the excluded range, and so their transactions were removed from any reports sent to the SEC.”
See http://www.theregister.co.uk/2016/07/13/coding_error_costs_citigroup_7m/

“The embedded Web server in the Cisco Cable Modem with Digital Voice … contains a buffer overflow vulnerability that can be exploited remotely without authentication. The flaw could result in arbitrary code execution.”
See http://www.itnews.com/article/3042664/cisco-patches-serious-flaws-in-cable-modems-and-home-gateways.html?token=%23tk.ITN_nlt_ITnews_Daily_2016-03-10&idg_eid=3223a51d2577b180c5ff5baf4855a77a&utm_source=Sailthru&utm_medium=email&utm_campaign=ITnews%20Daily%202016-03-10&utm_term=ITnews_Daily

“Juniper Networks is warning customers to patch their NetScreen enterprise firewalls against bad code that enables attackers to take over the machines and decrypt VPN traffic among corporate sites and with mobile employees. … attackers could exploit the code ‘to gain administrative access to NetScreen devices and to decrypt VPN connections,’ ... It would enable smart attackers to exploit the vulnerability and wipe out log files, making compromises untraceable.”
See, http://www.itnews.com/article/3016992/security/juniper-firewalls-compromised-by-spy-code-what-you-need-to-know.html

“Google has released one of the largest Android monthly security updates, fixing a total of 39 vulnerabilities — 15 rated critical, including four that can lead to a complete device compromise.”
See http://www.itnews.com/article/3052203/google-fixes-39-android-flaws-some-allow-hackers-to-take-over-your-phone.html?token=%23tk.ITN_nlt_ITnews_Daily_2016-04-06&idg_eid=3223a51d2577b180c5ff5baf4855a77a&utm_source=Sailthru&utm_medium=email&utm_campaign=ITnews%20Daily%202016-04-06&utm_term=ITnews_Daily

“More than 3,000 inmates in Washington state prisons were released early because of a software bug. The glitch caused the computer system to miscalculate the sentence reduction inmates received for good behavior, according to a press statement from the state governor’s office.”
See http://www.techinsider.io/washington-prisons-software-glitch-2015-12

For example, “… a Department of Revenue (DOR) contractor underestimated the complexity of adapting the sales and use tax software component of DOR’s Integrated Tax System. Doing so contributed to significant programming errors, … and compromised the accuracy of sales and use tax distributions to counties and professional sports districts.”
See http://legis.wisconsin.gov/lab/reports/07-5full.pdf

“US tax authorities (IRS) said … that the personal information of a further 390,000 individuals may have been accessed by cyber criminals. … the IRS said it discovered the data of 114,000 US taxpayers had been illegally accessed through the "Get Transcript" page … A review conducted by the agency has revealed over 700,000 people affected. … The IRS said it found a further 295,000 taxpayer accounts were targeted, but not accessed by cyber criminals.”
See http://www.bbc.com/news/business-35673999

“the current software could result in high temperatures on certain transistors and possibly damage them. When it fails, the error forces the car into failsafe mode. Toyota says that in rare circumstances, it could even shut the hybrid system down while the car is being driven.”
See http://www.autoblog.com/2014/02/12/toyota-recalling-1-9m-prius-models-globally/

“The result is that the code leaps over the vital call to sslRawVerify(), and exits the function. This causes an immediate ‘exit and report success’, and the TLS connection succeeds, even though the verification process hasn't actually taken place.”
See http://nakedsecurity.sophos.com/2014/02/24/anatomy-of-a-goto-fail-apples-ssl-bug-explained-plus-an-unofficial-patch/

“… results from improper input validation (due to a missing bounds check) in the implementation of the TLS heartbeat extension … The vulnerability is classified as a buffer over-read, a situation where more data can be read than should be allowed.”
See http://en.wikipedia.org/wiki/Heartbleed

“Microsoft has issued a security warning about a bug that could let attackers spy on supposedly secure communications … attackers force data travelling between a vulnerable site and a visitor to use weak encryption. This makes it easier to crack open the data and steal sensitive information.”
See http://www.bbc.com/news/technology-31765672

“An unusual bug on Facebook briefly labelled many people as dead… The error on Friday caused the social network to show a memorial banner on user profiles for people who were still alive”.
See http://www.bbc.com/news/technology-37957593

Matthew Squair then wrote:
“Back in the day when riverboat and steam engine boilers went ‘kaboom’ on a regular basis there was a clear recognition that this was technology problem and we got firm regulation and the supporting boiler safety codes. But software? Not so much it appears, my question is why is that?”

I argue it’s due to a number of issues, including but not limited to:

*) Developers are not held personally liable for their defects

*) Not enough people have been killed or otherwise significantly harmed to cause enough of a public outcry

Only when enough damage is done will the resulting public outcry lead to something actually improving. Until then . . .

— steve

From: systemsafety <systemsafety-bounces at lists.techfak.uni-bielefeld.de<mailto:systemsafety-bounces at lists.techfak.uni-bielefeld.de>> on behalf of Matthew Squair <mattsquair at gmail.com<mailto:mattsquair at gmail.com>>
Date: Thursday, October 26, 2017 at 2:03 AM
To: "systemsafety at lists.techfak.uni-bielefeld.de<mailto:systemsafety at lists.techfak.uni-bielefeld.de>" <systemsafety at lists.techfak.uni-bielefeld.de<mailto:systemsafety at lists.techfak.uni-bielefeld.de>>, "martyn at thomas-associates.co.uk<mailto:martyn at thomas-associates.co.uk>" <martyn at thomas-associates.co.uk<mailto:martyn at thomas-associates.co.uk>>
Subject: Re: [SystemSafety] Bursting the anti formal methods bubble

Perhaps we should say that software engineers have a denial problem? Updating Derek’s original 2009 post a little:

1. Qantas QF72 - a dozen or more serious injuries (Inability of software to handle Byzantine fault)
2. Ariane 501 - loss of multi-million dollar mission/payload (poor code reuse)
3. Titan IVB-32 loss of multi-million dollar mission/payload (poor code reuse)
4. Therac 25B multiple patient deaths (race conditions in software)
5. 2011 24% of medical device recalls by US FDA due to software (multiple reasons - mostly patient safety related )
6. Toyota cruise control software defect - at least one death and one severe injury (defective watchdog design + spaghetti code)
7. 2015 A400 crash - Four crew dead and two sever injuries (design fault - software cal data load error undetectable prior to flight)

Back in the day when riverboat and steam engine boilers went ‘kaboom’ on a regular basis there was a clear recognition that this was technology problem and we got firm regulation and the supporting boiler safety codes. But software? Not so much it appears, my question is why is that?

On 26 October 2017 at 3:43:31 am, Martyn Thomas (martyn at thomas-associates.co.uk<mailto:martyn at thomas-associates.co.uk>) wrote:

Some of them are here: JT James, A new, evidence-based estimate of patient harms associated with hospital care,
Journal of Patient Safety, 2013 Sep;9(3):122-8. doi: 10.1097/PTS.0b013e3182948a69., for the reasons that Harold Thimbleby has described.

Martyn

On 25/10/2017 17:23, Derek M Jones wrote:

Software engineering has a dead body problem:
http://shape-of-code.coding-guidelines.com/2009/11/18/where-are-the-dead-bodies/

_______________________________________________
The System Safety Mailing List
systemsafety at TechFak.Uni-Bielefeld.DE<mailto:systemsafety at TechFak.Uni-Bielefeld.DE>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.techfak.uni-bielefeld.de/mailman/private/systemsafety/attachments/20171026/87cfea1d/attachment-0001.html>