[SystemSafety] Software Glitch Caused 911 Outage For 11 Million People

[Taylor Johnson]

Software Glitch Caused 911 Outage For 11 Million People
http://it-beta.slashdot.org/story/14/10/22/1428214/software-glitch-caused-911-outage-for-11-million-people

How a dumb software glitch kept thousands from reaching 911
http://www.washingtonpost.com/blogs/the-switch/wp/2014/10/20/how-a-dumb-software-glitch-kept-6600-calls-from-getting-to-911/

"...But on April 9, the software responsible for assigning the codes
maxed out at a pre-set limit; the counter literally stopped counting
at 40 million calls. As a result, the routing system stopped accepting
new calls, leading to a bottleneck and a series of cascading failures
elsewhere in the 911 infrastructure..."

FCC Report of Incident:
http://transition.fcc.gov/Daily_Releases/Daily_Business/2014/db1017/DOC-330012A1.pdf

"The outage was caused by a software coding error in the Colorado
facility, and resulted in a loss of 911 service for more than 11
million people for up to six hours. Over 6,600 calls to 911 never
reached a PSAP. Although, fortunately, it appears that no one died as
a result, the incident - and the flaws it revealed - is simply
unacceptable. Americans rely on 911 as a reliable way to communicate
in an emergency, and lapses like this cannot be permitted."

"At 11:54 p.m. PDT20 on April 9, 2014, the PSAP Trunk Member's (PTM)
counter at Intrado's
Englewood, Colorado, ECMC exceeded its threshold and could send no
more 911 calls to PSAPs
using CAMA trunks. Under normal operations, the PTM assigns a unique
identifier for each call
that terminates using CAMA trunks. This is how Intrado has implemented
the ATIS protocol
commonly used to complete 911 calls over CAMA trunks, which (unlike
SS7) require additional
features to carry the signaling along the TDM path."

"In this case, the trunk assignment counter reached a pre-set capacity
limit to assign trunks, which meant that no additional database
entries to reserve a PSAP CAMA trunk could be created, no trunk
assignments for call delivery could be made for PSAPs with CAMA trunks
and, therefore, no 911 calls could be completed to these PSAPs or any
backup PSAP through the Englewood ECMC."

---

http://www.TaylorTJohnson.com/
Taylor Johnson
Assistant Professor
Computer Science and Engineering
University of Texas at Arlington
817-272-3610 (office)
Engineering Research Building 559 (office)
Engineering Research Building 105B (lab)