Vulnerability of Emergency Systems

[AK_SAR]

The recent bombing in Nashville illustrates once again just how fragile and vulnerable our emergency comm systems can be. The explosion shut down cell, and internet over a wide area. Even worse, a large number of 911 PSAPs were put out of action. Check out this article from the NYT:

When Nashville Bombing Hit a Telecom Hub, the Ripples Reached Far Beyond - The New York Times (nytimes.com)

Their 911 dispatch center was failing, and a backup system was only partly functioning. Veteran officers were having to teach younger colleagues how to use two-way radios to communicate with each other because their cellphones were not receiving a signal.
-------------------------------snip------------------
At the height of the outage, 46 counties in Tennessee had disruptions in 911 call center services, state officials said. As of Tuesday, four counties (Bledsoe, Cannon, Lincoln and Perry), home to about 70,000 people, were still without call center services and were using alternate numbers to route emergency calls.
---------------------------snip----------------------
Severe disruptions were also reported in Kentucky and Alabama.

 

[iMONITOR]

Not all of it is vulnerable:

33 Thomas Street

33 Thomas Street - Wikipedia

en.wikipedia.org

These towering, windowless, bomb-proof buildings in major US cities are reportedly part of an under-the-radar partnership between AT&T and the NSA

These towering, windowless, bomb-proof buildings in major US cities are reportedly part of an under-the-radar partnership between AT&T and the NSA

The NSA has been using AT&T facilities in New York, Atlanta, Chicago, Los Angeles, Dallas, San Francisco, Seattle, and Washington, DC, to tap into phone, text, and browsing records of people in the US and around the world. It is part of a massive surveillance program codenamed Fairview.

www.businessinsider.com

 

[mmckenna]

Back in the cold war days, AT&T did a really good job of hardening their networks against bombs, including nearby nuclear detonations.

If you look at the info from Nashville, the direct cause of the outages was not the bomb, the systems stayed up. It was power that got them after the battery plant was exhausted. If the site had not been a crime scene, they probably would have restored power quickly and there would have been no impact.

 

[RFI-EMI-GUY]

AT&T Nashville is lucky the bomber did not know what he was doing. It appears he built a crude Fuel-Air bomb. Had he used ANFO the AT&T building and its contents would have been destroyed.

 

[AK_SAR]

If you look at the info from Nashville, the direct cause of the outages was not the bomb, the systems stayed up. It was power that got them after the battery plant was exhausted. If the site had not been a crime scene, they probably would have restored power quickly and there would have been no impact.

Perhaps. However, a bigger or better designed bomb (Like Oklahoma City) could have taken out the whole building, in which case it would not have been so easy to restore systems. Also, in past threads we talked about how the California fires disrupted cell service over a very wide area, also at least in part due to power failures. And a few years back a hardware failure in Colorado knocked out 911 call centers for a day or more across a wide swath, clear out to Washington state. Our internet, cell, and emergency comms are now very intertwined. It all works remarkably well....right up until it doesn't.

Note that I'm not anti-ATT. They provide a service at the price that the public is willing to pay. I am merely pointing out that our systems, while very capable, are also much more fragile than most people realize. And much more fragile than providers such as ATT want to acknowledge. The complexity and interrelated nature of these systems make them very vulnerable to failure if the damage occurs at certain key points in the system. But hardening, physical security, back-ups, and redundancy cost $$$$, which are not always forthcoming.

 

[mmckenna]

Perhaps. However, a bigger or better designed bomb (Like Oklahoma City) could have taken out the whole building, in which case it would not have been so easy to restore systems. Also, in past threads we talked about how the California fires disrupted cell service over a very wide area, also at least in part due to power failures. And a few years back a hardware failure in Colorado knocked out 911 call centers for a day or more across a wide swath, clear out to Washington state. Our internet, cell, and emergency comms are now very intertwined. It all works remarkably well....right up until it doesn't.

I don't know anything about explosives, but from looking at the damage in Oklahoma City, I don't doubt it. With enough desire, any system can be taken down. Nashville was pretty lucky it wasn't worse.

The cell interruption in California was due to power issues. Either utility infrastructure was damaged, or power was purposely cut to help prevent fires. Either way, California Public Utility Commission has put rules in place that is requiring cell carriers to address that, either through much larger battery plants, or generators on site. 72 hours of back up power will be required at most cell sites in the near future.

As for the big 911 outages, that was a router issue, not damage or an equipment failure. Someone boogered up the router configs and took it down. People are often the weak point, and as these systems get more and more complex, there'll be more fat-fingering of router configs taking things down.

Note that I'm not anti-ATT. They provide a service at the price that the public is willing to pay.

Yeah, I hear you. They've gone down hill in the last 10 years. I've got a bunch of PRI circuits at work, and we rarely have issues with them. In fact, It's been probably 10 years or more since I've had any significant outages. Problem AT&T has is with their people. Most of the good old timers have either retired, or left out of frustration. The guys that are left are either over worked or not very good. Used to be we'd get the same techs at our site, now it's some random guy that may be coming from 2 counties over because he's the only one available that knows how to work on specific circuits, but he has no idea where any of the infrastructure is.

Over reliance on IP isn't good, either. Too many fingers in that pie. Unfortunately, packet switched is the way everything is going, or already gone.

Me? I'm over in the corner with my butt set clipped on an analog line doing just fine.

 

[radiomankc]

Keep in mind, however, that these critical infrastructure sites are vulnerable to many single points of failure. Power, generators, batteries have their own set of issues. But the fiber connections could render a facility absolutely useless. The fiber is accessible by sewers, all fiber ends up at the railroad tracks, everything could be back hauled states away, etc etc.

I have devoted my professional career to communications resiliency. And I can say without hesitation, we aren’t nearly close to where we need to be.
That is where the professional demeanor and expertise of the Amateur radio comes into play. And that service is invaluable.

I have helped build out many EOCs and they all have amateur radio incorporated. It’s a critical component to emcomm and disaster coordination and communication.

the next few weeks could be very interesting. Be prepared with all your communication capabilities.

 

[mmckenna]

Keep in mind, however, that these critical infrastructure sites are vulnerable to many single points of failure. Power, generators, batteries have their own set of issues. But the fiber connections could render a facility absolutely useless. The fiber is accessible by sewers, all fiber ends up at the railroad tracks, everything could be back hauled states away, etc etc.

I'm not aware of any carriers utilizing sewer systems for fiber. I know of at least one place that utilized an old storm drain for part of a path, but that was a short run to get under a highway.
Redundancy is a thing, and multiple paths in/out of a central office or colocation facility can help. Any good facility will have paths exiting in opposite directions. It's part of their ring topology, and they learned pretty quick that using the same cable to hold different segments wasn't a good idea. The only protection it gave them was from optical transceiver failures, nothing more.

Biggest issue I've seen is disgruntled employees of the carriers. The understand the infrastructure and know where the weak points are. We had that happen back in 2008 where several manholes were opened in precisely the right locations and took out a large area.

Anyone that -really- relies on that sort of infrastructure should be using multiple carriers and making sure the paths are not the same. Even the new NG911 systems are using 4 separate providers.

I have devoted my professional career to communications resiliency. And I can say without hesitation, we aren’t nearly close to where we need to be.

Yeah, and a lot of reasons for that. Some carriers rely on limited path diversity. Some cut corners to keep pricing down.

That is where the professional demeanor and expertise of the Amateur radio comes into play. And that service is invaluable.

I have helped build out many EOCs and they all have amateur radio incorporated. It’s a critical component to emcomm and disaster coordination and communication.

Amateur radio certainly has it's place.

the next few weeks could be very interesting. Be prepared with all your communication capabilities.

A lot of hot air. I don't think we have much to be concerned about.

 

[spacellamaman]

from that nyt article:

" Veteran officers were having to teach younger colleagues how to use two-way radios to communicate with each other because their cellphones were not receiving a signal. "

awww come on.... wtf? do reporters get tenure or something? why can you write something like that and still keep your job? you can't just make stuff up, regardless of how stupid you think people are.

and as for the site of the blast being a crime scene and that causing the issue...well i guess it still served as a means to an end, whatever that end was...

 

[RFI-EMI-GUY]

Post 2 shows 33 Thomas St and states it is not vulnerable. That may be true, but the power and fiber getting in and out of that place has limited number of approaches. You can be sure they are vulnerable somewhere.

 

[mmckenna]

Even before fiber, there were attacks on the links. Since many of the microwave sites were unmanned, they were easy targets:

AT&T Long Lines - Explosion Damage Photos

 

[RFI-EMI-GUY]

Even before fiber, there were attacks on the links. Since many of the microwave sites were unmanned, they were easy targets:

AT&T Long Lines - Explosion Damage Photos

I don't think they ever solved the who and why of those. Probably someone concerned with Reptilian Cattle Snatchers.

 

[danesgs]

We still have a copper line being paid for here. How safe is that versus the other stuff mentioned?

 

[mmckenna]

We still have a copper line being paid for here. How safe is that versus the other stuff mentioned?

Still originates from the same central office, one way or another.
But taking out copper cable doesn't impact as many users. Less of a target.

 

[Thunderknight]

from that nyt article:

" Veteran officers were having to teach younger colleagues how to use two-way radios to communicate with each other because their cellphones were not receiving a signal. "

awww come on.... wtf? do reporters get tenure or something? why can you write something like that and still keep your job? you can't just make stuff up, regardless of how stupid you think people are.

Probably more like "teaching younger colleagues how to find zone 4 on their radios"

 

[spacellamaman]

I don't think they ever solved the who and why of those. Probably someone concerned with Reptilian Cattle Snatchers.

and it appears their days are numbered, The Revolution Will Not Be Facebook Lived!

We still have a copper line being paid for here. How safe is that versus the other stuff mentioned?

Depends, the ALE in NC is more concerned with kids having a good time. It hard worked trolling mountain streams for Snuffy Smith.

 

[RFI-EMI-GUY]

and it appears their days are numbered, The Revolution Will Not Be Facebook Lived!

snip

Oh my,, Gil Scott-Heron will be rolling in his grave on that....