AT&T insights on FirstNet performance recovery efforts after Nashville explosion

[AK9R]

I think there's a lot of AT&T spin in this article, but it's interesting to see the issues they faced. In spite of the damage caused by the explosion, AT&T's systems continued to operate. It was several hours later, after the batteries were drained, that the system went down.

AT&T provides initial insights about FirstNet performance, recovery efforts after Nashville explosion - Urgent Comms

Some FirstNet subscribers experienced service outages after the massive Dec. 25 explosion at AT&T’s Nashville network hub, but backup battery power at the facility and the use of deployable cell sites meant that many FirstNet users were able to communicate “within hours,” according to AT&T...

urgentcomm.com

 

[mwjones]

Sounds like AT&T's disaster response/recovery plans worked about as good as they could. They used to do Disaster Recovery exercises in various parts of the country 4 times a year (I was a large business customer and got to see this exercise twice in the late '90s and early '00s), and can only presume that they still do something similar. Unless its a predicted event like a hurricane, it is unlikely that AT&T's "Network Recovery Trailers" and corresponding personnel will be in position to respond and become operational in anything less than 24-48 hours.

On 9/11/01, the AT&T switching office in the basement of the World Trade Center continued operating on battery for several hours after the towers came crashing down upon it, but long after network operations had shifted what traffic they could to the disaster recovery switching office.
But, the bottom line, even with a dedicated PTT over LTE Network like FirstNet, you are still susceptible to widescale outages in the early hours of an event. Something a traditional LMR system (when properly built) shouldn't suffer.

 

[03msc]

Have seen a lot of negative comments and posts about FirstNet around Facebook over the last week or so, many saying why this incident proves it's a failure. I guess perspective is everything.

 

[12dbsinad]

Hi, I'm Jake from FistNet, and I am here to tell you there is nothing better than to give us your money for a Public Safety network... in return we will give you a basket full of eggs.

And they say LMR will be obsolete..... Pfffff...

 

[902]

I think there's a lot of AT&T spin in this article, but it's interesting to see the issues they faced. In spite of the damage caused by the explosion, AT&T's systems continued to operate. It was several hours later, after the batteries were drained, that the system went down.

As a retired professional emergency manager, I would need to question the vendor on what, exactly, prevented it - in their role as a federal contractor - to coordinate with the incident commander (the FBI) to establish emergency power to critical systems immediately. Waiting days for batteries to run down tells me that either they were blocked from the site, which speaks to the relative degree of "essential," or that they had no alarms or capacity to acutely respond to alarms in place. I'd suspect they would hide behind "need-to-know" pretty quickly.

You want to play emergency or essential service, those trucks need to start rolling on receipt of major event notification/alarms, not 72 hours later.

 

[902]

Have seen a lot of negative comments and posts about FirstNet around Facebook over the last week or so, many saying why this incident proves it's a failure. I guess perspective is everything.

I'm very critical of FN, but I think it's a convenient tool. When it works, it, and other broadband networks can carry a lot of metadata that's not practical any other way. But their model puts the cart before the horse. Public safety still needs the training and autonomy to function safely and effectively when the convenient stuff goes away. FN bedazzles decision-makers with spectacular micromanagement tools, like running an incident from afar rather than the scene. That might not be the best model for public safety responders.

We need to think of communications systems as tools in a toolbox. Discrete tools designed for a task work much better than a Swiss Army Knife. FN is pretending to be the Swiss Army knife - "the only thing you'll ever need." That only makes sense to marketing people and IT people (slam deliberately intended). They don't Comm, bro, and that's a fact of life. Radio does radio best, not an app over cellular, or some kind of very elaborate off-network scheme.

The biggest problem with FN, in my view, is that they followed the FCC model and made it heavily loaded with marketing people vs. the "doers" who design and make things work. Those functions may have been wholly abdicated to the vendor. I likened it to the FCC, because the FCC is top-heavy with lawyers and economists at the expense of field enforcement agents and engineers. Lots of bosses and not enough workers (i.e., things aren't getting done quickly, so more managers need to be hired to make the workers work faster rather than more workers to actually do the work) and the concept between FN (as the Department of Commerce/First Repsonder Network Authority) and the vendor (AT&T) imperceptibly blurred to most people. It all turned into Public Safety Telephone instead of the vital adjunct it was intended to be during its formation.

All entirely my opinion, to quote a friend, "I could be wrong."

 

[AK9R]

As a retired professional emergency manager, I would need to question the vendor on what, exactly, prevented it - in their role as a federal contractor - to coordinate with the incident commander (the FBI) to establish emergency power to critical systems immediately. Waiting days for batteries to run down tells me that either they were blocked from the site, which speaks to the relative degree of "essential," or that they had no alarms or capacity to acutely respond to alarms in place. I'd suspect they would hide behind "need-to-know" pretty quickly.

AT&T had equipment in the basement which flooded with water from the efforts to extinguish the fire. The building was declared a crime scene so AT&T staff was prohibited from entering the building.

As for the "relative degree of 'essential'" comment, that's a darn good question.

 

[bchappuie]

Well First Net is not a true separate network. Everything First Net is AT&T network. It may be separated from a logical perspective, however all the hardware is shared resource. So regular customers had same issues as FN. Lots of talk by politicians about harding the network, however that would require putting everything in a bunker. Not feasible.

 

[mmckenna]

Well First Net is not a true separate network. Everything First Net is AT&T network. It may be separated from a logical perspective, however all the hardware is shared resource. So regular customers had same issues as FN. Lots of talk by politicians about harding the network, however that would require putting everything in a bunker. Not feasible.

Not all the hardware. The LTE core is separate. There are sites where the Band 14 base station and antennas are separate. But yeah, a lot of the infrastructure in between is the same or at least in shared cables/networks.


---------

I have never heard anyone who has been in the industry for any reasonable length of time claim that FirstNet was ever going to be a replacement for LMR. FirstNet will even back down when you push them on that. What is driving most of this is their marketing. Their marketing people will bend words to make you think things exist that do not.
And then you have customers that drink the Kool-Aide and just repeat what the marketing people tell them.

What is true is that AT&T has a LOT of experience in hardening networks. Or at least they did. Back in the cold war days, a lot of their network was hardened against nuclear attacks. Most of their older buildings were built to withstand quite a bit. Their old L-carrier networks had the cable buried several feet deep to protect against EMP. Line repeaters were also underground to protect them.
When things went to fiber, it wasn't as critical, and things got a bit sloppy. New construction wasn't up to the same level as the cold war days. As competition heated up, they cut corners. "Redundant fiber rings" were sometimes within the same cable sheath. That bit them in the posterior a few times and in most places they've fixed that. In the rush to move off copper plant, they rushed to fiber before it had enough redundancy.

Usually what gets these CO's and colocation sites is the power. Generally carrier equipment is supported with a minimum of 8 hours of battery backup, often more. On site generators with substantial fuel storage can usually support central offices for several days at least.
What looks like happened in Nashville is utility power was impacted, so the site went on battery. Whatever transfer switches/distribution for the commercial/generator power was damaged or flooded.

The nature of the crime scene kept the service restoration guys out for longer than the batteries would hold. If it hadn't been a crime scene, it's likely the network never would have gone down.

It's easy to tear these things apart after the fact with a bunch of "well, they shoulda…." but truth is it takes a lot of planning to design the systems to sustain substantial damage and keep working. No matter how well you harden the plant, there's always going to be weak spots. Usually a nut-job in an RV with a bomb parking in front of the CO isn't real high on the list of probability. Maybe it needs to be.

What doesn't make the news is how many times the PSTN or cellular networks are NOT impacted by something. Truth is, we have pretty robust networks and failures are not that common.

And any agency that fell for the FirstNet or even the old Nextel thing sort of deserves whatever happened. Too bad for the public, but the agencies should have known better. This is usually what happens when the bean counters wield too much power.

 

[AK9R]

I used to have a cousin who worked for AT&T Long Lines in northern Indiana. Yes, those facilities were well hardened. There's at least one 60-year-old 250-foot-tall concrete tower in northwest Indiana that is now being used as a repeater site for all kinds of systems. It'd take more than a few sticks of dynamite to bring it down.

Some of the AT&T microwave sites in Indiana had steel towers sitting on top of underground bunkers where the switch gear lived. Those sites were converted to fiber when microwave became passé. At one site I'm familiar with, the generators and HVAC equipment originally were in the bunker, but AT&T moved them up to grade level in order to free up more room in the bunker. Should a tornado blow a 2x4 through a generator's radiator or a chiller's condenser, the bunker will be compromised.

 

[KevinC]

I used to have a cousin who worked for AT&T Long Lines in northern Indiana. Yes, those facilities were well hardened. There's at least one 60-year-old 250-foot-tall concrete tower in northwest Indiana that is now being used as a repeater site for all kinds of systems. It'd take more than a few sticks of dynamite to bring it down.

Some of the AT&T microwave sites in Indiana had steel towers sitting on top of underground bunkers where the switch gear lived. Those sites were converted to fiber when microwave became passé. At one site I'm familiar with, the generators and HVAC equipment originally were in the bunker, but AT&T moved them up to grade level in order to free up more room in the bunker. Should a tornado blow a 2x4 through a generator's radiator or a chiller's condenser, the bunker will be compromised.

My favorite thing in those old AT&T MW sites is the Incinolet

 

[mmckenna]

My favorite thing in those old AT&T MW sites is the Incinolet

https://www.amazon.com/Everyone-Poops-Taro-Gomi/dp/1797202642/ref=sr_1_1?dchild=1&keywords=everyone+poops&qid=1609724280&sr=8-1

 

[mmckenna]

I used to have a cousin who worked for AT&T Long Lines in northern Indiana. Yes, those facilities were well hardened. There's at least one 60-year-old 250-foot-tall concrete tower in northwest Indiana that is now being used as a repeater site for all kinds of systems. It'd take more than a few sticks of dynamite to bring it down.

Longlines was a pretty cool division of Ma Bell. Some of the designs are fascinating. I've been to a number of old Longlines sites over the years, and they still amaze me.

The city I grew up in had one of those concrete towers with the big bunker underneath. My brother in law isntalled some microwave gear in it a few years ago and took a bunch of photos for me. A lot of the old equipment was still in place.
Back then, Ma Bell really knew her stuff.

 

[ten13]

All this "FirstNet" stuff came about as a result of the World Trade Center, when Federal bureaucrats and politicians claimed that the entire communications systems working at and around the WTC were insufficient or failed outright. Yet, with all the hullabaloo about "radios" and the like at that scene, none of the major agencies in and around NYC changes their radio ways of operation since then.

The problem was too many people, both inside and outside politics, commented...mostly negatively...about what they heard on after-the-fact recordings, and putting civilian "spin" on what they heard, based on "knowledge" they gleaned from watching TV police and fire shows (similar to those who say that a cop apprehending an armed felon should have "shot him in the hand..or leg...").

After AT&T cashes all the checks from distributing this FirstNet, much of the hardware involved will sit in cars' glove compartments or in office draws, seldom used even for drill purposes, causing a lack of operational knowledge and a general lack of interest in using it.

At some point, someone, somewhere, will come up with an idea of using it for some interstate foot or bicycle race between communities, but not much more.

 

[mmckenna]

All this "FirstNet" stuff came about as a result of the World Trade Center, when Federal bureaucrats and politicians claimed that the entire communications systems working at and around the WTC were insufficient or failed outright. Yet, with all the hullabaloo about "radios" and the like at that scene, none of the major agencies in and around NYC changes their radio ways of operation since then.

Not so much. The big push for interoperability came about in response to 9/11. Back then, mobile broadband wasn't much of a thing, the technology just wasn't there.
700MHz and P25 were heavily pushed in response to 9/11, even though neither of those would have helped. The idea was 'interoperability' required digital. Not the case. Analog works fine, but radios have to be programmed. If you look at the DHS NIFOG, you'll see a lot of what 9/11 generated.
There was some idea that NPSBN/FirstNet would be used for some level of communications in a disaster, but that included telephone calls. While there is the ability to use NPSBN/FirstNet for PTT over LTE, it's not as useful as some hoped due to delays in call setup. Kind of like NexTel when new users would cut off the first part of their message.

The problem was too many people, both inside and outside politics, commented...mostly negatively...about what they heard on after-the-fact recordings, and putting civilian "spin" on what they heard, based on "knowledge" they gleaned from watching TV police and fire shows (similar to those who say that a cop apprehending an armed felon should have "shot him in the hand..or leg...").

True, but that's usually what happens. Winston Churchill said "Never let a good crisis go to waste". When, where and why did Winston Churchill say, “Never let a good crisis go to waste? ” (2020) - Quora

And many companies that profit off public safety make good use of that. Suddenly "interoperability" required buying all new equipment and systems. NPSBN rode in on that, but it was the US Government that pushed for it. AT&T didn't think it up on their own, they were just one of 3 big bidders.

Truth is, there were already some priority telecom solutions in place, but it required dialing a prefix to access. Voice phone calls over FirstNet just automate the process.

After AT&T cashes all the checks from distributing this FirstNet, much of the hardware involved will sit in cars' glove compartments or in office draws, seldom used even for drill purposes, causing a lack of operational knowledge and a general lack of interest in using it.

Not sure what gives you that idea. FirstNet was used to replace a lot of the mobile data terminal systems. The old mobile terminal systems used in public safety were painfully slow and costly to maintain. After all, you can only do so much with a few kilobytes per second. Going to FirstNet, Verizon or any other LTE system opened up a lot of possibilities that would have been difficult for agencies to produce on their own. Having high speed data in the field allows photos, video, building plans and a lot of other tools to be used. Without the LTE networks, that would have not been possible for most agencies.
And no agency is required to use FirstNet. They can use any carrier they want. Many agencies also use more than one carrier. The CradlePoint type devices will often support more than one SIM card, so having multiple carriers available solves a lot of coverage issues and adds to reliability. In addition, the CradlePoint devices can use WiFi, so having hots spots available for agencies is a handy option.

Many agencies have also migrated their hand held devices to FirstNet, so it's not something that's going to get tossed in drawer. The people use them every day for phone calls, data, texting, etc. The rates are pretty cheap, so it's been a cost savings for some agencies, although Verizon has done their best to compete.

Nice thing is there are choices, and that's a good thing.

 

[russbrill]

Well First Net is not a true separate network. Everything First Net is AT&T network. It may be separated from a logical perspective, however all the hardware is shared resource. So regular customers had same issues as FN. Lots of talk by politicians about harding the network, however that would require putting everything in a bunker. Not feasible.

FirstNet = Federally Subsidized LTE buildout for AT&T investors... Ya know, the Good Ole Boys network

 

[MTS2000des]

FirstNet = Federally Subsidized LTE buildout for AT&T investors... Ya know, the Good Ole Boys network

Yep. If there was ever an example of waste,fraud and abuse and a call to "drain the swamp", this was it. The concept of FN is great, the execution sucks. All carriers should have been allowed to connect to a central managed core and leverage carrier competition as well as offering carrier aggregation and some form of redundancy for the critical "last mile" of wireless connectivity. Instead, a sole source vendor 20 year contract worth billions. A great way to let the taxpayers fund one of the giant telecom cartel's upgrade to their poor performing network. And because they get to "allow" consumer traffic onto this supposedly exclusive band 14 spectrum at their sites, they can get a subsidized build out.

Brilliant scam actually.

My father who worked 35 years in commercial finance used to tell me "Son, if you're banker won't loan you money, why should I?"
I would joke about "just go to the government! They'll hand out money"
and that's exactly what went down.

 

[prcguy]

I'm a retired satellite broadcast engineer. In my last line of work the entire broadcast chain to the customer is fully redundant from bringing in the signal we turn around to backup receivers or fiber equipment, backup receive antennas, fully redundant internal router switching and fully redundant uplink systems. Even the fiber bringing in content was from two different companies (ATT and Verizon) entering the premises, via different physical paths for redundancy. In one case we did have a fiber failure over a very long path from South America to California, we found the two fibers converged into one building in some podunk country where it was accidently dug up and that was addressed when discovered.

I would think a high priority public service communications system like FirstNet would take at least the same if not more precautions to deliver continuous service with no interruptions or failures for any reason. Otherwise they need to change their name to "SecondNet" or "ThirdNet" and charge accordingly.

I'm very critical of FN, but I think it's a convenient tool. When it works, it, and other broadband networks can carry a lot of metadata that's not practical any other way. But their model puts the cart before the horse. Public safety still needs the training and autonomy to function safely and effectively when the convenient stuff goes away. FN bedazzles decision-makers with spectacular micromanagement tools, like running an incident from afar rather than the scene. That might not be the best model for public safety responders.

We need to think of communications systems as tools in a toolbox. Discrete tools designed for a task work much better than a Swiss Army Knife. FN is pretending to be the Swiss Army knife - "the only thing you'll ever need." That only makes sense to marketing people and IT people (slam deliberately intended). They don't Comm, bro, and that's a fact of life. Radio does radio best, not an app over cellular, or some kind of very elaborate off-network scheme.

The biggest problem with FN, in my view, is that they followed the FCC model and made it heavily loaded with marketing people vs. the "doers" who design and make things work. Those functions may have been wholly abdicated to the vendor. I likened it to the FCC, because the FCC is top-heavy with lawyers and economists at the expense of field enforcement agents and engineers. Lots of bosses and not enough workers (i.e., things aren't getting done quickly, so more managers need to be hired to make the workers work faster rather than more workers to actually do the work) and the concept between FN (as the Department of Commerce/First Repsonder Network Authority) and the vendor (AT&T) imperceptibly blurred to most people. It all turned into Public Safety Telephone instead of the vital adjunct it was intended to be during its formation.

All entirely my opinion, to quote a friend, "I could be wrong."

 

[KJS777]

Yep. If there was ever an example of waste,fraud and abuse and a call to "drain the swamp", this was it. The concept of FN is great, the execution sucks. All carriers should have been allowed to connect to a central managed core and leverage carrier competition as well as offering carrier aggregation and some form of redundancy for the critical "last mile" of wireless connectivity. Instead, a sole source vendor 20 year contract worth billions. A great way to let the taxpayers fund one of the giant telecom cartel's upgrade to their poor performing network. And because they get to "allow" consumer traffic onto this supposedly exclusive band 14 spectrum at their sites, they can get a subsidized build out.

Brilliant scam actually.

My father who worked 35 years in commercial finance used to tell me "Son, if you're banker won't loan you money, why should I?"
I would joke about "just go to the government! They'll hand out money"
and that's exactly what went down.

You hit the nail on the head! Too expensive a proposition to build a new LTE network from the ground up. For AT&T it was a way to get the taxpayers to pay to upgrade their systems in network where VZW had significantly better coverage and resiliency and they are still not there yet. Nashville is proof that FirstNet is far from the hardened network it was supposed to be. Need more money...

 

[902]

AT&T had equipment in the basement which flooded with water from the efforts to extinguish the fire. The building was declared a crime scene so AT&T staff was prohibited from entering the building.

As for the "relative degree of 'essential'" comment, that's a darn good question.

I see, regarding the crime scene. I hadn't though about that, although if it were vital enough, an evidence preservation team probably could have escorted them in.

Maybe if they hired more engineers one of them would've figured out that putting a MTSO in a basement with a sprinkler system instead of maybe a Halon dump wasn't the best idea. Reminds me of all the CentraCom CEBs that always seemed to end up in utility closets right under leaky pipes.

Sorry for being facetious, Bob, but in my view they should've known better than to put something like that in a soft target.