Spokane's $47 million dollar radio blackout

Status
Not open for further replies.

SCPD

QRT
Joined
Feb 24, 2001
Messages
0
Location
Virginia
Anything electronic will fail at one point. For one to takes words literal he shouldn't have. They could have got a quality system for half that what they paid. What bothers me is another example where not one radio had any fail safe simplex operation enabled, or state interop or national interop. This is where on those dual band radios the old repeaters still be programmed in so they could revert. Heck they have licenses still on the old stuff. I know cities who use the public works channel as a backup in event law pr fire goes down. Then endless simplex or other agreements in place with other entities to use repeaters. They bought a name and did not research a thing. Motorola is a dying entity and its quality is going down.
 

kd7kdc

Completely Banned for the Greater Good
Banned
Joined
Jan 18, 2005
Messages
642
Spokane does have both pre/post rebanded NSPAC 800 channels in their radios. As for failsoft setup I don't know why they didn't use that.


Sent from my iPhone using Tapatalk
 

Baker845

Member
Joined
Dec 24, 2005
Messages
380
Location
anywhere
why do they also interview people that have no clue about radios systems? They make it sound like the TRS system would never fail, but in the real world they do and being almost 2 years old, issues will come up from time to time.
 
Last edited:

mmckenna

I ♥ Ø
Joined
Jul 27, 2005
Messages
23,617
Location
Hiding in a coffee shop.
“They promised that this would never happen.”

Issue 1. Trusting "they" and not having an independent review.
Issue 2. Using the term "never".

“We couldn’t talk to each other,” he said.

Issue 4. Lack of planning and training.
Issue 5. For pete's sake, the guys a Battalion Chief, he should understand the technology he is using and be able to adapt to failures.

"Once they determined the radios were out, crews were called on their cellphones and told to switch over to the old VHF radio system, which works on line of sight, said Schaeffer."

Issue 6. "Old VHF radio system" should be referred to as "back up system" or "contingency plan".

"But the station alerting system also went off line, which meant that crews were dispatched using pagers. They knew an address and the type of call, but nothing else. Dispatchers usually relay details about the patient and/or the location, Schaeffer said."

Issue 7. For the love of God, please tell me that they still have wireline telephones in the stations, or did some beancounter get rid of those, too?

Issue 8. No backup system? No wireline fall back? No backup repeaters? If this is a trunked system, it should have gone into failsoft mode.

"Experts are flying out to diagnose the problem and make sure it doesn’t happen again, Knezovich said."

Issue 9. Relying on "experts" that have to be "flown out". Not having local resources is a failure.

Issue 10. "make sure it doesn't happen again". This exact failure might not happen again, but a failure WILL happen again. Thinking that a system is failure proof shows a complete lack of understanding of technology and failure to train radio users to have a fall back plan.

"“We were promised that there was no way there would ever be a single point failure, which happened last night,” he said"

Issue 11. The single point of failure seems to be a combination of lack of training and foolish politicians that buy this line of tripe. There is ALWAYS a single point of failure. It's called the user. There is often a secondary point of failure. It's called the vendor.

"“They’re not supposed to do that.”"

Issue 12. Famous last words.

"...and they will be working to come up with a plan in case there are future outages."

Issue 13. Closing the barn doors after the cows get out. This should have been planned out before the system went live.

What's annoying here is that politicians will spend taxpayer dollars on whatever the vendors tell them to. There is no local on staff subject matter expert that can act as a resource, they'll just trust the vendor, who will tell them whatever they want to hear to get the money.
Failure to understand the tools that the first responders are using, and not having a backup plan that users are trained on ahead of time is a major disconnect.

I know, it's easy to Monday morning quarterback these things, but when it keeps happen time after time and it become apparent that we are not learning from our mistakes, the core issue is a big glaring thing that's hard to ignore.
 

kd7kdc

Completely Banned for the Greater Good
Banned
Joined
Jan 18, 2005
Messages
642
That's funny they are bringing someone over from the west side when the Motorola project manager (for SRECS) and a Motorola senior technical rep who both installed the microwave backbone and the repeaters live right here in the Spokane area.


Sent from my iPhone using Tapatalk
 

RFI-EMI-GUY

Member
Joined
Dec 22, 2013
Messages
6,859
The more complex the system the harder they fail. Imagine having the fire alerting system entirely dependent upon the trunk radio system. Why wouldn't these systems be independent?

I was involved in the installation and acceptance of the very first Smartnet system. It was so new that the part numbers for some options began with INV for 'invention".

The system had a failsoft mode that was supposed to take over if the central controllers (plural because there were two, one hot standby) were to fail. The way this failsoft worked was the repeater stations would detect the loss of a tickle pulse (data) from the controller and would after a short delay revert to a conventional repeater mode. The subscribers sensing a loss of control data would revert to one of these conventional repeaters.

Well the reality was not one of a hard failure but of defective software code that continued to "tickle" the repeaters while transmitting bogus control data. The system went dead, nobody could communicate. The customer was livid that the system could get jammed up in this way without any back up. So I came up with a remote control that would entirely kill power to the central controllers from a guarded panic switch at the 911 center. This way if TSHTF, the operators could flip a red cover over the switch and force the failsoft to take over. It was brute force, but the communications could be backed up in seconds. That switch got used a few more times.
 

RFI-EMI-GUY

Member
Joined
Dec 22, 2013
Messages
6,859
That's funny they are bringing someone over from the west side when the Motorola project manager (for SRECS) and a Motorola senior technical rep who both installed the microwave backbone and the repeaters live right here in the Spokane area.


Sent from my iPhone using Tapatalk

Its always more impressive to fly an expert from afar even when the local techs no exactly what went wrong.
 

mmckenna

I ♥ Ø
Joined
Jul 27, 2005
Messages
23,617
Location
Hiding in a coffee shop.
So I came up with a remote control that would entirely kill power to the central controllers from a guarded panic switch at the 911 center. This way if TSHTF, the operators could flip a red cover over the switch and force the failsoft to take over. It was brute force, but the communications could be backed up in seconds. That switch got used a few more times.

That's a good solution. I like it.
I replaced my old SmartNet system with a Kenwood NexEdge system. Instead of a central controller the individual repeaters talk to themselves via Ethernet. There's also a separate sync/framing link that's looped through the repeaters.
Taking down the Ethernet switch will effectively knock it into failsoft mode. I had to take some steps on our system to make sure our data network engineers didn't force upgrades into the network switch without contacting me first. Luckily, haven't had an issue. I do like your idea, though. I might have to set up a similar setup.
On the other hand, I took some of the money I saved by not going with Motorola and was able to install a completely separate analog repeater at an alternate site, as well as having a simplex channel in all the radios. Now it's user training that is key.
 

RFI-EMI-GUY

Member
Joined
Dec 22, 2013
Messages
6,859
(snip)
On the other hand, I took some of the money I saved by not going with Motorola and was able to install a completely separate analog repeater at an alternate site, as well as having a simplex channel in all the radios. Now it's user training that is key.

That works. You cannot 100% expect that the core system and sites will be there when you need them.

When we upgraded a neighboring system to P25 CAI and simulcast some years later. We suggested sites for coverage and stipulated we wanted to have a redundant site with in cabinet failsoft repeat in the event simulcast control point was lost. Motorola's solution was to transmit simulcast from only the norther most and southern most sites and put a stand alone failsoft repeater in the center. This is an area with very dense buildings. I said, That's a waste! and showed them how they could simulcast all three sites and also remotely force the in cabinet repeat failsoft on the middle site via the option connectors on the repeaters. I don't think it cost a penny more since all the simulcast control equipment supported four sites at minimum.
 

Rred

Member
Joined
Nov 21, 2014
Messages
830
Most people are unaware that after 9/11, the NYC EMS dispatch system went down entirely. The antenna for the new system was on the WTC. The old system (in Maspeth, Queens) had been deemed too costly and unnecessary to keep intact as a backup. "Ooopsie."

NYC also lost all major broadcast television for quite some time, and some radio, as 'everyone' except some minor stations and independent radio facilities had moved to the WTC. Except, WCBS-TV, who were still in the Empire State Building because moving was going to be [chorus] "too expensive".

That Seattle had their older system and was still able to revert to it, are kudos for Seattle.

A number of web sites dealing with EMP generators and theory were quietly taken off the internet, at government request, after 9/11. Precisely because intentional attacks on communications equipment are anticipated. (In fact, the US used to distribute pamphlets to "freedom fighters" telling them how to disrupt communications in the 50's and 60's, and those are still reprinted, even if a bit obsolete by now.)

Ma Bell, R.I.P. Communications will always find a way to go down.
 

BlueDevil

Member
Feed Provider
Joined
May 5, 2010
Messages
770
Location
WA
I really enjoyed reading all these comments and break down analysis of this article and situation. It's happened before and it will happen again. Sounds like a failure to plan ended just as Benjamin Franklin stated.
 

RFI-EMI-GUY

Member
Joined
Dec 22, 2013
Messages
6,859
(snip)(In fact, the US used to distribute pamphlets to "freedom fighters" telling them how to disrupt communications in the 50's and 60's, and those are still reprinted, even if a bit obsolete by now.)

Ma Bell, R.I.P. Communications will always find a way to go down.

I would like to see one of those if you can find a link. Please send!

A big vulnerability are the UPS systems that these networks rely upon. Often they are a single point of failure and they also use SNMP to remotely control and diagnose them. What is to stop a foe from remotely shutting down the charger and alarms and allowing the batteries to deplete? From reading some news reports I suspect this has already occurred to some federal installations.
 

mmckenna

I ♥ Ø
Joined
Jul 27, 2005
Messages
23,617
Location
Hiding in a coffee shop.
I would like to see one of those if you can find a link. Please send!

A big vulnerability are the UPS systems that these networks rely upon. Often they are a single point of failure and they also use SNMP to remotely control and diagnose them. What is to stop a foe from remotely shutting down the charger and alarms and allowing the batteries to deplete? From reading some news reports I suspect this has already occurred to some federal installations.

Good point.

Most modern higher end network equipment (like what should be used in a public safety radio system) are available with dual power supplies. We use those at work for everything. One power supply gets plugged into "normal" power. The other power supply gets plugged into UPS power, or if at one of our main sites, inverter system power.

As for hacking them, out of band networks are the way to go. We have a very large network to manage, and we built a complete separate network for just management. It doesn't touch the rest of the network, and the points of connection to the rest of the world (to allow access from home) are buried behind VPN, firewalls, etc.

I'm almost always surprised at the half-azed ways that some of the larger vendors do their systems. It's like they are completely unaware of the risks.


The big trouble with UPS's is lack of maintenance. Often they get ignored until they catastrophically fail.
 

RFI-EMI-GUY

Member
Joined
Dec 22, 2013
Messages
6,859
Good point.

Most modern higher end network equipment (like what should be used in a public safety radio system) are available with dual power supplies. We use those at work for everything. One power supply gets plugged into "normal" power. The other power supply gets plugged into UPS power, or if at one of our main sites, inverter system power.

As for hacking them, out of band networks are the way to go. We have a very large network to manage, and we built a complete separate network for just management. It doesn't touch the rest of the network, and the points of connection to the rest of the world (to allow access from home) are buried behind VPN, firewalls, etc.

I'm almost always surprised at the half-azed ways that some of the larger vendors do their systems. It's like they are completely unaware of the risks.


The big trouble with UPS's is lack of maintenance. Often they get ignored until they catastrophically fail.

I remember when Smartnet systems were "air gapped" by unplugging the maintenance dial up modem. Everyone did that. Then suddenly Motorola wanted to sell full time network monitoring and suddenly it was VPN's, firewalls and virus protection being required. All this to protect from intrusions that would not happen had these systems not been connected to the "net".

Yes UPS fail rather abruptly as it takes only one weak battery in the string.
 

Rred

Member
Joined
Nov 21, 2014
Messages
830
"Most modern higher end network equipment (like what should be used in a public safety radio system) are available with dual power supplies. We use those at work for everything. One power supply gets plugged into "normal" power. The other power supply gets plugged into UPS power, or if at one of our main sites, inverter system power. "

That's not why they have dual power supplies. Historically, the single power supply has been one of the greatest failure points in any computer. When the PS goes down--which it will frequently do because of spikes and mains power garbage--the whole machine goes down. Doesn't matter what you have it plugged into.

So the companies making servers started building redundant dual power supplies in order to ensure that when the power supply blew, odds were that it was only one of them failing and the entire system kept working. Meanwhile you replaced the bad one, and then you replaced the other one just in case it was spike or power line damage.

BOTH of the power supplies should be plugged into redundant UPSes, so that both of them get "clean" power as well as backup protection. Or at least one plugged into a UPS, with the other plugged into a conditioned power line. As a practical matter, dual UPSes and full redundancy aren't a bad idea since UPS batteries have at best a 4-year life anyway, and UPSes also fail. Like the night I had one from Tripplite take powerline damage during a storm. Fortunately the Panasonic batteries in it had internal fuses, or else the entire building would have burned down. And Tripplite was so disinterested in that event, that they got permanently removed from the vendor list.
 

krokus

Member
Premium Subscriber
Joined
Jun 9, 2006
Messages
5,964
Location
Southeastern Michigan
The more complex the system the harder they fail. Imagine having the fire alerting system entirely dependent upon the trunk radio system. Why wouldn't these systems be independent?

This is a fight I am having right now. The powers that be want to take fire paging off of analog VHF, onto the statewide P25. I want to maintain VHF for reliability, and backup comms.

Sent via Tapatalk
 

krokus

Member
Premium Subscriber
Joined
Jun 9, 2006
Messages
5,964
Location
Southeastern Michigan
That's not why they have dual power supplies.

That is one reason, as what you listed being one. A system I worked on, in a military communication site, had three power supplies in each unit, and two were needed to power the unit. All three were plugged into separate supplies.

Sent via Tapatalk
 

pmb1010

Member
Joined
Jun 29, 2015
Messages
24
BOTH of the power supplies should be plugged into redundant UPSes, so that both of them get "clean" power as well as backup protection. .

This is what we did in critical business systems in the stuff I was responsible for. The servers, network switches, everything had 2 power supplies, on 2 separate UPS units. Dual data paths. And linked to generators.

I was surprised to hear a "switch" would have taken out these comms. I wonder what happened. Switches can be configured to deal with communication loops and have redundant paths.
 
Status
Not open for further replies.
Top