Spokane's $47 million dollar radio blackout

[mmckenna]

"Most modern higher end network equipment (like what should be used in a public safety radio system) are available with dual power supplies. We use those at work for everything. One power supply gets plugged into "normal" power. The other power supply gets plugged into UPS power, or if at one of our main sites, inverter system power. "

That's not why they have dual power supplies. Historically, the single power supply has been one of the greatest failure points in any computer. When the PS goes down--which it will frequently do because of spikes and mains power garbage--the whole machine goes down. Doesn't matter what you have it plugged into.

So the companies making servers started building redundant dual power supplies in order to ensure that when the power supply blew, odds were that it was only one of them failing and the entire system kept working. Meanwhile you replaced the bad one, and then you replaced the other one just in case it was spike or power line damage.

BOTH of the power supplies should be plugged into redundant UPSes, so that both of them get "clean" power as well as backup protection. Or at least one plugged into a UPS, with the other plugged into a conditioned power line. As a practical matter, dual UPSes and full redundancy aren't a bad idea since UPS batteries have at best a 4-year life anyway, and UPSes also fail. Like the night I had one from Tripplite take powerline damage during a storm. Fortunately the Panasonic batteries in it had internal fuses, or else the entire building would have burned down. And Tripplite was so disinterested in that event, that they got permanently removed from the vendor list.

That may be the way the were designed, but not they way they get used around here.

Big issue we have isn't failing power supplies, although it does happen and having a backup helps.
The issue we see is either someone doing something stupid or a generator that fails to start.

We split things up so a tech error, generator error or ups failure won't take everything down. The UPS systems with extra battery packs get us about 2 hours, which is good enough for edge switches. On the router side and distribution switch side, we've set all our sites up with large -48vdc plants for the PBX and Core routers. Distribution switches and stuff that won't run off DC power get put on an inverter plant run off the 48 volt systems. Works pretty well, but there's always going to be a place where Murphy can get in. Best you can do is try.

I'm surprised that the radio vendors are not getting better at this stuff. With these high profile failures, you'd think they'd bring in someone to look at these common failure modes and develop a better plan. Single network switch failure shouldn't have to take the whole system down.

 

[BlueDevil]

What ended up being the issue? Or do they know? Are they still having issues?

 

[kd7kdc]

They said it appears to be a traffic overload of a port (microtik?) on the digital microwave system that caused a freezup not allowing the second port to take over because the logic circuit didn't detect the failure.
Bottom of page 4
http://www.spokanecounty.org/AgendaCenter/ViewFile/Minutes/09272016-1535


Sent from my iPhone using Tapatalk

 

[AZDon]

Systems can crash but if they are designed right there should be a couple of levels of redundancy. Not too terribly long ago the Detroit PD radio system crashed. Thought I heard a BIM failure caused the crash. As a last resort the systems should go into a default standalone repeater mode allowing some communications.