Richmond EDACS System Down

[SAR923]

I don't know what's happening but the Richmond EDACS trunked system has been down since about 0930 this morning. The control channel on 866.100 will occasionally come back on the air, usually with a weak signal, and fade out again. It seems as though the failsoft system isn't working either since fire traffic is being dispatched by Consolidated Fire. I don't know how the PD traffic is being handled. This affects Richmond, Pinole, San Pablo, El Cerrito, Hercules, and the Contra Costa USD PD.

Gives you a warm, fuzzy feeling on how well things will work when we have the big earthquake.

 

[Eng74]

Just remember, built by the lowest bidder to the specks on the RFP.

 

[selgaran]

Gives you a warm, fuzzy feeling on how well things will work when we have the big earthquake.

No kidding. On the very thin silver lining side, they're getting to practice communications failure before the big one...

 

[hardware1197]

Richmond EDACS Down

So what are they doing? Back to UHF? Cans and string?

There's all kinds of bad stuff going on in Richmond these days

 

[SAR923]

It's 2145 now and the Richmond system is still down. Apparently all the fire traffic is being handled by CoCo Consolidated Fire since all the fire units still have VHF radios but I've yet to find out where the PD traffic is being handled. The system was up for about two mnutes around 1700 and I heard one officer ask if they should continue to use Nextel or the regular radios. Unfortunately, the control channel went down again in the middle of the dispatcher responding so I didn't hear the answer. There's no unusual traffic on the old UHF frequencies, CoCo S.O., or the state 800 mHz mutual aid channels so it may be that they really are handling the PD traffic by Nextel. :roll: I can now get their control channel on 866.100 but there is no traffic moving on the system.

This has now been going on for at least 13 hours and I'm suprised there's nothing on the media about this yet. One would think that, with all the complaining the media does about access to public safety frequencies, they'd notice when a major trunk system in the Bay Area isn't working all day. :shock:

 

[hardware1197]

it's got me thinking

We'd be in big trouble in the City. We'd be on talkaround on the state 800Mhz "mutual aid" Fire Police and Sheriff on one repeatered channel. That would be fun.

 

[SAR923]

I just gone done talking to a media guy. They are working on the story for the morning news but Richmond has apparently been none too cooperative so far. He hasn't even been able to get them to confirm how they are handling the PD calls for service. The system is still down as of this posting.

As far as my pea-sized brain understands EDACS systems, this should never have happened. The system should have gone to failsoft and used conventional channels with the repeater. Apparently the repeater failed also or maybe a repeater failure was the precipitating event. Whatever the cause, I hope the media digs into this. There were six homicides in Richmond in the last two days and the City Council wants to declare a "state of emergency" so they can get $2 million from the state for...more police. I would think that a radio system that's completely failed for the better part of a day would be a bigger "state of emergency" issue right now.

 

[hardware1197]

Keep the heads down

I'm praying for the troops out there...I'm guessing cellphones are being used. Careful. I'd be freaking out.

 

[SAR923]

Richmond's system came back online for test about 0330 and was back up in "normal" operations at about 0700. There's a little more information about the failure at http://www.contracostatimes.com/mld/cctimes/email/news/11938788.htm . I'm amazed, if it's true, that the failure of a single fiber optic cable could bring down the entire system for 36 hours. I have no idea what this "homeland security" radio system was supposed to be but KGO-TV had film of the Richmond dispatch center with what appeared to mag mount antennas on top of the dispatch consoles!

I'm sure the average citizen neither knows nor cares about this story but I hope it gives pause to some of the communications managers that are setting up new systems. Imagine what the failure rate of similar systems will be in the face a major earthquake. Even though we've been assured over and over again that these systems can't fail and there is enough redunduncy built in to still continue to operate if it does fail, this is obviously not always the case.

 

[RolnCode3]

Jim, that link requires registration, just FYI.

 

[SAR923]

Thanks, Bill. For some reason, it didn't when I first viewed the story. At any rate, here is the text:

Main radio system fails, West County police report

Equipment on the main radio system failed Saturday at the Richmond Communications Center, disrupting dispatchers' ability to talk with police officers in Richmond, Hercules, Pinole, San Pablo, El Cerrito and Kensington, according to police.

Officials created a temporary radio system using equipment held in reserve in the event of an attack involving weapons of mass destruction, which had been purchased for a regional homeland security group.

The temporary system was slower but also worked, and no major public safety problems occurred, according to police.

The temporary system was set up by the Contra Costa Office of Emergency Services, West County law enforcement agencies and fire agencies.

The equipment that failed was a fiber optic multiplex. Police did not say when it would be fixed.

 

[SAR923]

As a follow up to the Richmond EDACS fiasco, not only was this caused by a failed fiber optic multiplexer, Richmond apparently never even considered the possibility of the mux failing and had no backup or spare. MA/COM also had no spare anywhere in the western US and they had to do an emergency air freight shipment from back east Saturday. The new mux didn't arrive until 0100 Sunday and the system wasn't back online until 0700 Monday. The system actually failed at about 0830 Saturday morning so it was offline for almost 36 hours.

The "homeland security" radio system consisted of Richmond using one of the Contra Costa S.O. VHF radio frequencies and the other cities that are part of the system using VHF mutual aid frequencies running through a portable repeater. All this was done with HT's and Richmond actually did have mag mount antennas in the consoles so they could talk to the portable repeater. They ran two officer units since they couldn't depend on this "system" to be reliable if a backup was needed.

To Contra Costa County's credit, they were able to cobble together this sytsem in about three hours. During that three hour gap, the communication was by Nextel, cell phones, and MDC's, which weren't affected by the outage. The PIO stated that none of this affected their calls for service and everything went smoothly. Right. :roll:

The most amazing part of this to me is what poor disaster recovery planning was done when this trunk system was set up. The only connection between the dispatch center is the fiber optic line. There's no landline or microwave backup. If some contractor dug up their fiber optic line in error, they'd be in big trouble again. If nothing else, the mux should have been considered a critical point of failure in the system and they should have a dual mux or a mux spare on hand.

All this makes me wonder how many other systems are subject to this type of critical failure and have done the same poor job of planning.

 

[vince48]

As a follow up to the Richmond EDACS fiasco, not only was this caused by a failed fiber optic multiplexer, Richmond apparently never even considered the possibility of the mux failing and had no backup or spare. MA/COM also had no spare anywhere in the western US and they had to do an emergency air freight shipment from back east Saturday. The new mux didn't arrive until 0100 Sunday and the system wasn't back online until 0700 Monday. The system actually failed at about 0830 Saturday morning so it was offline for almost 36 hours.

The "homeland security" radio system consisted of Richmond using one of the Contra Costa S.O. VHF radio frequencies and the other cities that are part of the system using VHF mutual aid frequencies running through a portable repeater. All this was done with HT's and Richmond actually did have mag mount antennas in the consoles so they could talk to the portable repeater. They ran two officer units since they couldn't depend on this "system" to be reliable if a backup was needed.

To Contra Costa County's credit, they were able to cobble together this sytsem in about three hours. During that three hour gap, the communication was by Nextel, cell phones, and MDC's, which weren't affected by the outage. The PIO stated that none of this affected their calls for service and everything went smoothly. Right. :roll:

The most amazing part of this to me is what poor disaster recovery planning was done when this trunk system was set up. The only connection between the dispatch center is the fiber optic line. There's no landline or microwave backup. If some contractor dug up their fiber optic line in error, they'd be in big trouble again. If nothing else, the mux should have been considered a critical point of failure in the system and they should have a dual mux or a mux spare on hand.

All this makes me wonder how many other systems are subject to this type of critical failure and have done the same poor job of planning.

Jim
Nice report. I can not imagine the communications officer or manager or
MA/COM not planning properly. This is not a good signal for or public agencies to project.

 

[SteveS]

to be fair, it was not M/A-COM's fault this occured nor was it M/a-com's part of the system. The actual radio system never did fail. It was some failed copper, and Macom does not make copper to the best of my understanding. To get that part brought in over Father's day weekend is a miracle in and of itself.

Richmond though is not run by the 'best and brightest' so all things considered it probably went the best it could and thank God that no officers were hurt because some copper fell apart.

 

[SAR923]

From further discussions I've had with people who were involved in fixing this mess, it was an MA/COM multiplexer that failed. I'm not sure what you mean by copper but this wasn't a failure of the comm lines, it was a failure of the mux that's needed to convert the fiber optic signal back to analog. The delay was due to the fact that MA/COM had no replacements anywhere on the west coast so a replacment had to be flown from the east coast. To the extent that MA/COM (or Ericisson, as it was then) was involved in the original system plan and failure recovery plan, I can certainly blame MA/COM. The customer bears an even greater share of the blame since, ultimately, any vendor will do what the customer wants within the budget they have. Not specifying a dual mux or having a spare means someone dropped the ball when they did a failure analysis of this system.

You're right, though, that they were lucky no officer was hurt because of this. There was no wide area disaster and the cell phones and Nextels still worked. I still have to wonder how many system failures are lurking out there when we have the big quake, when cell phones and Nextel's probably won't be working very well either.

 

[ECPD279]

Glad I didn't work that weekend!!