FL, West Palm Beach communications system failure

jrholm · #1 (**permalink**) 10-09-2009, 08:41 PM

Found this on another site and didn't see it posted here yet, anybody have further.

Emergency Workers Lose Radio Communication - West Palm Beach News Story - WPBF West Palm Beach

MTS2000des · #2 (**permalink**) 10-10-2009, 09:13 AM

maybe Mark (KS4VT) can advise, this is his system I believe.

blantonl · #3 (**permalink**) 10-10-2009, 11:44 AM

Mark manages the Palm Beach system, not the West Palm Beach system. However, he probably indeed has some insight as to what happened.

productionguy85 · #4 (**permalink**) 10-10-2009, 02:23 PM

They mention Palm Beach County and and West Palm Beach Police which are on a completely different system. So for that to be correct 2 totally independent systems would have to go down... Don't think so.

911scanner · #5 (**permalink**) 10-10-2009, 11:19 PM

"Engineers said they're working to find the exact cause of the problem."

I understand these systems are quite complex, and I also worked with electronics for nearly 15 years, so I can understand "ghosts in the machine".

That being said, considering the level of importance of a system like this and the training/education of the team of techs working to find the exact cause of the problem, it gives me a great deal of pause to think they haven't found the cause yet.

Sure glad I don't live in West Palm.

MM

b7spectra · #6 (**permalink**) 10-10-2009, 11:25 PM

>"Engineers said they're working to find the exact cause of the problem."<

Of course they are! 5 to 1 they will say it's WPB's problem and will charge them for it!

jrholm · #7 (**permalink**) 10-11-2009, 12:45 AM

You would think these systems would have enough redundancy built in to protect this from happening. I wouldn't however be surprised to find Moto saying, "Why yes and we offered it to the city but they didn't want to spend the extra $$$"

MTS2000des · #8 (**permalink**) 10-11-2009, 06:52 AM

not sure about WPB, but many agencies in my area suffer from the same poor decision making when the Motorola salesman came pushing their 800 trunking systems, they choose to complete abandon their already operating conventional VHF/UHF systems with no regard to redundancy or interoperability.

Recently the worst flood in over a century hit my county. I was at work when the waters rose. We have every radio system at our hospital command center, and I was monitoring the situation. One of the biggest issues was mutual aid, our county has an Astro 25 800MHz trunking system, but all the agencies responding (including the State DNR) are VHF conventional. Despite the fact that our local mutual aid group and state law REQUIRES all fire engines to have a VHF high analog radio capable of operating on state and Federal mutual aid channels, apparently this memo never made it to the powers that be. This hindered many operations and created problems coordinating a response.

We've seen TRS failures (both planned an unplanned) resulting in degraded communications, all the time I ask myself why was perfectly good, functioning and paid for equipment removed from service? The cost to maintain such systems is often minimal, and always cheaper than trying to deal with a complete loss of a system at the worst possible time.

In all, 45 minutes of downtime isn't that bad, but that can be a lifetime or a life if there isn't a hot standby available. Moreover, if personnel aren't trained and know how to access such backup resources or systems, you might as not have them online at all.

alphabravo1 · #9 (**permalink**) 10-11-2009, 07:04 AM

Oh great another opportunity (for posters) to bash Motorola.

First we don’t know what happened, the news media doesn’t understand or care about radio systems. It does appear both systems had problems; and for only a short time,
PBCFR was able to dispatch using a back-up system.

This is a good, well-designed system, and it was so under the old VHF system as well.

Another clue is the news said the other time this happened was during a hurricane…

Points to communications link failure, microwave unlikely, so I would say AT&T phone line failure. When I was down there (under BellSouth) working for an alarm company, disconnecting an active phone line (or circuit) for dedicated alarm systems was quite common.

However, I don’t know and I’m sure Mark will let us know more details if he is able.

KS4VT · #10 (**permalink**) 10-11-2009, 09:13 AM

Greetings all, the outage was due to something occurring to the "T-Bar" switch that connects the Main and Secondary Site controllers. For some reason it got stuck in the middle while doing a transition where the two controllers were, for lack of a better term, fighting it out. Due to this the system wouldn't allow any channel grants, although the control channel remained in operation so someone with a scanner probably wouldn't of noticed anything except for no voice traffic.

Once arriving it was back in operation in less than 3 minutes and it took us longer to drive to the prime site than it did to fix the problem. Over all downtime, approximately 45 minutes but most of the agencies have alternative communications available. Most of the LE agencies have MDT's and the FD's that are dispatched by PBCFR have alpha pagers, and a majority of the agencies moved to the NPSPAC MA's repeaters, so they were never out of total communications.

As the article posts, the system is very reliable and has been our only issue since 2003 where hurricane Frances caused water intrusion into the on-site generator that failed 1.5 days after the hurricane left us and the dispatch center that was tasked with monitoring the alarm system never advised us that the site was on UPS power and the batteries ran out. I average 7,000 PTT's an hour, or over 6 million a year and our users are very happy with the system in both coverage and reliability.

One other point is that we exercise each controller monthly as we roll them on the same day we do the database updates, so we do exercise both of them equally allowing for each one to be the main on opposite months. Also we have no AT&T T-1's connections between any of the sites.

We can close this chapter and move on as these things do happen but we do our best to identify any issues that can cause embarrassment.

MTS2000des · #11 (**permalink**) 10-11-2009, 09:53 AM

good work Mark. In no way was I ever inferring that your system is inferior, I wish we had you up here running the show. We might actually have 95 percent on hip portable coverage, reliability and interoperability on a single metro wide system that everyone shared. But I doubt you'd like the weather here and like even less the politics that Atlanta is so (not) famous for.

The comments I made referred to our situation, which is quite unique. As far as the media, don't get me started on their misinformation on public safety radio communications. If hear this "2013 digital mandate" one more time...(gee I wonder where this work of fiction originates from?)

KS4VT · #12 (**permalink**) 10-11-2009, 11:19 AM

Quote:

Originally Posted by MTS2000des

good work Mark. In no way was I ever inferring that your system is inferior, I wish we had you up here running the show.

No problem and I never took your other post personally. I do have friends in the burbs of NE Atlanta and it is very nice up there. It might be on my list as a possible retirement locations after I'm done with my 30 years of service in mid-2018.

Mark

911scanner · #13 (**permalink**) 10-11-2009, 01:52 PM

And my original post was too pointed towards the system techs/managers, believing what I read in the article. My apologies to the system techs/managers.

Might I recommend that you send a comment to the news outlet to print a correction, as the problem was assessed & fixed prior to the publishing of that statement.

It makes the system techs/managers look bad, when it was an equipment failure (which happens) that the techs/managers had fixed quicker than the reporter could make a phone call.

MM

dave7933 · #14 (**permalink**) 11-12-2009, 04:57 PM

Mark--I have noticed something strange on the Palm Beach County system the past couple of months or so. I have listened for years to PBSO District 4/6 channel as I live in that district. Recently I notice that every so often when the dispatcher transmits, the transmission sounds like it is coming across at about 25% power with lower volume and a hissing sound. After a few seconds of this it bounces into full power and the transmission sounds fine. Any idea of what this is/the cause? I have heard it on some other frequencies on the County system as well although I only monitor a few of them.

slicerwizard · #15 (**permalink**) 11-14-2009, 06:46 AM

Quote:

Originally Posted by KS4VT

Greetings all, the outage was due to something occurring to the "T-Bar" switch that connects the Main and Secondary Site controllers. For some reason it got stuck in the middle while doing a transition where the two controllers were, for lack of a better term, fighting it out.

Sounds like a typical failure by Motorola to properly deal with race conditions. Since they don't get triggered that often, they never get fixed.