Police Chief blames city of Detroit, Motorola for 2-hour system crash

[rivermersey]

Craig blames city of Detroit, Motorola for 2-hour system crash | The Detroit News

Detroit — Police Chief James Craig said the provider of the department’s dispatch services and the city itself are to blame for a two-hour communications system crash last week.

One of the communications towers owned by Motorola, the city’s vendor for the dispatch system, failed due to overheating July 5.

“Let’s face it, equipment fails but it’s how we respond to it that counts,” Craig told reporters during a news conference in the city’s new public safety headquarters Friday.

Craig said the department is “holding Motorola accountable” for what he called a hardware failure, but added that he’s looking at how the city may have some responsibility for the downed system. He said the tower, one of 10 used by the system citywide, lacked air-conditioning

From The Detroit News: Craig blames city of Detroit, Motorola for 2-hour system crash | The Detroit News

 

[dcr_inc]

What, a MOTOROLA system failure....Nah.. It must of been something else...

It's a Motorola owned and maintained site..

Ever heard of temperature alarms at the sites?.. At least Harris installs them at EVERY site and checks them as part of their PM's...

Shame on you Bat Wings !!

 

[PJH]

Its called cost vs benefit vs budgets.

In every proposal that I have seen (I dealt with Motorola mostly), the customer was given the "full package" proposal.

Not with Motorola specifically, but all the contracts I delt with, the customer (us goverment) would widdle down the equipment needed to our specifications to the amount we had for the budget. We would ask the world of the vendor, but would only pay for a 3/4. It all comes negioation and risk.

Without going into detail, usually the equipment shelter would be owned by the government, but the equipment was the vendor's. It was typically our responsibility to have the proper generators, grounding and HVAC systems in place.

 

[MTS2000des]

It wasn't a "tower failure", unless the structure itself falls to the ground, I fail to grasp how a tower itself "overheats"....can't these illiterate morons write something accurate? The equipment at the SITE may overheat, but unless the tower is on fire and leaning over, it didn't fail!

 

[PJH]

We know that, however the lay person reading the paper (if they actually newsprint) wouldn't.

 

[N4DES]

The real question is who's responsibility it is to monitor the interior temperate alarms? In my 25 years of this business I have never seen any tower equipment shelter that didn't have some sort of temperature monitoring device. It could be something as simple as a thermostat tied to an intrusion alarm panel that could send a message to a central station or a pager. This isn't rocket science, but what I have seen in the past is that dispatch centers that are "tasked" to monitor these types of alarms usually don't know or forget what to do when these types of alarms show up. I think that is why the police chief isn't totally blaming Motorola, especially if it is his staff that was tasked to monitor the alarms. Will we see a follow up? Probably not.

 

[zz0468]

Even in the best of circumstances, these things will happen in spite of one's best efforts. I've seen redundant air conditioning failures at a site, high temp alarms triggered, and before the building can be cooled down, it reaches 180 degrees (desert, summer) inside and systems go down.

Preventing communications failures not only takes due diligence on the site facility, but also in a fault tolerant system design, effective backups, and user training. It would not surprise me at all to find out that their system contains adequate backups to maintain communications, but the users failed to take advantage, either because they don't know how, or they don't like the loss of functionality when operating in failsoft, or other conventional modes.

 

[scannerman5555]

it's like a blackout no radio's ............. I work for a taxi we had a repeater system use to go down in hot weather all the time so no more repeater now base to car only now ! we also got hit by lighting at the base lost everything plus the phones too also radio bases over heat some times too !

 

[balibago]

Dertroit! wow that's where that rapper Eminem is from. Hope neither he nor his group D2 were put at risk by this. Is there anything we can do to help?

 

[Raccon]

Follow up:

5 days later, Detroit Police have no answers on dispatch system crash | The Detroit News

[...]

Craig also said he was surprised and “appalled” when he asked when the backup communications system had last been tested, and was told it had never been tested.

 

[DaveNF2G]

Having seen the documentary Burn, I have to wonder whether the Detroit police are as bad off as the fire department.

According to the documentary, Detroit is broke.

 

[greenthumb]

For $1.6 million to maintain the system, one would think that Motorola would drop in any network monitoring (MOSCAD) equipment needed to take on that responsibility if the customer didn't buy it outright. I've seen that happen more than once, and it's surprising if they didn't have it there anyway. In order to make an entire simulcast system go dark like that, you'll need to hose the prime site, so if they're not going to monitor remote sites in the system for temperature alarms to save a $10 thermostat from Home Depot and a MOSCAD RTU per site...fine...but at least monitor the prime site! Geez...

I think there are still some questions here - does the site have a temperature alarm? Is it monitored? Who was responsible for monitoring it if it is monitored, and why didn't they respond to the alarm? If it's Motorola GNMO, Detroit deserves a credit for (at least) one year of GNMO monitoring for what they've gone through here.

 

[dcr_inc]

As They say: You can buy better, but you'll never pay More..

Even got 'em on the backup system.. A Total reaming... Go Moto !!

 

[PJH]

You can even have moscad failures that will still not show the site in trouble status. Rare, but does happen. We had a site that went dark yet the alarms were still all "green".

If a human invented it, it can fail.

 

[fdscan]

Having seen the documentary Burn, I have to wonder whether the Detroit police are as bad off as the fire department.

According to the documentary, Detroit is broke.

I was gonna say...

 

[mmckenna]

Preventing communications failures not only takes due diligence on the site facility, but also in a fault tolerant system design, effective backups, and user training. It would not surprise me at all to find out that their system contains adequate backups to maintain communications, but the users failed to take advantage, either because they don't know how, or they don't like the loss of functionality when operating in failsoft, or other conventional modes.

This is very true and a valid point. For many public safety folks, the radio is just another tool, nothing more, nothing less. As long as it works, they don't give a krap about it. They won't do training, and when they do, they screw around and don't pay attention because after all, it's -just- a radio. But the first time that radio doesn't do exactly what they want exactly when they want it, suddenly the radio system is "junk". Been there, done that. Try and point out that it's a training issue and you become the bad guy. You have to remember, that most officers, fire fighters, sheriff deputies, etc are in that line of work because that is what they are good at. It doesn't automatically make the radio experts, although there are those who will want you to think that.

On the other hand, many years ago I used to pay Motorola $36,000 a year for "maintenance" on a single site 5 channel trunked system. For that $3000 a month, we'd play phone tag trying to track down who the local MRSS shop was that year. They'd change who our shop was every 12 months and usually forget to tell us. Good luck finding your local Motorola rep anytime that wasn't M-F 9-5. When we did finally find the "MRSS shop de-jour" they'd always send out someone who had no clue about trunking systems. They'd knock the system into failsoft just about every time. They'd do a "complete" PM, we think. When something actually broke, we'd have to wait -days- to get help, because we "weren't a priority". After a few years of this we dumped our service contract. We'd get better service by doing a time and materials request from the closest shop. We'd usually get the same guy who knew our system and was smart enough not to knock it down. We'd pay less, too. When it came time to replace the system, Motorola was standing there with there hand out wanting money. They couldn't understand why we didn't want to go with Motorola! Bought a different brand that actually treated us like a paying customer and respected us.

If Motorola was being paid $1.6 million a year to maintain the system and it failed, they certainly played the game, and now likely will try to pass the blame to someone else. They don't get that money for sitting on their hands! This is so similar to what we went through.. Some Motorola sales drone sold Detroit this system and service contract and then went and played golf. Getting paid that much money to maintain a system comes with responsibilities. It's obvious that Motorola didn't take those responsibilities seriously. They seem to have a pretty good track record of this sort of stuff the last 10 years or so, take the money and run, screw public safety, maximize profits at the taxpayers expense.

 

[b7spectra]

Folks, we are talking about Detroit! The heat alarms were installed, but someone stole them!

 

[PJH]

For most people reading this...

In general, Motorola itself supplies the hardware and software for all systems - large and small. 9 out of 10 times it is the local Motorola Service Shop (MSS) that actually installs and maintains the equipment. Such MSS is NOT owned or operated by Motorola. They are local or regional independantly owned and operated businesses.

Years back, they were owned and/or managed by Motorola and operated locally, but thats no longer the case. Some are allowed to do the work on high-tier radios/systems, others simply cater to the commerical business crowd. Some are better than others. In the end, it really isn't Motorola Solutions doing the work.

Now, Motorola Solutions does have people in the field to help out and install some of the complicated stuff alongside the MSS workers, but daily PM and other work is the responsbiliy of the MSS and the customer.

Now don't take it that Motorola Solutions is the golden child, but unless the customer is maintaning its own equipment, its really not their fault so to speak.

Motorola does/did have a requirement that owners of wide area trunked systems have a network interface to Moto Illinois monitoring NOC which provided realtime network support. This freed up customers from having to man their own NOC and watch for everything. I believe at the time we had ours, if you had your own NOC with 24/7 staffing it wasn't required but still recommended.

Still wouldn't have made a difference if the alarm equipment wasn't interfaced with anything or transmitted in this case.

 

[Josh]

The blame game always goes out when the system fails, and here the local shop or others are being blamed for "shotty maintenance" and what not.

The Detroit subsystem of the MPSCS has been up and running for around 8 years now. Yes system upgrades have occurred and maybe a single tower site brought down temporarily too.There 10 sites on this system scattered throughout the city and aside from planned system outages and what not , when was the last time it had a failure to this extent?

If you tally the math up, the reliability has been at 99.997% and I'd certainly dare anyone to find something else to operate as reliably 24/7 for 8 straight years. They claim a "2 hour" failure, but to me it sure seemed like an all-day thing maybe even a couple.

The issue here was the result of heat caused by a failed A/C unit. Electronic devices are rated to work up to a certain temperature reliably, then after that are prone to failure. These ratings are what separate consumer grade to professional grade and/or commercial grade. Radio equipment, especially that rated at 100% duty cycles and high wattages create a lot of heat. Any comm room is or should be air conditioned/ventilated to keep components from failing. I'm sure if the A/C went from "working fine" to failed, it wouldn't take long at all for one of those sealed up comm boxes to swelter inside and start dropping system components.

 

[greenthumb]

The issue is that this failure caused the -entire- simulcast system to go down, which caused emergency calls for service from the public to be put on hold for hours.

Is everything going to be available 100% of the time? No. I have seen Cisco access layer switches with an uptime of almost 10 years though, and the MTBF on some of them is on the order of 21 years. But anyway, this is why they use redundant switches, redundant routers, trunking technology, redundant backhaul, etc. - to keep things going when something fails. What 'availability' means on a complex system like this is nebulous - do you count individual site outages, channel outages, etc. in the calculation of availability? Just with that hit, the Detroit system availability is only 4-9s, so if you take into account normal maintenance outages and other minor failures, their numbers for this year will be pretty bad.

We can only speculate on why this happened because we don't have all of the facts. I do know, however, that most equipment shelters are shipped with redundant air conditioning units in them, so if one fails, an alarm is sent and the other one carries on. If they both fail, an alarm is sent and someone goes out to fix it - no matter what time of the day it is. If alarms don't make it to where they're supposed to go (which Motorola's MOSCAD goes two places - the NFM server and UEM) then there are other issues with the Network Management system design. The MOSCAD traps to UEM are SNMP, and SNMP supports TCP so it should be configured for guaranteed delivery. If you can't rely on your network management system, you can't rely on anything. Motorola does not require any GNMO monitoring of systems - they can't force anyone to purchase that service. The CO statewide system does not have GNMO watching the whole thing, and it's one of the largest systems out there. They rely upon MOSCAD and UEM to send messages to the technicians in the field responsible for the remote sites and/or master sites.

In the end, when someone pays a vendor to monitor a system and respond to alarms to keep things running, that's exactly what should happen. It looks like Detroit's maintenance contract is with Motorola, so if Motorola subcontracts an MSS to do the work, it is still up to Motorola to maintain the SLA defined in the contract terms and Motorola is directly responsible for the competence of the selected MSS and will be liable for any breech of the contract SLA.