EMCOMM and public safety systems

[mmckenna]

GETS is free. WPS is $4.50 a month per line, and up to $0.75/min per call. You can stack WPS and GETS, but then there are other, carrier-specific issues.

AT&T has never charged us monthly for WPS. They did charge the activation. Might depend on the contract, though. I've got WPS on the cell phones for all my staff, and they each have a GETS card. Cheap insurance.

 

[AK4FD]

I actually just went to a meeting a few months ago here in NC that was put on by the area ARRL Manager & the state Emerg Mgr contact person. The technical name is now known as AuxComm. It was the annual meeting for the NC area and was quite informative on what AuxComm is and what is all entailed within it. It appears to be combining the roles of ARES & RACES into one subject heading now. A lot of areas near me don't have many RACES teams anymore anyways so usually they have to pull an ARES member in there for help. But AuxComm has several courses (COML, etc) that Emerg Mgmt & FEMA puts on that interested persons can take to assume more higher-level roles in the emergency communications field should an incident occur. It's worth looking into a little more if interested. I'm not sure where the name EmComm came from but FEMA & State Emerg Mgmt officials only know & use AuxComm from what I've seen so far...

David

 

[jim202]

Having been involved in public safety communications for many years, both in designing their systems and picking up the pieces after a hurricane has gone through, many interesting facts and details have been learned. To put it bluntly, just about any communications system along the Gulf Coast is subject to damage by major storms. Even though they are designed to take the beating, things like antennas being moved on their mounts, microwave dishes moved off their path alignment, generators that have run out of fuel because you can't get to them, towers that literally have failed and the list goes on.

In many cases the repairs to these radio systems may take weeks or longer to repair and get them back on line. Flooding is a major concern in the design of the radio site locations and the equipment elevations above the ground. There are some locations where the platform may be as high as 20 feet or more above the ground. It also is where the generators are mounted. Getting diesel fuel to these locations has been by boat and 5 gallon containers.

During the tower outages, radio coms can be via the simplex operation on the national interop channels. But this all needs to have a person with some common sense steering the boat so to speak. I have been involved in many instances of communication damage like this. At the start it is just pandemonium. But after the real understanding of where the failures are, plans can be implemented to work around the problem areas.

The next problem comes in when the Red Cross starts setting up shelters. Many of the people involved with the Red Cross activities rely on cell phones and maybe even the state radio systems. When these fail or become overloaded, now they turn to the ham radio community to help them out. The problem comes in when you start to try and use a simplex operation and can't communicate very far due to lack of a decent antenna. In some locations, external 2 meter antennas have been installed to allow connection to these antennas to allow reasonable communications between the pre designated shelters.

They also even have some ham repeaters set up with emergency power and rugged antennas to survive the storms we get along the Gulf Coast. Not saying this is the case in many locations, but where there is a good relation between the ham radio groups and the red Cross, there are pre planed activities already in place.

In your areas, this may not be the case. Depends on the type of storms you may have to work through and the managers of the different agencies that may be looking for communication help. It needs to be planned ahead of time and not when the crap hits the fan.

Your local ham clubs need to get with the agencies that may need any help with their communications. Sit down with them and talk about what problems they have had in the past and if they are looking for a backup option. The Salvation Army has their own communication systems, but it wouldn't hurt to contact them if your club is offering their abilities. Just remember that this is a commitment that you are offering, so make sure you have the bodies to support it. You may have 20 or 30 members, but only say 2 of them can drop what they are doing on short notice to come help. Then there is the issue of how long the support is going to be needed. It doesn't take long to burn out your people. Replacements will be required.

There is much more to this emergency communications backup. Walk slowly through the steps and make sure you have the bodies that will be required. Also remember that roads will be blocked, power will be out and you will need to have your own food and water for a number of days until the Red Cross and Salvation Army come in and get functional. Fuel for your vehicles will be in high demand and low to no supply. Diesel fuel will be much easier to come by than gasoline. Keep this in mind when trying to put a response plan into place.

Hope this provides some ideas for thought. It is not easy trying to just survive in an area that has had a hurricane go through or a tornado that has torn up an area.

 

[DaveNF2G]

Hams involved in localized emergency communications don't seem to think of HF as a possible solution to VHF/UHF range problems.

 

[AK4FD]

Well actually, most FEMA & AuxComm teams will use HF on emergency deployments just for that reason that it gets longer range. There are specific frequencies in each area that teams use on HF for communications and such.

 

[AK9R]

When you can cover an entire county (approx 400 sq miles) with a few well-placed 2m simplex stations, I don't see HF as being necessary. A useful tool to have at your disposal, but not necessary IMHO.

 

[AI7PM]

When you can cover an entire county (approx 400 sq miles) with a few well-placed 2m simplex stations, I don't see HF as being necessary. A useful tool to have at your disposal, but not necessary IMHO.

Mountainous terrain makes it a need to have in the box.

 

[AK_SAR]

A bit more info on the root cause of the CenturyLink 911 failure is available at:
CenturyLink's outage started in Denver, spread across country

As noted in the OP, there is no way amateur radio AUXCOMM could substitute for a 911 system. However, my concern is that this incident highlights the fragility of most other modern digital networks. Is FirstNet any more robust? How about P25 networks? The more feature laden these systems become, the complex they become. That very complexity leads to multiple potential points of failure. And, as the linked article shows, the more difficult it is to isolate and fix those problems. And, this appears to be a simple, garden variety hardware failure. What might happen if there were a malicious attack on FirstNet?

 

[jim202]

Most P25 networks are well designed and have redundancy already built into them. Single point of failure bringing a system down is not very common these days. You might have a single site go off the air, but generally there is enough overlap of adjacent towers that some communications will be present. To take a single site off the air from a well designed system, you have to really work at it.

The newer radio systems are using a microwave loop system, so that if one hop is taken out for what ever reason, then the data will switch and flow in the other direction maintaining communications. Battery banks are installed to allow for a long duration of operation to allow for a generator failure. This allows for time to pull a trailer mounted generator to the site and plug it in. Again in the newer systems, there is a manual transfer switch inside the equipment shelter to allow for an external generator to be plugged into the shelter and run the site.

The older systems that don't have this feature, then shame on those managers that have not starting a program to modify the electrical wiring to plan for this type of failure. It is easy to make the modification and really doesn't cost that much. The same thing needs to be done at the main 911 center to allow for generator maintenance during a long power outage. You need to be able to change the oil in the engine and just be able to service it. If you do not have the ability to have a second generator be brought on line easily, then stir the pot and get this implemented. You will have some very happy people if this ability is there.

Have been involved with a number of consulting companies over the years and this is a major push from all of them. If your consulting company is not bringing this to the table, you just might want to look around for a more experienced consultant.

Planning for flexibility is just a way of life for the public safety radio systems and equipment. Your staff needs to have an open mind and the ability to make changes as needed.

If your not sure what you may need or how ti get it put in place, get on the chat group here and post away. There are some real knowledgeable people here that can sure help out with suggestions.

 

[AK_SAR]

Most P25 networks are well designed and have redundancy already built into them. Single point of failure bringing a system down is not very common these days. You might have a single site go off the air, but generally there is enough overlap of adjacent towers that some communications will be present. To take a single site off the air from a well designed system, you have to really work at it.

I'm sure if we had asked CenturyLink on December 26, they would have said something similar about their 911 system.

Have been involved with a number of consulting companies over the years and this is a major push from all of them. If your consulting company is not bringing this to the table, you just might want to look around for a more experienced consultant.

CenturyLink has said the outage was due to a "third party network". Maybe they "might want to look around for a more experienced consultant"?

The point being, everyone will tell your their system is "well designed", "redundant", "has multiple paths", "fail safe", etc. Right up until actual events show they are not.

 

[AK9R]

CenturyLink has said the outage was due to a "third party network".

For the most part, the SAFE-T system in Indiana uses AT&T fiber or a dedicated microwave system for linking. There are parts of the state that AT&T doesn't serve and microwave isn't practical due to the hills (yes, there are hills in Indiana). So, some SAFE-T sites are linked using other carriers. When the EF4 tornado hit Henryville in 2012, the closest SAFE-T site went into site trunking and was that way for several days while the mom and pop telco that served the area figured out how to restore the fiber.

State DHS used sat phones and cell phones to keep in touch with their headquarters in Indianapolis. Some amateur radio was used on the ground, but not much for long distance.

 

[Project25_MASTR]

I'm sure if we had asked CenturyLink on December 26, they would have said something similar about their 911 system.
CenturyLink has said the outage was due to a "third party network". Maybe they "might want to look around for a more experienced consultant"?

The point being, everyone will tell your their system is "well designed", "redundant", "has multiple paths", "fail safe", etc. Right up until actual events show they are not.

It really depends still. Unfortunately, a 911 system and a LMR system have to different view points that depend on how they are affected by outages. Since the PSAP is connected to the CO, if the CO goes down the 911 system will typically go down and unfortunately as a customer, you have little to no control over the CO. LMR is a little different though as with planned alternates, it is more forgiving of failure. You are right, you can build in as much redundancy that you want but there is still an event that will bring it all crashing down.

Today you see heavy site overlap (typically to the N+1 level) in many systems, redundant equipment, redundant power sources, etc. When designing this stuff you really have to plan for failure as it is unreasonable to expect everything to remain 100% operational.

Here are three great examples of failures that were really were easily worked around.

1. During Hurricane Harvey, Brazoria county's simulcast system somehow became disconnected from the TxWARN core. It was a network issue, and I don't have more details other than that but the county's simulcast system was still operational but was operating in Site Trunking. Of course, all of the subscribers able to grab a site not in site trunking roamed to those sites but some where not able to do so. Simple fix until networking was established was patches between "conventional resources" and/or patches between "conventional resources" and the core on consoles still connected to the core to reestablish wide area talkgroups.
2. During Hurricane Harvey, high winds went through the Austin area and affected a portion of GATRRS. One of the microwave links started dropping and reconnecting due to wind shifting the dishes (over-engineered links with little to no literal wiggle room) which was affecting Blanco county's sites. The sites were going to alternate routes but that takes about 5 seconds for OSPF to reroute so for 5 seconds at a time the sites would go into Site Trunking. When the link would seemingly stabilize, the route would become preferred again as it was the shortest path and again reroute. The process would then repeat. Simple fix, unplug the unstable link from the switch (yes, this did eliminate the redundant option) until the wind died down and it could be reconnected (and a tower crew climb to tighten the dishes up).
3. Several sites in one of the VHF portions of GATRRS were occasionally going into failsoft last spring due to VHF tropospheric ducting. If interference is detected on a control channel, the control channel rolls. If the all of the control channels (4 typically designated in Motorola systems, TxWARN is an exception to this) detect interference...the site goes into failsoft. Enough overlapping coverage (each county in that region has roughly 2-3 overlapping sites) so it wasn't a huge deal as most subscribers simply roamed. Failure yes, but one of the issues you occasionally see in OBT trunking and sometimes even in 7/800 MHz.

Point being, don't rely on others for redundancy. If you need it, rely on yourself and have a workaround if all else fails.

 

[AK_SAR]

I don't want to give the impression that I don't like LMR. Quite the contrary, LMR is a great thing. While I'm hardly an expert on LMR, I do understand the basic concepts of how it works. And I do use our local "ALMR" (Alaska Land Mobile Radio) system. Our local volunteer SAR teams, and some key individuals on those teams, have been issued ALMR capable radios. (We also have regular VHF radios for tactical use.) We have a number of pre-defined "Incident Command" talk groups. When we do a mission or training exercise, we can contact Alaska State Trooper Dispatch, either by "Hail" or cell phone, and request use of one or more of those IC talk groups.

What I do want to point out is that as these systems (LMR, 911, FirstNet, whatever) become bigger, and more feature laden, they also become more complex. That complexity inevitably adds more potential failure points. And sometimes that complexity makes it more difficult to locate and fix those failures (as was apparently the case with CenturyLink).

Note that our local ALMR system seemed to do quite well in the recent M7.0 earthquake. See http://alaskalandmobileradio.org/Documents/Newsletters/2019/20190110_Vol13 Iss1.pdf for a discussion of how it performed. Do read the next to last paragraph, however. Our recent quake had strong shaking that lasted about 30 seconds. The 1964 M9.2 Alaska Earthquake had much more severe shaking that lasted 3-4 minutes! In 1964 there was much greater, and much more widespread damage to critical infrastructure. And while the Anchorage area has some overlap of coverage, due to our mountainous terrain much of the area covered by ALMR does not have multiple sites overlapping. When the next M9 "really big one" hits, ALMR and all other communications systems will be put to a very severe test.

Having multiple alternative means of communications is never a bad thing.

 

[mmckenna]

I don't want to give the impression that I don't like LMR. Quite the contrary, LMR is a great thing. …. Having multiple alternative means of communications is never a bad thing.

Properly programmed public safety radios should all have the DHS IFOG channels programmed into them. With any radio built in the last 15 years, there is plenty of room to fit them in. Issue is that agencies are either short sighted, or just lazy.
With the IFOG channels, that's a bunch of simplex channels that should be programmed into all public safety radios of the same band, regardless of agency or discipline.
Some of the channels can be run as repeaters. With pre-installed repeaters in strategic locations, a fall back communications resource is available by just changing channels.
Most large agencies have (or should have) portable repeaters available that can be put up in a short amount of time. With patching systems, it's easy to link VHF low, VHF High, UHF, 700, 800 as well as cell, satellite, HF, wireline, military, air band, IP etc. all together.

So, simplex or repeated communications that are available to all public safety agencies already exist without bringing in volunteers. It just takes some additional programming work by the radio shops.

 

[AK9R]

So, simplex or repeated communications that are available to all public safety agencies already exist without bringing in volunteers. It just takes some additional programming work by the radio shops.

And, some training and drills so that personnel are familiar with how to find those channels in their radios and how they can be used.

 

[AK_SAR]

Properly programmed public safety radios should all have the DHS IFOG channels programmed into them. With any radio built in the last 15 years, there is plenty of room to fit them in. Issue is that agencies are either short sighted, or just lazy.
………..
So, simplex or repeated communications that are available to all public safety agencies already exist without bringing in volunteers. It just takes some additional programming work by the radio shops.

To the best of my knowledge, all Alaska ALMR radios have a standardized VHF channel line up programmed in, for emergency interoperability. Contrary to your somewhat snarky remark, we are neither "..short sighted, or just lazy."

"Alternative means of communications" means a great deal more than just simplex or repeater VHF. For example, here in the Anchorage area, volunteers are currently building out an AREDN MESH network. When complete we will have nodes at the State EOC, the Anchorage Muni EOC, and (eventually) outlying areas to the North and South. We are currently in the process of installing a node at the SAR equipment cache at State Trooper Headquarters. Capabilities include Winlink, VOIP phones, video feed, etc. By policy, all nodes are installed to a high standard, and have backup power. For more info on the Anchorage MESH network see AARC MESH Network (Note that this is a work in progress, and that page needs to be updated with regards to what we currently have installed.)

Also, keep in mind that during a major disaster, even if LMR and other systems stay fully operational, the sheer volume of traffic can become a problem. (This was noted in the previous ALMR link I sent.) Alternative links such as a MESH network can be used to carry important but lower priority traffic, and thus take some of the load off of primary public safety systems.

 

[mmckenna]

To the best of my knowledge, all Alaska ALMR radios have a standardized VHF channel line up programmed in, for emergency interoperability. Contrary to your somewhat snarky remark, we are neither "..short sighted, or just lazy."

That's good, and what I'd expect for a state that's used to dealing with wide open spaces, natural disasters, and harsh weather.

And it wasn't meant as a snarky remark. It's from my own experience from working in the industry for a couple of decades. Many agencies don't program in the interop frequencies. It's an issue that gets talked about often, and the DHS is having a hard time getting it accomplished.

"Alternative means of communications" means a great deal more than just simplex or repeater VHF. For example, here in the Anchorage area, volunteers are currently building out an AREDN MESH network. ...

Yep, I'm familiar with that. A guy I worked with for many years and retired recently is helping build out a system like that in California. He was a network engineer, an amateur radio operator, belongs to the local search and rescue, and good all around guy. He's using his IP routing skills to do a lot with that system. Done right, it's a good system. They've used it during recent brush fires to establish communications into some remote area.

 

[BlueDevil]

GETS is free. WPS is $4.50 a month per line, and up to $0.75/min per call. You can stack WPS and GETS, but then there are other, carrier-specific issues.

It appears Verizon is charging me just over $20/month for WPS. Horrible investment. I will be canceling my subscription immediately. I can invest in a radio system and outfit all my family with a radio for that price.

 

[BlueDevil]

...With pre-installed repeaters in strategic locations, a fall back communications resource is available by just changing channels.

I very much like this idea however the repeaters should have the ability to be remotely turned off. I have had some horrible interference with geographical NIFOG repeaters when trying to setup portable repeaters on All-Hazard assignments.

 

[mmckenna]

It appears Verizon is charging me just over $20/month for WPS. Horrible investment. I will be canceling my subscription immediately. I can invest in a radio system and outfit all my family with a radio for that price.

Yeah, $20? You are getting hosed. Many agencies have been able to negotiate the costs way down.