Starcom 21 Down

[MikeyC]

Just caught an announcement on ISP 06-A that "District Chicago radios and computers are down" and that "Pontiac will be taking District 5 emergency traffic only".

This is in addition to the ISP CAD system being down systemwide I think (At least for districts dispatched from Sterling, Pontiac and Chicago)

 

[Blunt630]

Lots of issues lately

 

[werinshades]

Just caught an announcement on ISP 06-A that "District Chicago radios and computers are down" and that "Pontiac will be taking District 5 emergency traffic only".

This is in addition to the ISP CAD system being down systemwide I think (At least for districts dispatched from Sterling, Pontiac and Chicago)

Yes...hearing District 1 (Sterling) taking District-Chicago status updates and District 6 (Pontiac) taking District 5 status updates.

 

[werinshades]

....and...just like a snap of the fingers, we're back!

 

[werinshades]

It appears whatever issue occurred, the consoles lost connectivity with the south sites. IDOT, ISP ,DuCom were all affected south. but mobiles were working Identical issue as before.

Occurred again...past hour now. Site 1-002 is trunking, mobiles calling in, no response. Consoles lost connectivity to south side. Site 1-001 not trunking south.

 

[JD21960]

Starcom21 down

SITE 149. I had my G5 Pager out at 4:35PM today and heard the ITHA HELP TRUCKS on RT-88 notifying dispatch that their radios were showing "SITE TRUNKING" and they were having trouble with them. I had no problems monitoring any of it, ha.

 

[werinshades]

I had my G5 Pager out at 4:35PM today and heard the HELP TRUCKS on 88 notifying dispatch that their radios were showing "SITE TRUNKING" and they were having trouble with them. I had no problems monitoring any of it, ha.

This has been a system problem in the past. Car's are going 10-41/10-42 and asking "is anyone hearing me?". I don't know the technical term behind it, but Site 1-002 isn't able to dispatch from the console. I can receive Site 1-001 from my location and it's not showing a control channel. IDOT, Metra, Pace all use Site 1-002 and it's the same problem. I'll be in bed soon, and I'm sure it'll be fixed by the morning.

 

[rbrtklamp2]

All seems to be well now.

Sent from my SM-N960U using Tapatalk

 

[werinshades]

All seems to be well now.

Sent from my SM-N960U using Tapatalk

Yes..this morning all is back to normal. 4th time I'm aware this has happened. As it was mentioned by Johnny Dollar, ITHA HELP trucks reporting their radios were showing Site Trunking, I'll assume this was all related. Site Controller failure? Lost connectivity with District-Chicago? Radios went into Fail Safe mode?

 

[KE4ZNR]

I don't know the technical term behind it, but Site 1-002 isn't able to dispatch from the console.

It is called "Site Trunking".
We made this video here locally to show what terms like "Site Trunking" and "Failsoft" mean in regards to Trunking Systems...even though it is for my system it is applicable to Starcom 21 as well:

https://youtu.be/p0RThgWZ5oU

If you make it to the end of the video you will see a familiar name in the credits

Marshall KE4ZNR

 

[werinshades]

It is called "Site Trunking".
We made this video here locally to show what terms like "Site Trunking" and "Failsoft" mean in regards to Trunking Systems...even though it is for my system it is applicable to Starcom 21 as well:

https://youtu.be/p0RThgWZ5oU

If you make it to the end of the video you will see a familiar name in the credits

Marshall KE4ZNR

We know that guy! Thanks Marshall for creating a very informative video. Yes, Site Trunking is what some of the users reported and those are the symptoms. What would be the reason for this occurring primarily isolated to one site?

 

[KE4ZNR]

We know that guy! Thanks Marshall for creating a very informative video. Yes, Site Trunking is what some of the users reported and those are the symptoms. What would be the reason for this occurring primarily isolated to one site?

Loosing connectivity to the rest of the system at the referenced site. Various ways it can happen but the main thing to know is that the biggest change is that the console positions at dispatch won't work for the time being and dispatchers will have to revert to using handheld radios until normal trunking is restored.

In an absolute worst case scenario where the world has ended and our system is no longer on the air at all we have contingency plans which allow us to use backup resources on the statewide VIPER system or our neighboring Wake County radio system. We also provide the same capability to VIPER and to Wake County with backup talkgroups on our system.

On the system I help maintain/manage we hold a monthly system test where we bring our system down into either site trunking or failsoft conditions to get our 911 centers and our users in the field comfortable working with the system in a reduced operating environment.
The goal is to (hopefully) prevent panic when a real site trunking or failsoft event occurs.

Even with the above referenced monthly test we still have users that freak (freq?) out when the test occurs and we end up having to reassure them that things will be ok.

Marshall KE4ZNR

 

[werinshades]

Loosing connectivity to the rest of the system at the referenced site. Various ways it can happen but the main thing to know is that the biggest change is that the console positions at dispatch won't work for the time being and dispatchers will have to revert to using handheld radios until normal trunking is restored.

In an absolute worst case scenario where the world has ended and our system is no longer on the air at all we have contingency plans which allow us to use backup resources on the statewide VIPER system or our neighboring Wake County radio system. We also provide the same capability to VIPER and to Wake County with backup talkgroups on our system.

On the system I help maintain/manage we hold a monthly system test where we bring our system down into either site trunking or failsoft conditions to get our 911 centers and our users in the field comfortable working with the system in a reduced operating environment.
The goal is to (hopefully) prevent panic when a real site trunking or failsoft event occurs.

Even with the above referenced monthly test we still have users that freak (freq?) out when the test occurs and we end up having to reassure them that things will be ok.

Marshall KE4ZNR

This appears to be occurring frequently at one site, the one I primarily monitor so I notice the issue immediately. Other Starcom monitors are hearing them through other sites, so it appears radios are switching to other control channels if their location is within the range. I didn't hear any transmissions when I scanned conventionally the 700 Mhz frequencies either. This has been the behavior the 4 times I have seen this happen.

 

[Blunt630]

DuPage site was going in and out. Cook South and North seemed fine.

 

[werinshades]

DuPage site was going in and out. Cook South and North seemed fine.

Site 1-002 was transmitting a data signal, but the dispatch consoles were unable to talk to the mobiles. Mobiles were checking in and not getting responses. Site 1-001 was not transmitting a data signal from my location (Midway Airport).

This morning, all sites are back on line and normal dispatch operations have resumed. The District-Chicago North Dispatcher was doing radio checks about 0910 Hrs. as no one was responding. Now she can be heard. I don't know if this was "packet failure" as described in the video or something else going on.

When the failure does occur, it appears to be isolated at Site 1-002, then Site 1-001 follows. The rest of the system sites do not appear to be affected as many hear report they hear everything, but are monitoring different sites.

 

[Blunt630]

I grew up next to midway... Go ole Archer Park

 

[werinshades]

I grew up next to midway... Go ole Archer Park

West end boy here...Garfield Ridge.

 

[MikeyC]

On the Type 2 system I monitored years ago in Southern California when trunking would fail the users would fail over to a pre-programmed frequency on the trunked system and ideally use that. Is there a similar concept on P25 systems? Granted with only a handful of sites to cover a whole county and 10 - 20 frequencies per site it's probably a bit easier than hundreds of sites some only with 3 frequencies.

 

[KE4ZNR]

On the Type 2 system I monitored years ago in Southern California when trunking would fail the users would fail over to a pre-programmed frequency on the trunked system and ideally use that. Is there a similar concept on P25 systems? Granted with only a handful of sites to cover a whole county and 10 - 20 frequencies per site it's probably a bit easier than hundreds of sites some only with 3 frequencies.

Yes it's called Failsoft. The video I linked earlier in this thread goes into it a little but Failsoft occurs when the Control Channel is lost systemwide and the system reverts to conventional mode where users are assigned one of the system frequencies in conventional mode.

Marshall KE4ZNR

 

[Blunt630]

Anyone know which Department is responsible for Starcom and maintaining Starcom?