PA 911 Outage

[smorris]

Ok, so as many of you probably know by now, PA announced an intermittent 911 outage this afternoon. I don’t understand how it’s a statewide issue. Every county, or groups of counties, have their own PSAP. I just assumed each PSAP operated, or could operate independently of each other. I figured the phone companies forwarded 911 calls for a set geographical area for mobile users, and local phone exchanges for landlines to each PSAP. I’m not sure how the state figures into all this, and how we could have a system that could crumble like a cookie, affecting every PSAP in the state. Anyone have knowledge about how this system actually works?

 

[EMT1301]

Over the last few years, the Commonwealth transitioned to a single statewide Next Generation 911 system. 911 Program | PA Emergency Management Agency

 

[trentbob]

In my area, most police departments districts have a non-emergency number to call in calls that are not urgent. It goes right to County dispatch. There are also County dispatcher numbers that they put out that you could call. The lines are taped and you are talking to one of the dispatchers it's just for non-emergency. That's how they got by in my County.

I was at the market when the alert came over the phone.. people did kind of feel like they were on their own if something happened.

One guy said in the store that if you have an LTC today's the day to use it LOL. I have the non-emergency number for County dispatch programmed into my phone.

 

[cavmedic]

Ok, so as many of you probably know by now, PA announced an intermittent 911 outage this afternoon. I don’t understand how it’s a statewide issue. Every county, or groups of counties, have their own PSAP. I just assumed each PSAP operated, or could operate independently of each other. I figured the phone companies forwarded 911 calls for a set geographical area for mobile users, and local phone exchanges for landlines to each PSAP. I’m not sure how the state figures into all this, and how we could have a system that could crumble like a cookie, affecting every PSAP in the state. Anyone have knowledge about how this system actually works?

—> click me <—

 

[smorris]

Over the last few years, the Commonwealth transitioned to a single statewide Next Generation 911 system. 911 Program | PA Emergency Management Agency

Anytime you see “internet protocol-based, that’s how the cookie crumbles, in my opinion. I don’t remember any 911 issues like this in Monroe County prior to this new system. I think each PSAP should be able to operate independently, because apparently, there is no fail safe. I remember this happening in NYC a couple years ago. 911 was down in the city the whole day. Luckily, I know the direct number to Monroe Co’s PSAP. 570-992-9911. We use it to report dead deer on the road, malfunctioning traffic lights, trees down, etc.

 

[cavmedic]

In the past, there wasn’t Social
Media platforms to disseminate information to the public like there is now.

Phone companies still had issues with 911 trunk lines, but the average person didn’t typically know when it happened.

 

[bunangst]

Anytime you see “internet protocol-based, that’s how the cookie crumbles, in my opinion. I don’t remember any 911 issues like this in Monroe County prior to this new system. I think each PSAP should be able to operate independently, because apparently, there is no fail safe. I remember this happening in NYC a couple years ago. 911 was down in the city the whole day. Luckily, I know the direct number to Monroe Co’s PSAP. 570-992-9911. We use it to report dead deer on the road, malfunctioning traffic lights, trees down, etc.

Most 911 calls these are coming from cell phones and are already IP based (LTE). Traditional copper telephone lines and switching infrastructure that supported the original 911 systems are expensive to maintain when most customers are looking to high speed, high bandwidth internet traffic.

 

[Dispatcher308]

Ok, so as many of you probably know by now, PA announced an intermittent 911 outage this afternoon. I don’t understand how it’s a statewide issue. Every county, or groups of counties, have their own PSAP. I just assumed each PSAP operated, or could operate independently of each other. I figured the phone companies forwarded 911 calls for a set geographical area for mobile users, and local phone exchanges for landlines to each PSAP. I’m not sure how the state figures into all this, and how we could have a system that could crumble like a cookie, affecting every PSAP in the state. Anyone have knowledge about how this system actually works?

Its called NEXTGEN 911, where computers, routers, switches and firewalls are involved, sometimes stuff breaks or goes bad. It is very unfortunate that this happened, I hope there will be an AAR. putting all of our eggs in ONE basket is not acceptable anymore.

 

[cavmedic]

Its called NEXTGEN 911, where computers, routers, switches and firewalls are involved, sometimes stuff breaks or goes bad. It is very unfortunate that this happened, I hope there will be an AAR. putting all of our eggs in ONE basket is not acceptable anymore.

Or don’t upgrade on a Friday afternoon……. Shall see what was “changed” cause yeah

 

[bunangst]

Or don’t upgrade on a Friday afternoon……. Shall see what was “changed” cause yeah

IT rule #1

 

[bunangst]

Its called NEXTGEN 911, where computers, routers, switches and firewalls are involved, sometimes stuff breaks or goes bad. It is very unfortunate that this happened, I hope there will be an AAR. putting all of our eggs in ONE basket is not acceptable anymore.

These things aren't in "ONE" basket. All 911 PSAP equipment and connections I've work on are redundant at least N+1. This was an intermittent outage. If I had to guess, either DNS or Routing related.

 

[u2brent]

Its a good thing, redundancy with intermittency. Didn't they remove the wauwei.

 

[Dispatcher308]

These things aren't in "ONE" basket. All 911 PSAP equipment and connections I've work on are redundant at least N+1. This was an intermittent outage. If I had to guess, either DNS or Routing related.

Ok well if the system was truly redundant as you say it should be, why did we have a failure? Shouldn't it have been seamless and no one would have been the wiser, and the Network Administrators would have seen the device was inoperable and then put in a trouble ticket for it to be fixed, NOT TAKING DOWN THE WHOLE DAMN STATE INTERMIITENTLY?

 

[u2brent]

I guess they didn't/don't do "Ask Yourself"
why am I doing this,
is this the correct time,
am I qualified,
am I authorized,
what is the back out plan.
who do I call, are there current alarms..
blah, blah

 

[maus92]

Our county PSAP lost a router - or THE router that sends 911 calls to our town last month. It took over a day for it to be repaired / replaced (luckily I wasn't working those days as apparently it was a cluster). You would think there would have been a backup router in place like the radio system has, but apparently not. I'm unclear if the E911 system is a statewide thing in Maryland, but the failure was located at a state node where that the county colocates some equipment.

 

[mmckenna]

I'm unclear if the E911 system is a statewide thing in Maryland, but the failure was located at a state node where that the county colocates some equipment.

Not trying to justify NG911 or say that the failure is OK. But similar to what you are saying, the old E911 system was far from perfect.

An outage is an outage, and if someone is trying to make a 911 call and it doesn't go through, it's a big deal. If that outage is statewide or just a local PSAP, the issue is the same.

A lot of those old E911 routers were reliant on 56K phone company circuits, or fractional T1's. The'd fail. The skill to work on those old switched circuits is quickly retiring. I hired a tech that used to work for the local exchange carrier as the 911/HiCap guy for the area, and he didn't get much time off he was so busy. Keeping those old circuits running is going to get harder and harder.

LEC's are not installing those old circuit switched circuits anymore. They want to land fiber and hand off as IP. That's happening across the country.

Even on the E-911 side, a router at the Tandem switch can fail and take down multiple PSAP's. Getting hard to find guys that know how to work on that stuff.

Properly set up, a NG911 PSAP should have multiple links to the core, and there should be more than one core. Maybe PA hasn't completed the roll out of the system yet.

NG911 when fully built out, should be more reliable, but I think we're not there yet.

 

[smorris]

Not trying to justify NG911 or say that the failure is OK. But similar to what you are saying, the old E911 system was far from perfect.

An outage is an outage, and if someone is trying to make a 911 call and it doesn't go through, it's a big deal. If that outage is statewide or just a local PSAP, the issue is the same.

A lot of those old E911 routers were reliant on 56K phone company circuits, or fractional T1's. The'd fail. The skill to work on those old switched circuits is quickly retiring. I hired a tech that used to work for the local exchange carrier as the 911/HiCap guy for the area, and he didn't get much time off he was so busy. Keeping those old circuits running is going to get harder and harder.

LEC's are not installing those old circuit switched circuits anymore. They want to land fiber and hand off as IP. That's happening across the country.

Even on the E-911 side, a router at the Tandem switch can fail and take down multiple PSAP's. Getting hard to find guys that know how to work on that stuff.

Properly set up, a NG911 PSAP should have multiple links to the core, and there should be more than one core. Maybe PA hasn't completed the roll out of the system yet.

NG911 when fully built out, should be more reliable, but I think we're not there yet.

I don’t know why it has to be such a complicated, cumbersome system. Why can’t the TELCO’s simply forward 911 calls to a dedicated trunk at each designated PSAP? Or is that just too simple? I know they want all these extra, advanced features, but is it really worth all these points of failure? In some ways, it was easier when you had to dial a seven digit number for emergencies.

 

[bunangst]

I don’t know why it has to be such a complicated, cumbersome system. Why can’t the TELCO’s simply forward 911 calls to a dedicated trunk at each designated PSAP? Or is that just too simple? I know they want all these extra, advanced features, but is it really worth all these points of failure? In some ways, it was easier when you had to dial a seven digit number for emergencies.

System has to be capable of routing to other local PSAPs in case of failures or over call handling capacity. System has to be capable of transferring calls to other PSAPs for 3rd party reports, active situation hand offs, wireless territory overlaps, etc. The original POTS 911 lines could handle all of that. The common denominator in the entire system (new and old) was designed by humans...system bugs aren't bugs until they show themselves.

 

[mmckenna]

, it was easier when you had to dial a seven digit number for emergencies.

All PSAP's have a 7 digit emergency number. It's a real good idea to have those saved in your cell phone.
They also have a non-emergency 7 digit number. You'll make a lot of points with the dispatchers if you use that over 911 in non-life threatening situations.

@bunangst described it well.

The old systems (E911) were not simple. The ability to reroute calls was always there. We could always reroute our calls to our County PSAP on that system. Handling the actual location info required data circuits as it couldn't be done over the phone line itself.

The "simple" phone services died years ago. Most are not circuit switched anymore, just about everything is now packet switched.

 

[talviar]

I don’t know why it has to be such a complicated, cumbersome system. Why can’t the TELCO’s simply forward 911 calls to a dedicated trunk at each designated PSAP? Or is that just too simple? I know they want all these extra, advanced features, but is it really worth all these points of failure? In some ways, it was easier when you had to dial a seven digit number for emergencies.

The Phone company is no longer upgrading/replacing the TELCO switches at the CO capable of handling 911 Trunks. 911 Trunks are CAMA Trunks (aka PBX trunks) and date back to the very early days of Office multi line telephony.
By todays telephony standards this is Antique technology. Trunks basically ride copper via the "last mile" from the CO to the 911 Center. Currently the folks making the VOIP phone back end equipment won't do much regarding 911 due to "Liability" Considering all the other items that NG 911 has brought to the mix and brings to the mix, it is using IP to deliver 911 Calls, Text to 911 calls along with the ANI/ALI provided to the dispatchers. The big thing that NG911 brought to the table is ensuring the call goes to the correct 911 Center. With Cell Phones- its a big issue. Prior to NG911, the 911 Telco System was not capable of using the GPS Coordinates to against an accurate GIS system route calls. Call routing was done by sectors on the Cell site. If this sector had calls that primarily would need to go to 911 Center X thats where the calls went. Sometimes Center X would have to transfer the Call to Center Y as the call was in the other county.

Lastly NG911 wasn't created because people in 911 thought Hey we should totally disrupt 911 and redo the entire system. NG911 was created due to FEDERAL MANDATE. Trying to continue doing 911 on a per county basis for all equipment is very expensive. Even with the old TDM Trunks the phone system running 911 was capable of running a small central office, or a large business. Hence the "shared systems" and more eggs in one basket.

While I can't say specifically say what took out 911 that evening- in a nut shell it was a software bug hitting multiple pieces of equipment in a cascade fashion. The same stuff that happened in the early days of 911 on TDM routing.....