PA 911 Outage

[smorris]

Ok, so as many of you probably know by now, PA announced an intermittent 911 outage this afternoon. I don’t understand how it’s a statewide issue. Every county, or groups of counties, have their own PSAP. I just assumed each PSAP operated, or could operate independently of each other. I figured the phone companies forwarded 911 calls for a set geographical area for mobile users, and local phone exchanges for landlines to each PSAP. I’m not sure how the state figures into all this, and how we could have a system that could crumble like a cookie, affecting every PSAP in the state. Anyone have knowledge about how this system actually works?

 

[EMT1301]

Over the last few years, the Commonwealth transitioned to a single statewide Next Generation 911 system. 911 Program | PA Emergency Management Agency

 

[trentbob]

In my area, most police departments districts have a non-emergency number to call in calls that are not urgent. It goes right to County dispatch. There are also County dispatcher numbers that they put out that you could call. The lines are taped and you are talking to one of the dispatchers it's just for non-emergency. That's how they got by in my County.

I was at the market when the alert came over the phone.. people did kind of feel like they were on their own if something happened.

One guy said in the store that if you have an LTC today's the day to use it LOL. I have the non-emergency number for County dispatch programmed into my phone.

 

[cavmedic]

Ok, so as many of you probably know by now, PA announced an intermittent 911 outage this afternoon. I don’t understand how it’s a statewide issue. Every county, or groups of counties, have their own PSAP. I just assumed each PSAP operated, or could operate independently of each other. I figured the phone companies forwarded 911 calls for a set geographical area for mobile users, and local phone exchanges for landlines to each PSAP. I’m not sure how the state figures into all this, and how we could have a system that could crumble like a cookie, affecting every PSAP in the state. Anyone have knowledge about how this system actually works?

—> click me <—

 

[smorris]

Over the last few years, the Commonwealth transitioned to a single statewide Next Generation 911 system. 911 Program | PA Emergency Management Agency

Anytime you see “internet protocol-based, that’s how the cookie crumbles, in my opinion. I don’t remember any 911 issues like this in Monroe County prior to this new system. I think each PSAP should be able to operate independently, because apparently, there is no fail safe. I remember this happening in NYC a couple years ago. 911 was down in the city the whole day. Luckily, I know the direct number to Monroe Co’s PSAP. 570-992-9911. We use it to report dead deer on the road, malfunctioning traffic lights, trees down, etc.

 

[cavmedic]

In the past, there wasn’t Social
Media platforms to disseminate information to the public like there is now.

Phone companies still had issues with 911 trunk lines, but the average person didn’t typically know when it happened.

 

[bunangst]

Anytime you see “internet protocol-based, that’s how the cookie crumbles, in my opinion. I don’t remember any 911 issues like this in Monroe County prior to this new system. I think each PSAP should be able to operate independently, because apparently, there is no fail safe. I remember this happening in NYC a couple years ago. 911 was down in the city the whole day. Luckily, I know the direct number to Monroe Co’s PSAP. 570-992-9911. We use it to report dead deer on the road, malfunctioning traffic lights, trees down, etc.

Most 911 calls these are coming from cell phones and are already IP based (LTE). Traditional copper telephone lines and switching infrastructure that supported the original 911 systems are expensive to maintain when most customers are looking to high speed, high bandwidth internet traffic.

 

[Dispatcher308]

Ok, so as many of you probably know by now, PA announced an intermittent 911 outage this afternoon. I don’t understand how it’s a statewide issue. Every county, or groups of counties, have their own PSAP. I just assumed each PSAP operated, or could operate independently of each other. I figured the phone companies forwarded 911 calls for a set geographical area for mobile users, and local phone exchanges for landlines to each PSAP. I’m not sure how the state figures into all this, and how we could have a system that could crumble like a cookie, affecting every PSAP in the state. Anyone have knowledge about how this system actually works?

Its called NEXTGEN 911, where computers, routers, switches and firewalls are involved, sometimes stuff breaks or goes bad. It is very unfortunate that this happened, I hope there will be an AAR. putting all of our eggs in ONE basket is not acceptable anymore.

 

[cavmedic]

Its called NEXTGEN 911, where computers, routers, switches and firewalls are involved, sometimes stuff breaks or goes bad. It is very unfortunate that this happened, I hope there will be an AAR. putting all of our eggs in ONE basket is not acceptable anymore.

Or don’t upgrade on a Friday afternoon……. Shall see what was “changed” cause yeah

 

[bunangst]

Or don’t upgrade on a Friday afternoon……. Shall see what was “changed” cause yeah

IT rule #1

 

[bunangst]

Its called NEXTGEN 911, where computers, routers, switches and firewalls are involved, sometimes stuff breaks or goes bad. It is very unfortunate that this happened, I hope there will be an AAR. putting all of our eggs in ONE basket is not acceptable anymore.

These things aren't in "ONE" basket. All 911 PSAP equipment and connections I've work on are redundant at least N+1. This was an intermittent outage. If I had to guess, either DNS or Routing related.

 

[u2brent]

Its a good thing, redundancy with intermittency. Didn't they remove the wauwei.

 

[Dispatcher308]

These things aren't in "ONE" basket. All 911 PSAP equipment and connections I've work on are redundant at least N+1. This was an intermittent outage. If I had to guess, either DNS or Routing related.

Ok well if the system was truly redundant as you say it should be, why did we have a failure? Shouldn't it have been seamless and no one would have been the wiser, and the Network Administrators would have seen the device was inoperable and then put in a trouble ticket for it to be fixed, NOT TAKING DOWN THE WHOLE DAMN STATE INTERMIITENTLY?

 

[u2brent]

I guess they didn't/don't do "Ask Yourself"
why am I doing this,
is this the correct time,
am I qualified,
am I authorized,
what is the back out plan.
who do I call, are there current alarms..
blah, blah

 

[maus92]

Our county PSAP lost a router - or THE router that sends 911 calls to our town last month. It took over a day for it to be repaired / replaced (luckily I wasn't working those days as apparently it was a cluster). You would think there would have been a backup router in place like the radio system has, but apparently not. I'm unclear if the E911 system is a statewide thing in Maryland, but the failure was located at a state node where that the county colocates some equipment.

 

[mmckenna]

I'm unclear if the E911 system is a statewide thing in Maryland, but the failure was located at a state node where that the county colocates some equipment.

Not trying to justify NG911 or say that the failure is OK. But similar to what you are saying, the old E911 system was far from perfect.

An outage is an outage, and if someone is trying to make a 911 call and it doesn't go through, it's a big deal. If that outage is statewide or just a local PSAP, the issue is the same.

A lot of those old E911 routers were reliant on 56K phone company circuits, or fractional T1's. The'd fail. The skill to work on those old switched circuits is quickly retiring. I hired a tech that used to work for the local exchange carrier as the 911/HiCap guy for the area, and he didn't get much time off he was so busy. Keeping those old circuits running is going to get harder and harder.

LEC's are not installing those old circuit switched circuits anymore. They want to land fiber and hand off as IP. That's happening across the country.

Even on the E-911 side, a router at the Tandem switch can fail and take down multiple PSAP's. Getting hard to find guys that know how to work on that stuff.

Properly set up, a NG911 PSAP should have multiple links to the core, and there should be more than one core. Maybe PA hasn't completed the roll out of the system yet.

NG911 when fully built out, should be more reliable, but I think we're not there yet.