6/10/2020 - Broadcastify Outage Followup - Switch back to Production

[blantonl]

Good morning everyone,

We're going to start the process to switch back the audio infrastructure from the backup servers to the production servers.

This process is starting at 7:36 AM CT. Updates to follow

FINAL UPDATE: 8:42 AM CT: Everything is back and running. We should be past all the drama. Thanks for all your patience and enjoy the rest of your week.

UPDATE: 7:57 AM CT: All master servers and relay servers have been moved back to the production environment. Feeds and listeners are recovering nicely. All management and alerting components have been reactivated. We'll be verifying the archive servers are up and recording feeds next. Feed providers who's feeds have not reconnected yet should REBOOT THEIR COMPUTERS and THEN restart their broadcast software. Reboot first... very important.

UPDATE: 7:36 AM CT: Starting to move the master servers. Feed providers will see their feeds go offline and then come back online in the next 15 minutes. All listeners will be down

 

[lmarc]

Honestly, why? The backup site seemed to work better and you're not under IBM's umbrella...

 

[blantonl]

Honestly, why? The backup site seemed to work better and you're not under IBM's umbrella...

Can you advise what "work better" means? Our primary infrastructure environment was working absolutely fantastic right up until IBM had a networking outage yesterday. We were cranking out 300K listeners last week on that without breaking a sweat.

The environment that we failed over to is extremely cost prohibitive for the use case. Our current production environment is a pre-negotiated under contract infrastructure for 800 TB of bandwidth a month. Do you have any idea how expensive that is for an on-demand infrastructure?

 

[lmarc]

It was a tongue in cheek comment, mainly referring to having to be under IBM's control. Sorry!

 

[jeepsandradios]

Lindsay thanks for the continued updates and dedication to the restoration process.

 

[blantonl]

It was a tongue in cheek comment, mainly referring to having to be under IBM's control. Sorry!

That makes sense - no worries

 

[cpfinlay]

I was impressed how well you performed your switch back and forth. Well played, sir!

 

[kayn1n32008]

@blantonl any idea what caused the outage?

 

[K9DAK]

@blantonl any idea what caused the outage?

Thinking this... 6/9/2020 - Broadcastify Outage

 

[kwco]

That makes sense - no worries

A now deceased friend (nermel) setup a Raspberry Pi in Chickasha, OK for Grady Co. Ok. It continued to work until the big outage and has never returned to service. I suspect the IP address was hard-coded in instead of resolving the address via DNS. Is it possible to share the address (normal address) so I can ping and figure out what that IP has changed to? I am attempting to have an IT friend modify the Pi setup to get it back online again. I'd hate to stop the service due to a configuration change. Thanks. (I have already tried a power reset recently.... no luck)