6/10/2020 - Broadcastify Outage Followup - Switch back to Production

Status
Not open for further replies.

blantonl

Founder and CEO
Staff member
Super Moderator
Joined
Dec 9, 2000
Messages
11,095
Location
San Antonio, Whitefish, New Orleans
Good morning everyone,

We're going to start the process to switch back the audio infrastructure from the backup servers to the production servers.

This process is starting at 7:36 AM CT. Updates to follow

FINAL UPDATE: 8:42 AM CT: Everything is back and running. We should be past all the drama. Thanks for all your patience and enjoy the rest of your week. ;)

UPDATE: 7:57 AM CT: All master servers and relay servers have been moved back to the production environment. Feeds and listeners are recovering nicely. All management and alerting components have been reactivated. We'll be verifying the archive servers are up and recording feeds next. Feed providers who's feeds have not reconnected yet should REBOOT THEIR COMPUTERS and THEN restart their broadcast software. Reboot first... very important.

UPDATE: 7:36 AM CT: Starting to move the master servers. Feed providers will see their feeds go offline and then come back online in the next 15 minutes. All listeners will be down
 
Last edited:

blantonl

Founder and CEO
Staff member
Super Moderator
Joined
Dec 9, 2000
Messages
11,095
Location
San Antonio, Whitefish, New Orleans
Honestly, why? The backup site seemed to work better and you're not under IBM's umbrella...
Can you advise what "work better" means? Our primary infrastructure environment was working absolutely fantastic right up until IBM had a networking outage yesterday. We were cranking out 300K listeners last week on that without breaking a sweat.

The environment that we failed over to is extremely cost prohibitive for the use case. Our current production environment is a pre-negotiated under contract infrastructure for 800 TB of bandwidth a month. Do you have any idea how expensive that is for an on-demand infrastructure?
 

kwco

Newbie
Joined
Dec 5, 2007
Messages
1
Location
Chickasha, OK
That makes sense - no worries :)
A now deceased friend (nermel) setup a Raspberry Pi in Chickasha, OK for Grady Co. Ok. It continued to work until the big outage and has never returned to service. I suspect the IP address was hard-coded in instead of resolving the address via DNS. Is it possible to share the address (normal address) so I can ping and figure out what that IP has changed to? I am attempting to have an IT friend modify the Pi setup to get it back online again. I'd hate to stop the service due to a configuration change. Thanks. (I have already tried a power reset recently.... no luck)
 
Status
Not open for further replies.
Top