Folks,
Our data center provider where a lot of our master and relay servers are hosted is experiencing connectivity problems - which is causing close to a systemwide outage on Broadcastify right now.
I'm going to see how long it takes for recovery before putting plans in action to fail over to other provides. Stay tuned for updates here.
Thanks for your patience and sorry for the problems.
FINAL UPDATE 10:38 PM CT: Root cause of the outage was networking configuration problems affecting IBM Cloud globally. Our audio servers (8 of them) are hosted on IBM Cloud’s Softlayer bare metal servers. We failed everything over to Amazon Web services and are currently running on that backup environment. Our plan is to move everything back to the production environment on Softlayer tomorrow morning around 7:00 AM CT. Thanks for everyone’s patience... our disaster recovery plan was tested and worked nicely.
UPDATE 8:55 PM CT: Coverage has returned to 94% of pre-outage feeds online. We're operating completely on backup systems. Later this week, presumably in the early morning hours, we will fail back over to the primary provider and set of servers. Next update will be what we know about the outage. Stay tuned.
UPDATE 8:12 PM CT: If you are feed provider, and your feed has not come back online, simply reboot your computer and start your broadcast client software back up. That should bring the feed back online
UPDATE 8:01 PM CT: Brining the archive servers back online now. 6,557 out of 7025 feeds are back online, and listeners are recovering quickly.
UPDATE 7:53 PM CT: About 90% of all the feeds are back online and broadcasting. All of the archive servers that record all the feeds are currently down and not capturing feeds right now but we plan on bringing those back up shortly.
UPDATE 7:09 PM CT: We have failed over all servers to a new provider and we're coming back up!
UPDATE 6:47 PM CT: We are failing over to a new provider. We will bring up the master servers first and then the relay servers. Updates to follow...
UPDATE 6:16 PM CT: We are in the dark right now with IBM (Softlayer) - their status pages are also down. We are beginning to unpack our contingency guidebook to fail over the complete audio system to another provider.
We'll wait for now to determine if we get a status before embarking upon that journey.
UPDATE 6:06 PM CT: This is a major outage at the data center affecting multiple customers. We have trouble tickets opened with Softlayer and are awaiting a response. Currently all feeds are down since all connectivity to the Dallas softlayer data center has been lost. Still no ETA.
UPDATE 5:40 PM CT: I have no ETA at this time - we are investigating the cause. It is a networking issue affecting multiple softlayer customers in their Dallas data center.
Our data center provider where a lot of our master and relay servers are hosted is experiencing connectivity problems - which is causing close to a systemwide outage on Broadcastify right now.
I'm going to see how long it takes for recovery before putting plans in action to fail over to other provides. Stay tuned for updates here.
Thanks for your patience and sorry for the problems.
FINAL UPDATE 10:38 PM CT: Root cause of the outage was networking configuration problems affecting IBM Cloud globally. Our audio servers (8 of them) are hosted on IBM Cloud’s Softlayer bare metal servers. We failed everything over to Amazon Web services and are currently running on that backup environment. Our plan is to move everything back to the production environment on Softlayer tomorrow morning around 7:00 AM CT. Thanks for everyone’s patience... our disaster recovery plan was tested and worked nicely.
UPDATE 8:55 PM CT: Coverage has returned to 94% of pre-outage feeds online. We're operating completely on backup systems. Later this week, presumably in the early morning hours, we will fail back over to the primary provider and set of servers. Next update will be what we know about the outage. Stay tuned.
UPDATE 8:12 PM CT: If you are feed provider, and your feed has not come back online, simply reboot your computer and start your broadcast client software back up. That should bring the feed back online
UPDATE 8:01 PM CT: Brining the archive servers back online now. 6,557 out of 7025 feeds are back online, and listeners are recovering quickly.
UPDATE 7:53 PM CT: About 90% of all the feeds are back online and broadcasting. All of the archive servers that record all the feeds are currently down and not capturing feeds right now but we plan on bringing those back up shortly.
UPDATE 7:09 PM CT: We have failed over all servers to a new provider and we're coming back up!
UPDATE 6:47 PM CT: We are failing over to a new provider. We will bring up the master servers first and then the relay servers. Updates to follow...
UPDATE 6:16 PM CT: We are in the dark right now with IBM (Softlayer) - their status pages are also down. We are beginning to unpack our contingency guidebook to fail over the complete audio system to another provider.
We'll wait for now to determine if we get a status before embarking upon that journey.UPDATE 6:06 PM CT: This is a major outage at the data center affecting multiple customers. We have trouble tickets opened with Softlayer and are awaiting a response. Currently all feeds are down since all connectivity to the Dallas softlayer data center has been lost. Still no ETA.
UPDATE 5:40 PM CT: I have no ETA at this time - we are investigating the cause. It is a networking issue affecting multiple softlayer customers in their Dallas data center.
Last edited:
