6/9/2020 - Broadcastify Outage

[cpfinlay]

ISSUE: Continuous Delivery
Component: Continuous Delivery
Location: Tokyo, Frankfurt, Washington DC, London, Dallas, Seoul, Sydney, Chennai
Start time: 9 Jun 2020 6:04 PM
End time: N/A
Update time: 9 Jun 2020 6:41 PM
Type: Incident
ID: INC2434130
Description: We have been experiencing issues with the following components:

Continuous Delivery

Our support engineers are currently looking into the problem.

 

[mkhakanson]

Folks,

Our data center provider where a lot of our master and relay servers are hosted is experiencing connectivity problems - which is causing close to a systemwide outage on Broadcastify right now.

I'm going to see how long it takes for recovery before putting plans in action to fail over to other provides. Stay tuned for updates here.

Thanks for your patience and sorry for the problems.

UPDATE 6:16 PM CT: We are in the dark right now with IBM (Softlayer) - their status pages are also down. We are beginning to unpack our contingency guidebook to fail over the complete audio system to another provider. We'll wait for now to determine if we get a status before embarking upon that journey.

UPDATE 6:06 PM CT: This is a major outage at the data center affecting multiple customers. We have trouble tickets opened with Softlayer and are awaiting a response. Currently all feeds are down since all connectivity to the Dallas softlayer data center has been lost. Still no ETA.

UPDATE 5:40 PM CT: I have no ETA at this time - we are investigating the cause. It is a networking issue affecting multiple softlayer customers in their Dallas data center.

when will it be fix?

 

[ecweather]

ISSUE: Continuous Delivery
Component: Continuous Delivery
Location: Tokyo, Frankfurt, Washington DC, London, Dallas, Seoul, Sydney, Chennai
Start time: 9 Jun 2020 6:04 PM
End time: N/A
Update time: 9 Jun 2020 6:41 PM
Type: Incident
ID: INC2434130
Description: We have been experiencing issues with the following components:

Continuous Delivery

Our support engineers are currently looking into the problem.

So softlayer is having a major outage in several areas around the world?

 

[cpfinlay]

ISSUE: Cloud Platform
Component: Cloud Platform
Location: Washington DC, London, Dallas, Sydney, Tokyo, Frankfurt
Start time: N/A
End time: N/A
Update time: 9 Jun 2020 6:47 PM
Type: Incident
ID: INC2434131
Description: We have been experiencing issues with the following components:

Cloud Platform

Our support engineers are currently looking into the problem.

 

[cpfinlay]

IBM Cloud Status

IBM Cloud Status page

cloud.ibm.com

 

[rickdd]

Thanks for the heads up Lindsay.

Currently, in addition to Broadcastify, I am unable to connect with Orange County Register correctly, and also Wunderground.com

 

[EW1407]

And we are back online!

 

[norwichfeeder]

IBM Cloud outage downs customer websites globally | ZDNet

IBM Cloud suffers prolonged outage | TechCrunch

The IBM Cloud is currently suffering a major outage, and with that, multiple services that are hosted on the platform are also down, including everybody's

techcrunch.com

 

[GeekNJ]

I'll guess someone put in a bad route and all hell broke loose. Usually routing if it impacts multiple data centers.

 

[tomn2tsr]

Thank goodness the forums aren't also hosted on the IBM Cloud.

 

[KI5DQG]

Just got back online, but is still having trouble staying connected.

 

[n0nhp]

Still waiting for DNS servers to catch the update here

 

[keech]

It is showing the feeds are "Online" but when i try to hit on it, I get an error.

 

[SDEC_SCAN]

Fantastic quick work!

 

[W1cjf-1]

Awesome back online. Thanks again!!

 

[kgowen44]

Mine is actually not coming back up. Update says feeds should be up.

 

[W1cjf-1]

Mine is actually not coming back up. Update says feeds should be up.

I restarted scannercast when i saw it went out.. try that plus i bet the dns servers are catching up.

 

[n0nhp]

You might have to flush the DNS cache in your computer/router to get it to download the new addresses, also make sure your feed source is not using an IP address but using the URL to access the server

 

[pharmboy]

Folks,

Our data center provider where a lot of our master and relay servers are hosted is experiencing connectivity problems - which is causing close to a systemwide outage on Broadcastify right now.

I'm going to see how long it takes for recovery before putting plans in action to fail over to other provides. Stay tuned for updates here.

Thanks for your patience and sorry for the problems.

UPDATE 7:53 PM CT: About 90% of all the feeds are back online and broadcasting. All of the archive servers that record all the feeds are currently down and not capturing feeds right now but we plan on bringing those back up shortly.

UPDATE 7:09 PM CT: We have failed over all servers to a new provider and we're coming back up!

UPDATE 6:47 PM CT: We are failing over to a new provider. We will bring up the master servers first and then the relay servers. Updates to follow...

UPDATE 6:16 PM CT: We are in the dark right now with IBM (Softlayer) - their status pages are also down. We are beginning to unpack our contingency guidebook to fail over the complete audio system to another provider. We'll wait for now to determine if we get a status before embarking upon that journey.

UPDATE 6:06 PM CT: This is a major outage at the data center affecting multiple customers. We have trouble tickets opened with Softlayer and are awaiting a response. Currently all feeds are down since all connectivity to the Dallas softlayer data center has been lost. Still no ETA.

UPDATE 5:40 PM CT: I have no ETA at this time - we are investigating the cause. It is a networking issue affecting multiple softlayer customers in their Dallas data center.

Good Job Lindsay!

 

[cpfinlay]

Thank you, Lindsay!