6/9/2020 - Broadcastify Outage

Status
Not open for further replies.

blantonl

Founder and CEO
Staff member
Super Moderator
Joined
Dec 9, 2000
Messages
11,095
Location
San Antonio, Whitefish, New Orleans
Folks,

Our data center provider where a lot of our master and relay servers are hosted is experiencing connectivity problems - which is causing close to a systemwide outage on Broadcastify right now.

I'm going to see how long it takes for recovery before putting plans in action to fail over to other provides. Stay tuned for updates here.

Thanks for your patience and sorry for the problems.

FINAL UPDATE 10:38 PM CT: Root cause of the outage was networking configuration problems affecting IBM Cloud globally. Our audio servers (8 of them) are hosted on IBM Cloud’s Softlayer bare metal servers. We failed everything over to Amazon Web services and are currently running on that backup environment. Our plan is to move everything back to the production environment on Softlayer tomorrow morning around 7:00 AM CT. Thanks for everyone’s patience... our disaster recovery plan was tested and worked nicely.

UPDATE 8:55 PM CT: Coverage has returned to 94% of pre-outage feeds online. We're operating completely on backup systems. Later this week, presumably in the early morning hours, we will fail back over to the primary provider and set of servers. Next update will be what we know about the outage. Stay tuned.

UPDATE 8:12 PM CT: If you are feed provider, and your feed has not come back online, simply reboot your computer and start your broadcast client software back up. That should bring the feed back online

UPDATE 8:01 PM CT: Brining the archive servers back online now. 6,557 out of 7025 feeds are back online, and listeners are recovering quickly.

UPDATE 7:53 PM CT: About 90% of all the feeds are back online and broadcasting. All of the archive servers that record all the feeds are currently down and not capturing feeds right now but we plan on bringing those back up shortly.

UPDATE 7:09 PM CT: We have failed over all servers to a new provider and we're coming back up!

UPDATE 6:47 PM CT: We are failing over to a new provider. We will bring up the master servers first and then the relay servers. Updates to follow...

UPDATE 6:16 PM CT: We are in the dark right now with IBM (Softlayer) - their status pages are also down. We are beginning to unpack our contingency guidebook to fail over the complete audio system to another provider. ;) We'll wait for now to determine if we get a status before embarking upon that journey.

UPDATE 6:06 PM CT: This is a major outage at the data center affecting multiple customers. We have trouble tickets opened with Softlayer and are awaiting a response. Currently all feeds are down since all connectivity to the Dallas softlayer data center has been lost. Still no ETA.

UPDATE 5:40 PM CT: I have no ETA at this time - we are investigating the cause. It is a networking issue affecting multiple softlayer customers in their Dallas data center.
 
Last edited:

1arrow

Newbie
Joined
Nov 4, 2012
Messages
1
Location
OR
Folks,

Our data center provider where a lot of our master and relay servers are hosted is experiencing connectivity problems - which is causing close to a systemwide outage on Broadcastify right now.

I'm going to see how long it takes for recovery before putting plans in action to fail over to other provides. Stay tuned for updates here.

Thanks for your patience and sorry for the problems.

UPDATE 5:40 PM CT: I have no ETA at this time - we are investigating the cause. It is a networking issue affecting multiple softlayer customers in their Dallas data center.
thank you for the update
 

drbeede

Member
Feed Provider
Joined
Apr 5, 2019
Messages
126
Location
Gatewood,Missouri
Looks like all the feeds are down as of 6:04 pm CST. hope everything gets lined out soon, I enjoy being a feed provider and I also enjoy listening to other feeds from around the country. Good luck in getting every thing back online and thanks for the heads up.
My Feeds:
 

bigRoN18

Member
Premium Subscriber
Joined
Aug 12, 2001
Messages
141
Location
Roseville, CA
Trump is hacking us for talking bad about him and trying to scan him? Lol.. But yeah good time to run some updates on my feed PC. A new windows 10 update version 2004.. Do I dare?
I ended up moving my feed PCs to Ubuntu but since the public feed was down, I checked for updates... apparently a kernel update is available today, so good timing.
 

ecweather

Newbie
Feed Provider
Joined
Apr 4, 2017
Messages
3
Location
Eau Claire, WI
Not a good time when a Sub Tropical System is making its way through Wisconsin of all places.
Is there redundancy or a backup data center that be switched over too?
I am a newer feed provider that does NOAA WX Radio feeds. which is why I ask.
 

cstandifer

Newbie
Joined
Dec 31, 2009
Messages
1
Location
bakersfield
Folks,

Our data center provider where a lot of our master and relay servers are hosted is experiencing connectivity problems - which is causing close to a systemwide outage on Broadcastify right now.

I'm going to see how long it takes for recovery before putting plans in action to fail over to other provides. Stay tuned for updates here.

Thanks for your patience and sorry for the problems.

UPDATE 6:16 PM CT: We are in the dark right now with IBM (Softlayer) - their status pages are also down. We are beginning to unpack our contingency guidebook to fail over the complete audio system to another provider. ;) We'll wait for now to determine if we get a status before embarking upon that journey.

UPDATE 6:06 PM CT: This is a major outage at the data center affecting multiple customers. We have trouble tickets opened with Softlayer and are awaiting a response. Currently all feeds are down since all connectivity to the Dallas softlayer data center has been lost. Still no ETA.

UPDATE 5:40 PM CT: I have no ETA at this time - we are investigating the cause. It is a networking issue affecting multiple softlayer customers in their Dallas data center.


I hope the server is back up soon. I listen 20 hours a day.
 

gjkastra

Newbie
Feed Provider
Joined
Dec 26, 2011
Messages
1
Location
Biloxi, MS
Folks,

Our data center provider where a lot of our master and relay servers are hosted is experiencing connectivity problems - which is causing close to a systemwide outage on Broadcastify right now.

I'm going to see how long it takes for recovery before putting plans in action to fail over to other provides. Stay tuned for updates here.

Thanks for your patience and sorry for the problems.

UPDATE 6:16 PM CT: We are in the dark right now with IBM (Softlayer) - their status pages are also down. We are beginning to unpack our contingency guidebook to fail over the complete audio system to another provider. ;) We'll wait for now to determine if we get a status before embarking upon that journey.

UPDATE 6:06 PM CT: This is a major outage at the data center affecting multiple customers. We have trouble tickets opened with Softlayer and are awaiting a response. Currently all feeds are down since all connectivity to the Dallas softlayer data center has been lost. Still no ETA.

UPDATE 5:40 PM CT: I have no ETA at this time - we are investigating the cause. It is a networking issue affecting multiple softlayer customers in their Dallas data center.

Thank you for letting us know up front on you website. So few do not do that. I cycled my Broadcasify PC then later saw your not.

Again Thank you for letting us know.

Gus.
 
Status
Not open for further replies.
Top