6/9/2020 - Broadcastify Outage

Status
Not open for further replies.

K9JDN

Member
Feed Provider
Joined
Jan 31, 2005
Messages
40
Location
Madison County, Illinois (STL Metro-area)
HA! I thought I had an issue here. I just installed a new router yesterday because my old one was giving me problems....Thanks SO MUCH for the heads-up notice! I noticed Weather Underground is also down....they must be affected somehow by the outages.
 

belvdr

No longer interested in living
Joined
Aug 2, 2013
Messages
2,567
Thats what Im currently getting, I refreshed DNS cache already.View attachment 86393
Yep thats what i am getting with scannercast and radiofeed. I am done messing with it tonight.
I'm showing a current IP of 52.207.231.97. I think your cache is still holding on to the old IP. Best thing to do is reboot.

The other thing to consider is if you are using your router as your DNS server. If so, its cache isn't updated, so a reboot of it, then your system should fix it.
 

ESF-12

Member
Joined
Mar 5, 2017
Messages
30
Location
Edmonton, Alberta
I'm showing a current IP of 52.207.231.97. I think your cache is still holding on to the old IP. Best thing to do is reboot.

The other thing to consider is if you are using your router as your DNS server. If so, its cache isn't updated, so a reboot of it, then your system should fix it.
4 reboots later and 10 relaunches one feed is working
 

belvdr

No longer interested in living
Joined
Aug 2, 2013
Messages
2,567
4 reboots later and 10 relaunches one feed is working
DNS caches are aggravating at times. You should always flush your DNS server(s) first, if applicable, then your systems. Otherwise, you simply pull the old data down from your DNS server(s). In many cases, home users do have a caching DNS server built into their router.

Streaming services, such as Broadcastify, can help when these failovers occur by keeping their TTL low at all times for their domains. You'll find a 5 minute (or less) TTL is common so users don't see a large outage when things move.
 

mkielsmeierhb

MK Productions
Premium Subscriber
Joined
Nov 11, 2010
Messages
11
Location
Huntington Beach, California
Folks,

Our data center provider where a lot of our master and relay servers are hosted is experiencing connectivity problems - which is causing close to a systemwide outage on Broadcastify right now.

I'm going to see how long it takes for recovery before putting plans in action to fail over to other provides. Stay tuned for updates here.

Thanks for your patience and sorry for the problems.

UPDATE 8:55 PM CT: Coverage has returned to 94% of pre-outage feeds online. We're operating completely on backup systems. Later this week, presumably in the early morning hours, we will fail back over to the primary provider and set of servers. Next update will be what we know about the outage. Stay tuned.

UPDATE 8:12 PM CT: If you are feed provider, and your feed has not come back online, simply reboot your computer and start your broadcast client software back up. That should bring the feed back online

UPDATE 8:01 PM CT: Brining the archive servers back online now. 6,557 out of 7025 feeds are back online, and listeners are recovering quickly.

UPDATE 7:53 PM CT: About 90% of all the feeds are back online and broadcasting. All of the archive servers that record all the feeds are currently down and not capturing feeds right now but we plan on bringing those back up shortly.

UPDATE 7:09 PM CT: We have failed over all servers to a new provider and we're coming back up!

UPDATE 6:47 PM CT: We are failing over to a new provider. We will bring up the master servers first and then the relay servers. Updates to follow...

UPDATE 6:16 PM CT: We are in the dark right now with IBM (Softlayer) - their status pages are also down. We are beginning to unpack our contingency guidebook to fail over the complete audio system to another provider. ;) We'll wait for now to determine if we get a status before embarking upon that journey.

UPDATE 6:06 PM CT: This is a major outage at the data center affecting multiple customers. We have trouble tickets opened with Softlayer and are awaiting a response. Currently all feeds are down since all connectivity to the Dallas softlayer data center has been lost. Still no ETA.

UPDATE 5:40 PM CT: I have no ETA at this time - we are investigating the cause. It is a networking issue affecting multiple softlayer customers in their Dallas data center.
Is there a way to contact Offical Stream owners to reboot? Ex: Metro Net Fire Dispatch in orange county California?
 

midnite630

Newbie
Feed Provider
Joined
Dec 5, 2009
Messages
2
Location
Torrington, CT
DNS caches are aggravating at times. You should always flush your DNS server(s) first, if applicable, then your systems. Otherwise, you simply pull the old data down from your DNS server(s). In many cases, home users do have a caching DNS server built into their router.

Streaming services, such as Broadcastify, can help when these failovers occur by keeping their TTL low at all times for their domains. You'll find a 5 minute (or less) TTL is common so users don't see a large outage when things move.

TTL on broadcastify.com, according to nslookup, is 1 day but the refresh is 2 hrs. So it just a matter of time.
I also noticed the serial number is 1. That can also be an issue if it reset.
 

belvdr

No longer interested in living
Joined
Aug 2, 2013
Messages
2,567
TTL on broadcastify.com, according to nslookup, is 1 day but the refresh is 2 hrs. So it just a matter of time.
I also noticed the serial number is 1. That can also be an issue if it reset.
That's the SOA record for the entire domain; it's essentially the default. It can be overwritten for individual records.

The serial number is only useful for DNS slaves to transfer any new records. Some time ago, tools would simply update the master, increment the serial, and allow the master to notify the slaves to do a zone transfer. These days, many tools no longer do that. Instead, they utilize multiple masters and update them at the same time, thus the serial number is irrelevant as there are no slaves.
 

n9mxq

Member
Premium Subscriber
Joined
Dec 15, 2005
Messages
1,846
Location
Belvidere IL
Currently, in addition to Broadcastify, I am unable to connect with Orange County Register correctly, and also Wunderground.com
Yea, I wasn't happy my station wasn't online at Wunderground, since Cristobal was cruising through the area..
 
Status
Not open for further replies.
Top