6/9/2020 - Broadcastify Outage

[rrunyon1575]

Did the sever IP Address change i can not reconnect

 

[W1cjf-1]

Did the sever IP Address change i can not reconnect

Try to restart scannercast. My feed is up with no issues.

 

[K9JDN]

HA! I thought I had an issue here. I just installed a new router yesterday because my old one was giving me problems....Thanks SO MUCH for the heads-up notice! I noticed Weather Underground is also down....they must be affected somehow by the outages.

 

[rrunyon1575]

Try to restart scannercast. My feed is up with no issues.

Tried that didnt work. I also have radiofeed and its saying cant connect to server

 

[ESF-12]

Thats what Im currently getting, I refreshed DNS cache already.

 

[rrunyon1575]

Thats what Im currently getting, I refreshed DNS cache already.View attachment 86393

Yep thats what i am getting with scannercast and radiofeed. I am done messing with it tonight.

 

[belvdr]

Thats what Im currently getting, I refreshed DNS cache already.View attachment 86393

Yep thats what i am getting with scannercast and radiofeed. I am done messing with it tonight.

I'm showing a current IP of 52.207.231.97. I think your cache is still holding on to the old IP. Best thing to do is reboot.

The other thing to consider is if you are using your router as your DNS server. If so, its cache isn't updated, so a reboot of it, then your system should fix it.

 

[blantonl]

if you are having a problem getting your feed back online

REBOOT YOUR COMPUTER!

 

[ESF-12]

I'm showing a current IP of 52.207.231.97. I think your cache is still holding on to the old IP. Best thing to do is reboot.

The other thing to consider is if you are using your router as your DNS server. If so, its cache isn't updated, so a reboot of it, then your system should fix it.

4 reboots later and 10 relaunches one feed is working

 

[belvdr]

4 reboots later and 10 relaunches one feed is working

DNS caches are aggravating at times. You should always flush your DNS server(s) first, if applicable, then your systems. Otherwise, you simply pull the old data down from your DNS server(s). In many cases, home users do have a caching DNS server built into their router.

Streaming services, such as Broadcastify, can help when these failovers occur by keeping their TTL low at all times for their domains. You'll find a 5 minute (or less) TTL is common so users don't see a large outage when things move.

 

[mkielsmeierhb]

Folks,

Our data center provider where a lot of our master and relay servers are hosted is experiencing connectivity problems - which is causing close to a systemwide outage on Broadcastify right now.

I'm going to see how long it takes for recovery before putting plans in action to fail over to other provides. Stay tuned for updates here.

Thanks for your patience and sorry for the problems.

UPDATE 8:55 PM CT: Coverage has returned to 94% of pre-outage feeds online. We're operating completely on backup systems. Later this week, presumably in the early morning hours, we will fail back over to the primary provider and set of servers. Next update will be what we know about the outage. Stay tuned.

UPDATE 8:12 PM CT: If you are feed provider, and your feed has not come back online, simply reboot your computer and start your broadcast client software back up. That should bring the feed back online

UPDATE 8:01 PM CT: Brining the archive servers back online now. 6,557 out of 7025 feeds are back online, and listeners are recovering quickly.

UPDATE 7:53 PM CT: About 90% of all the feeds are back online and broadcasting. All of the archive servers that record all the feeds are currently down and not capturing feeds right now but we plan on bringing those back up shortly.

UPDATE 7:09 PM CT: We have failed over all servers to a new provider and we're coming back up!

UPDATE 6:47 PM CT: We are failing over to a new provider. We will bring up the master servers first and then the relay servers. Updates to follow...

UPDATE 6:16 PM CT: We are in the dark right now with IBM (Softlayer) - their status pages are also down. We are beginning to unpack our contingency guidebook to fail over the complete audio system to another provider. We'll wait for now to determine if we get a status before embarking upon that journey.

UPDATE 6:06 PM CT: This is a major outage at the data center affecting multiple customers. We have trouble tickets opened with Softlayer and are awaiting a response. Currently all feeds are down since all connectivity to the Dallas softlayer data center has been lost. Still no ETA.

UPDATE 5:40 PM CT: I have no ETA at this time - we are investigating the cause. It is a networking issue affecting multiple softlayer customers in their Dallas data center.

Is there a way to contact Offical Stream owners to reboot? Ex: Metro Net Fire Dispatch in orange county California?

 

[midnite630]

DNS caches are aggravating at times. You should always flush your DNS server(s) first, if applicable, then your systems. Otherwise, you simply pull the old data down from your DNS server(s). In many cases, home users do have a caching DNS server built into their router.

Streaming services, such as Broadcastify, can help when these failovers occur by keeping their TTL low at all times for their domains. You'll find a 5 minute (or less) TTL is common so users don't see a large outage when things move.

TTL on broadcastify.com, according to nslookup, is 1 day but the refresh is 2 hrs. So it just a matter of time.
I also noticed the serial number is 1. That can also be an issue if it reset.

 

[blantonl]

TTL on broadcastify.com, according to nslookup, is 1 day but the refresh is 2 hrs. So it just a matter of time.
I also noticed the serial number is 1. That can also be an issue if it reset.

The TTL on the audio master servers is 60 seconds, and 1 hour on the relays.

 

[belvdr]

TTL on broadcastify.com, according to nslookup, is 1 day but the refresh is 2 hrs. So it just a matter of time.
I also noticed the serial number is 1. That can also be an issue if it reset.

That's the SOA record for the entire domain; it's essentially the default. It can be overwritten for individual records.

The serial number is only useful for DNS slaves to transfer any new records. Some time ago, tools would simply update the master, increment the serial, and allow the master to notify the slaves to do a zone transfer. These days, many tools no longer do that. Instead, they utilize multiple masters and update them at the same time, thus the serial number is irrelevant as there are no slaves.

 

[n9mxq]

Currently, in addition to Broadcastify, I am unable to connect with Orange County Register correctly, and also Wunderground.com

Yea, I wasn't happy my station wasn't online at Wunderground, since Cristobal was cruising through the area..