2/12/2013 Evening Outage

[blantonl]

Folks,

Around 6:30 PM CST yesterday, during a very high traffic event, our master database server experienced a hard drive failure in it's RAID array. Because we were experiencing such high traffic, it made it extremely difficult to fail over to a slave server and we had to relaunch a new master server to recover.

It took us a couple hours to re-provision the server and bring the entire site back online. During this time, the site was completely inaccessible and we were unable to record audio feeds during the outage.

Everything is now back online and operating normally - and we'll use this event as an opportunity to further harden the site from future outages.

Thanks!

 

[Utah_Viper]

Wirelessly posted (Mozilla/5.0 (Linux; Android 4.0.4; LG-MS770 Build/IMM76I) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Mobile Safari/535.19)

Curious.... Can you give us any insights to the amount of traffic? Both for the web site and feeds

 

[blantonl]

It was enormous - we were processing 4000/requests a second across 10 Web servers and 3 database servers. I believe the SB Feed had 30-40K listeners on it at one time.

 

[russellmaher]

WOW! That must be at the top of list for listeners, no? Lindsay, is this the most activity that's been on the feeds?

Russell

 

[ReddHead]

Incredible. I knew as soon as I got an alert that SB was over 9k listeners there would be problems lol.

 

[spectr17]

The SB feed shows 33082 listeners as a high for yesterday. Does crashing the server mean no incentive for high feed of the week?

I'm told someone gave the URL to CNN and that was the big increase.

 

[ReddHead]

The SB feed shows 33082 listeners as a high for yesterday. Does crashing the server mean no incentive for high feed of the week?

I'm told someone gave the URL to CNN and that was the big increase.

Wonder if they'll be disqualified for violating the TOS.

 

[spectr17]

Wonder if they'll be disqualified for violating the TOS.

What part of the TOS was violated?

 

[wtp]

thanks

thanks for being honest
not many would say sorry or even hint at a loss
i would have liked to listen but it seems so would the rest of the U.S.

 

[tsalmrsystemtech]

Thanks to RR for all of their feeds. Great Job

 

[Spitfire8520]

EDIT: Looks like the post I was replying to is no longer here. On a different note, I wonder what kind of confusion this will cause when Broadcastify comes along and takes over. There's hundreds of thousands of people who now know of the RadioReference name for live audio.

 

[tsalmrsystemtech]

I posted my post in the Los Angeles Forum. I got kind of carried away on a tangent.

 

[a388sig2]

Excuse me if it's been said already, but Fox News Channel was airing live audio, and crediting the site with the feed. So they had a graphic that read, "From Radioreference.com" at that point the site went down.

 

[desert-cheetah]

I had a feeling that had something to do with it. Too much use at one time caused a failure. Bummer it happened, but them's the breaks. Thank you for the report on what happened, Lindsay

 

[riccom]

Sooo now that the sb feed topped the 24 k mark, does the feed provider get a good reward for holding all them, and is there a way to fix the crash?

Sent from my SGH-T759 using Tapatalk 2