2/12/2013 Evening Outage

Status
Not open for further replies.

blantonl

Founder and CEO
Staff member
Super Moderator
Joined
Dec 9, 2000
Messages
11,097
Location
San Antonio, Whitefish, New Orleans
Folks,

Around 6:30 PM CST yesterday, during a very high traffic event, our master database server experienced a hard drive failure in it's RAID array. Because we were experiencing such high traffic, it made it extremely difficult to fail over to a slave server and we had to relaunch a new master server to recover.

It took us a couple hours to re-provision the server and bring the entire site back online. During this time, the site was completely inaccessible and we were unable to record audio feeds during the outage.

Everything is now back online and operating normally - and we'll use this event as an opportunity to further harden the site from future outages.

Thanks!
 

Utah_Viper

Member
Joined
Jun 21, 2005
Messages
1,464
Location
North Muskegon, MI
Wirelessly posted (Mozilla/5.0 (Linux; Android 4.0.4; LG-MS770 Build/IMM76I) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.1025.166 Mobile Safari/535.19)

Curious.... Can you give us any insights to the amount of traffic? Both for the web site and feeds
 

blantonl

Founder and CEO
Staff member
Super Moderator
Joined
Dec 9, 2000
Messages
11,097
Location
San Antonio, Whitefish, New Orleans
It was enormous - we were processing 4000/requests a second across 10 Web servers and 3 database servers. I believe the SB Feed had 30-40K listeners on it at one time.
 

russellmaher

KC1ANC
Joined
Apr 13, 2010
Messages
988
Location
Enfield, CT
WOW! That must be at the top of list for listeners, no? Lindsay, is this the most activity that's been on the feeds?

Russell
 

spectr17

Member
Joined
Nov 1, 2003
Messages
177
Location
CA
The SB feed shows 33082 listeners as a high for yesterday. Does crashing the server mean no incentive for high feed of the week? :D

I'm told someone gave the URL to CNN and that was the big increase.
 

ReddHead

Member
Joined
Sep 16, 2011
Messages
25
Location
Barnegat, NJ
The SB feed shows 33082 listeners as a high for yesterday. Does crashing the server mean no incentive for high feed of the week? :D

I'm told someone gave the URL to CNN and that was the big increase.

Wonder if they'll be disqualified for violating the TOS.
 

wtp

Member
Joined
Apr 3, 2008
Messages
5,914
Location
Port Charlotte FL
thanks

thanks for being honest
not many would say sorry or even hint at a loss
i would have liked to listen but it seems so would the rest of the U.S.
 

Spitfire8520

I might be completely clueless! =)
Joined
Jun 29, 2009
Messages
1,969
Location
Colorado
EDIT: Looks like the post I was replying to is no longer here. On a different note, I wonder what kind of confusion this will cause when Broadcastify comes along and takes over. There's hundreds of thousands of people who now know of the RadioReference name for live audio.
 

a388sig2

Member
Joined
Jul 15, 2004
Messages
293
Excuse me if it's been said already, but Fox News Channel was airing live audio, and crediting the site with the feed. So they had a graphic that read, "From Radioreference.com" at that point the site went down.
 

desert-cheetah

Member
Premium Subscriber
Joined
Sep 10, 2010
Messages
619
Location
In the desert someplace
I had a feeling that had something to do with it. Too much use at one time caused a failure. Bummer it happened, but them's the breaks. Thank you for the report on what happened, Lindsay
 

riccom

Upstate S.C.
Premium Subscriber
Joined
Jul 2, 2004
Messages
1,318
Location
Kansas City Mo
Sooo now that the sb feed topped the 24 k mark, does the feed provider get a good reward for holding all them, and is there a way to fix the crash?

Sent from my SGH-T759 using Tapatalk 2
 
Status
Not open for further replies.
Top