12/8/2011 - Slow Site due to the Virginia Tech shootings

Status
Not open for further replies.

blantonl

Founder and CEO
Staff member
Super Moderator
Joined
Dec 9, 2000
Messages
11,098
Location
San Antonio, Whitefish, New Orleans
Many have inquired why the site was running so slowly this afternoon. Well, there was a double homicide shooting incident on the Virginia Tech campus today which resulted in tens of thousands of listeners flooding in to the audio portions of the site within a period of about five minutes. Virginia Tech is a highly technical community so they are more online than most.

The site was mentioned by students being interviewed on CNN, as well as numerous viral posts in the social media circuit (Facebook, Twitter etc), which drove a large amount of traffic to the site in a short period of time.

Luckily, we learned an important lesson a few months ago when CNN mentioned us during the east coast earthquake, so we put in place "throttles" to prevent too much traffic from taking the site down until we could provision new servers. On that day when CNN mentioned us on the air, so much traffic so quickly to RadioReference caused our database servers to crash under the load and we were down for over 45 minutes! Running a platform that can instantly see tens of thousands of people flocking to it out of nowhere can be challenging since it is difficult and costly to keep servers constantly provisioned and prepared for such traffic. But, the good thing is the site stayed up and running, albiet slowly, and we were able to provision new servers to better handle the additional traffic we were seeing.

For those that want the nitty gritty on what happened today - we have a proxy server that load balances requests across our Web servers and also throttles requests so if we hit a traffic spike like we did today, we won't overwhelm what is already running. We currently throttle to 2000 req/sec and we were maxed out for most of the afternoon. After the news hit we brought up 2 additional Web servers and a database replica server to handle the load and things are back to reasonable levels of performance.

With that said - for those of you that were having trouble getting your RR fix this afternoon, our apologies. :cool:
 
Last edited:

abbadabbado

Member
Joined
Sep 6, 2011
Messages
168
Location
port richey
thanks

i dont know the teck side of what you just said but it sounds like you guys have it all together again thanks from west coast of florida
 

N1XDS

ÆS Ø
Joined
Nov 3, 2004
Messages
1,932
i dont know the teck side of what you just said but it sounds like you guys have it all together again thanks from west coast of florida

Lindsay was saying that the live audio feeds were hammered/flooded from the sad day of actions today at Virgina Tech. Glad the site is back up and running.
 

talkpair

Member
Joined
Apr 27, 2009
Messages
976
Location
Clinton County, MO
As the site becomes discovered by people that ordinarily would have no interest in scanning, it would seem like these balancing acts will get to be more commonplace.
 

trace1

Member
Joined
Jan 13, 2006
Messages
776
Location
EM73co
The site was mentioned by students being interviewed on CNN, as well as numerous viral posts in the social media circuit (Facebook, Twitter etc), which drove a large amount of traffic to the site in a short period of time.

I can't say that any of my personal post on Twitter or Facebook went viral, but I was glad to mention the live feeds here on RadioReference to all of my friends/followers!
 

POSTAL24060

Member
Joined
Jan 7, 2007
Messages
20
Location
Montgomery County VA
I just want to say thanks to RR for providing all the technical know-how to keep my feed online today. I have heard from several students and employees at Virginia Tech telling me that while they were on lock-down that my feed kept them more informed than other media which was slow to update their info.

Thanks again

Kevin
 

Highpockets

Member
Joined
Dec 31, 2003
Messages
3,836
Location
Toms River, New Jersey
I just want to say thanks to RR for providing all the technical know-how to keep my feed online today. I have heard from several students and employees at Virginia Tech telling me that while they were on lock-down that my feed kept them more informed than other media which was slow to update their info.

Thanks again

Kevin

Good example of not having encryption actually helped to do some good in an emergency situation.
 

jasonpeoria911

Member
Database Admin
Joined
Apr 24, 2003
Messages
518
Location
Peoria, IL
I would be curious if some of the VA Tech Admins were listening while they were in Washington. I'm sure they got all the inside info on the phone but I would bet they want to listen as it's happening.

On a side note, I tried connecting to the feed from my Droid phone at work when it was about 5,500 listeners. It kept saying unable to connect. I kept trying for about an hour and had no luck. Meanwhile the feed listeners kept rising and was at about 7,200 but I wasn't able to connect still. Not sure if it was the 10's of thousands trying to connect at once and I just wasn't lucky enough or what.

Jason
 

Dude111

An Awesome Dude
Joined
Aug 8, 2009
Messages
446
ANOTHER ONE???????

I thought this was about the one that happend there YEARS AGO!!!


Very sad indeed.... Would it be possible for RR to host all the scanner streams ON A DIFFERENT SERVER so it doesnt affect the main site??

Maybe a member would be willing to host them all.....
 

Bob_61

Member
Joined
Oct 23, 2009
Messages
298
Location
Enfield,Ct.
Many have inquired why the site was running so slowly this afternoon. Well, there was a double homicide shooting incident on the Virginia Tech campus today which resulted in tens of thousands of listeners flooding in to the audio portions of the site within a period of about five minutes. Virginia Tech is a highly technical community so they are more online than most.

The site was mentioned by students being interviewed on CNN, as well as numerous viral posts in the social media circuit (Facebook, Twitter etc), which drove a large amount of traffic to the site in a short period of time.

Luckily, we learned an important lesson a few months ago when CNN mentioned us during the east coast earthquake, so we put in place "throttles" to prevent too much traffic from taking the site down until we could provision new servers. On that day when CNN mentioned us on the air, so much traffic so quickly to RadioReference caused our database servers to crash under the load and we were down for over 45 minutes! Running a platform that can instantly see tens of thousands of people flocking to it out of nowhere can be challenging since it is difficult and costly to keep servers constantly provisioned and prepared for such traffic. But, the good thing is the site stayed up and running, albiet slowly, and we were able to provision new servers to better handle the additional traffic we were seeing.

For those that want the nitty gritty on what happened today - we have a proxy server that load balances requests across our Web servers and also throttles requests so if we hit a traffic spike like we did today, we won't overwhelm what is already running. We currently throttle to 2000 req/sec and we were maxed out for most of the afternoon. After the news hit we brought up 2 additional Web servers and a database replica server to handle the load and things are back to reasonable levels of performance.

With that said - for those of you that were having trouble getting your RR fix this afternoon, our apologies. :cool:

It's all good! Thanks for the good work !! :)
 

greenthumb

Colorado DB Administrator
Database Admin
Joined
Feb 29, 2004
Messages
1,942
Still 280 listening this evening - how many listeners did that feed peak out at today?
 

jasonpeoria911

Member
Database Admin
Joined
Apr 24, 2003
Messages
518
Location
Peoria, IL
The max may have been 7,500 but I bet it would of been over 20,000 listeners if everyone was able to connect. Lindsay said 10s of thousands of people were rushing the sight at the same time and I know I wasn't able to connect at all for an hour, I gave up after that.

I listened to the archives, you don't hear any of the original incident as the feed doesn't have VA Tech PD on it, you just hear some surrounding agencies responding for mutual aid.

Jason
 
Status
Not open for further replies.
Top