CNN - that's what took us down

Status
Not open for further replies.

blantonl

Founder and CEO
Staff member
Super Moderator
Joined
Dec 9, 2000
Messages
11,095
Location
San Antonio, Whitefish, New Orleans
Hi Folks,

We now definitively know what took us down during the earthquakes, See Page 9, middle of the page of the PDF attachment below. This is a transcript of the CNN newsroom during the earthquakes on the east cost last week. On air, CNN said:

One more thing, I was on RadioReference.com, one of my favorite Web sites. You guys should listen to scanners all over the world, basically. And we do have some reports in Richmond, Virginia, of people smelling natural gas. That's probably not unusual if you start
shaking the ground. Some of the pipes may start losing some of their structural rigidity. Especially those nat gas pipes.

That single statement, live on CNN, drove tens of thousands of simultaneous requests to our infrastructure within a few seconds knocking out our master database server. We don't know exactly why, but it was most likely due to the server running out of memory trying to immediately serve all those simultaneous requests.

What have we done to help prevent this in the future?

1) We've implemented throttling at our proxy level (the front end to the Web site) to make sure that we limit the number of requests per second to levels that our back end web servers and databases are currently provisioned to support. As our environment provisions new resources or auto-scales, we'll up that number accordingly. During a huge flood in traffic, the behavior that the visitor would see would be slightly slower response times based on the traffic coming in, but it won't crash the site. And, we'll be ready to increase servers within minutes to handle the load instead of responding to a down Web site.

2) We've provisioned more powerful, and more full time database replica servers which serve read-only content to the site. These servers are responsible for serving most of the content you see. In hindsight, we were not adequately provisioned in this area and that was my fault. But, we've made a good investment in much more database server capacity and that will help us in the long run.

Finally, we have a new site software release scheduled to go in the next few weeks that will great improve the performance of the site overall, but also the readability and usability. Stay tuned for more details on this - our admins are actively beta testing the new site format and features.

Hopefully this gives you guys some perspective on what happened and what we are doing to prevent further issues such as the one experienced during the earhquake. While we were only down for about 40 minutes, we had tens of thousands of new people clamoring to see the site and we weren't able to give them the taste of the sweet nectar of RadioReference. :)

To all the visitors, members, admins and senior leadership team, many thanks for your help this weekend during the hurricane and your efforts to make our platform awesome.

Warm regards,

Lindsay
 

texasemt13

CenTex DBA
Premium Subscriber
Joined
Jun 6, 2007
Messages
2,372
Location
Hunter, TX
Great rundown Lindsay...

From a user perspective, it's nice to see you're proactive about reinforcing the site.

From a member perspective it's awesome to see that one statement can have such an impact.

Lastly, how were you able to track it down? Did you get tips that it was mentioned on CNN?
 

KD0OTK

Member
Joined
Mar 5, 2011
Messages
153
Location
Elk River, MN
Oddly enough, I was watching CNN at the time. I was really shocked when the anchor said, Check out radioreference.com and listen to scanners! I bet that wasn't planned.
 

eorange

♦Insane Asylum Premium Member♦
Joined
Aug 20, 2003
Messages
2,925
Location
Cleveland, OH
You should return the favor - post an announcement which rumors that cnn.com is giving away free scanners :D
 

UPMan

In Memoriam
Premium Subscriber
Joined
Apr 19, 2004
Messages
13,296
Location
Arlington, TX
I was going to see about taking out an advertisement on that error page, but he had to go and fix the problem. :(

:)
 

APTN

Member
Joined
Apr 18, 2008
Messages
255
Location
Tennessee
Crash

Well, apparently I missed the crash, as I had my own system crash to deal with. :mad: However, I'm glad that both crashes are now fixed, and that the site got free advertisement. :D
 
Status
Not open for further replies.
Top