RadioReference Outage Today 2/22/2009

[blantonl]

As many of you probably know, we had a complete outage today on the site for more than 1 hour, due to our Web server's load being off the charts.

I arrived home from a nice relaxing hike with the family to see that our Web server was not responding, so a quick look at the server revealed this (after waiting almost 5 minutes for the login)

12:58:41 up 457 days, 23:08,  1 user,  load average: 255.79, 255.77, 252.85

For those of you that aren't Unix system administrators, the above output from the command "uptime" tells you primarily how long the server has been running since last reboot, how many users are logged on to the machine's consoles, and how much load the server is experiencing. Two staggering statistics in that output - one, our Web server hasn't been rebooted in 457 days! two, the load average was off the charts, the most I've ever seen out Web server at. It typically runs at about 2-4, and on a busy day 5-6. 255 means an unresponsive server and nothing is going to happen.

I brought down the database server instance (on another server) first to see if that would help since I couldn't really do anything on the Web server except execute a command and wait 5 minutes for a response. But that didn't help, so bringing down apache (our web server software) did the trick. After some updates, rebooting the entire infrastructure, and bringing eerything back up gracefully, it appears we are ok now.

That is the most downtime we've had in probably 2 years.

So what happened? Well, we don't definitively know. A possible bug in apache might have caused a memory leak, a backdated Kernel might have caused a loading problem, who knows. But after some patch updates, and a nice reboot, it looks like we're OK.

Warm regards,

 

[burner50]

Thanks for the swift response!


I guess the server was due for a reboot.

 

[RDCat]

Thanks

 

[bezking]

Heh. So much for the hike, I guess

 

[fineshot1]

Thanks Lindsay - that is a long time for no reboot even for unix. I used to do reboots on the one unix platform I was responsible for(about 10 years ago) at about 90 day intervals whether I thought it needed it or not. Most of the time it was a log or log files that would not auto purge and get too large that would gum up the works and make the system run unstable.

 

[j0nnyb1aze]

457 days is certainly nothing to sneeze at. I cant say I've ever seen loads like that though. Good job w/ the turn around.

 

[epilab]

At least you did not get a virus that coiuld have killed everyone

 

[bootsfirst]

DDoS, perhaps?

But I think it'd continue even after the server was brought back up if that was the case...shrug.

Since we're on the subject, how about a little geek porn with you telling us all about your servers? (hardware, setup, etc etc)

 

[burner50]

Since we're on the subject, how about a little geek porn with you telling us all about your servers? (hardware, setup, etc etc)

http://www.radioreference.com/forum...mand-post-version-4-december-17th-2008-a.html

 

[chrismol1]

Since we're on the subject, how about a little geek porn with you telling us all about your servers? (hardware, setup, etc etc)

ohh baby!
show me them quad core xeons!

 

[blantonl]

Here are the specs on the servers:

Production Environment - hosted in data center

1 Web Server
Dell PowerEdge 440
Dual Core Intel Xeon 3040
2GB RAM
250GB SATA II Drives
100 MB Public Facing Port
100 MB VPN Port

1 Database Server
Dell PowerEdge 1430
(2) Dual Core Intel Xeon 5120
4GB RAM
250GB SATA II Drives
100 MB Public Facing Port
100 MB VPN Port

Staging Server - Home

PowerEdge 2950
(2) Quad Core Intels
12 GB RAM
2TB RAID-50

Development

My Mac Pro Workstation, 2 quad core 6GB RAM

 

[SCPD]

Good job getting the site back up!!!

 

[roots]

that was the longest & boring around at work I ever has,

good to ear that everything is back to normal!

 

[mtindor]

e login)

12:58:41 up 457 days, 23:08,  1 user,  load average: 255.79, 255.77, 252.85

Heh, that isn't pretty. I agree I haven't seen a load that high in 19 years of unix. And I'm shocked you were able to even run top. Typically, on a machine with a load above 25 its difficult / time consuming to even gain access via SSH and even console access/program execution lags signfiicantly. I'll just say I hope I never have to deal with a server load that high

Mike

 

[Jay911]

Surprised the case didn't melt right off the machine...

 

[K8TEK]

Let me get this straight, you run a single dual core processor on your web server and experience loads of 2-4 on normal days? You do realize this means the machine is underpowered by at least half of what it needs to be.

 

[Bubba1661]

Don't Know About Everybody Else But I Was Going Into The DT'S. Good Job On Getting The Site Back Up!

 

[mtindor]

Let me get this straight, you run a single dual core processor on your web server and experience loads of 2-4 on normal days? You do realize this means the machine is underpowered by at least half of what it needs to be.

It's not all about CPU usage. There are a lot of factors taken into consideration when load averages are determined, including network activity, disk activity, CPU resources used, processes in use, resources needed by processes but awaiting access to that resource, etc. A load average of 2-4 would be an indication that a server has some activity, i.e. doing more than just idling, fairly active. But it doesn't mean that the server is underpowered or that the server cannot serve requests in a timely fashion. I see servers running all the time, particularly ones processing mail and performing antispam filtering, that can 10s of thousands or 100s of thousands of spamassassin scanning events per day and do it without fail, day in and day out, 365 days a year, with load averages ranging from 5 to 20 at any given time.

I'm not saying the server isn't working, but it's no indication of what the server is capable of delivering or a sole determinator on whether it is suffering.

Mike

 

[Caesar]

Wow, great job Lindsay, i was suffering through the day, i finnaly broke done and went to take an afternoon nap. luckily i get back up and see rr.com is back uP!

 

[blantonl]

Let me get this straight, you run a single dual core processor on your web server and experience loads of 2-4 on normal days? You do realize this means the machine is underpowered by at least half of what it needs to be.

Cue the know-it-alls... :roll: