Systemwide Outage This Afternoon (6/6/2011)

Status
Not open for further replies.

blantonl

Founder and CEO
Staff member
Super Moderator
Joined
Dec 9, 2000
Messages
11,095
Location
San Antonio, Whitefish, New Orleans
Well folks, we had some emergency maintenance that we needed to accomplish this afternoon at 2:30 PM to change out an NFS server that was reporting a hardware failure. Thus, you probably noticed much of the site down.

It was supposed to be about a 5 min outage, but it took us a little over 50 minutes to recover the entire environment which is very disappointing to us on our end. But, we gathered a lot of good lessons learned about recovering our environment in situations where hardware fails unexpectedly and we will be implementing changes to our recovery processes to better account for that.

In any case, all services are back up and running normally. Additionally, our audio archive servers were down during that period so our audio archives for our feeds will be unavailable for that 50 min timeframe (and most likely 30 min before and after). You'll want to give them at least another 30 minutes before they start populating archives back into the database.

Thanks for your patience.

Warm regards,
 

ke5fgc

Member
Premium Subscriber
Joined
Feb 28, 2006
Messages
140
Location
Tyler, TX
When i try to look up FCC licenses I am getting a database error saying the tables don't exist.
 

burner50

The Third Variable
Database Admin
Joined
Dec 24, 2004
Messages
2,231
Location
NC Iowa
For what its worth, it appears to be working once again.
 
Status
Not open for further replies.
Top