Systemwide Outage This Afternoon (6/6/2011)
Well folks, we had some emergency maintenance that we needed to accomplish this afternoon at 2:30 PM to change out an NFS server that was reporting a hardware failure. Thus, you probably noticed much of the site down.
It was supposed to be about a 5 min outage, but it took us a little over 50 minutes to recover the entire environment which is very disappointing to us on our end. But, we gathered a lot of good lessons learned about recovering our environment in situations where hardware fails unexpectedly and we will be implementing changes to our recovery processes to better account for that.
In any case, all services are back up and running normally. Additionally, our audio archive servers were down during that period so our audio archives for our feeds will be unavailable for that 50 min timeframe (and most likely 30 min before and after). You'll want to give them at least another 30 minutes before they start populating archives back into the database.
Thanks for your patience.