RadioReference on Facebook   RadioReference on Twitter   RadioReference Blog
 

Go Back   The RadioReference.com Forums > Site Administration Forums > System-wide Administration

System-wide Administration Support for issues related to subscriptions, server connectivity and commentary, and layout usability. Area specific content should be posted to its applicable forum above.

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1 (permalink)  
Old 06-06-2011, 4:05 PM
blantonl's Avatar
Founder and CEO
  Shack Photos
Shack photos
RadioReference Database Admininstrator
Database Admin
Audio Feed Provider
Audio Feed Provider
 
Join Date: Dec 2000
Location: Whitefish, MT
Posts: 8,196
Default Systemwide Outage This Afternoon (6/6/2011)

Well folks, we had some emergency maintenance that we needed to accomplish this afternoon at 2:30 PM to change out an NFS server that was reporting a hardware failure. Thus, you probably noticed much of the site down.

It was supposed to be about a 5 min outage, but it took us a little over 50 minutes to recover the entire environment which is very disappointing to us on our end. But, we gathered a lot of good lessons learned about recovering our environment in situations where hardware fails unexpectedly and we will be implementing changes to our recovery processes to better account for that.

In any case, all services are back up and running normally. Additionally, our audio archive servers were down during that period so our audio archives for our feeds will be unavailable for that 50 min timeframe (and most likely 30 min before and after). You'll want to give them at least another 30 minutes before they start populating archives back into the database.

Thanks for your patience.

Warm regards,
__________________
Lindsay C. Blanton III
CEO - RadioReference.com / Broadcastify
Facebook: RadioReference | Broadcastify | Twitter: @RadioReference
Reply With Quote
Sponsored links
  #2 (permalink)  
Old 06-06-2011, 4:21 PM
Member
  Premium Subscriber
Premium Subscriber
Amateur Radio Operator
Amateur Radio
 
Join Date: Feb 2006
Location: Tyler, TX
Posts: 113
Default

When i try to look up FCC licenses I am getting a database error saying the tables don't exist.
Reply With Quote
  #3 (permalink)  
Old 06-06-2011, 5:18 PM
burner50's Avatar
Member
  RadioReference Database Admininstrator
Database Admin
 
Join Date: Dec 2004
Location: Ft. Worth, TX
Posts: 2,313
Default

Same Result here
Reply With Quote
  #4 (permalink)  
Old 06-06-2011, 5:33 PM
Member
  Premium Subscriber
Premium Subscriber
 
Join Date: Nov 2007
Location: St. Louis County, MO
Posts: 1,691
Default

Quote:
Originally Posted by burner50 View Post
Same Result here

Yep, it's broke.
Reply With Quote
  #5 (permalink)  
Old 06-06-2011, 11:40 PM
burner50's Avatar
Member
  RadioReference Database Admininstrator
Database Admin
 
Join Date: Dec 2004
Location: Ft. Worth, TX
Posts: 2,313
Default

For what its worth, it appears to be working once again.
Reply With Quote
Sponsored links
  #6 (permalink)  
Old 06-10-2011, 7:33 PM
Member
  Audio Feed Provider
Audio Feed Provider
 
Join Date: May 2008
Location: Castle Rock, Co.
Posts: 744
Default 6-10-2011

It appears all of Colorado is off line at this time. Did I miss an system wide outage announcement.
__________________
Pro-2096
Pro-106
(3)Pro-197
BCD396XT
Reply With Quote
  #7 (permalink)  
Old 06-10-2011, 7:40 PM
Member
  Audio Feed Provider
Audio Feed Provider
 
Join Date: Jan 2005
Location: Marietta Ohio
Posts: 12
Default

evrything shows offline but I can still listen to most of my usual ones
Reply With Quote
  #8 (permalink)  
Old 06-10-2011, 7:42 PM
datainmotion's Avatar
Member
  Premium Subscriber
Premium Subscriber
 
Join Date: Jan 2001
Location: Colorado
Posts: 2,490
Default

Yep - CO feeds are fine - they just have an offline tag currently.
__________________
Travelling through Colorado?
KDEN/ZDV Feeds
wir hören zu
Reply With Quote
  #9 (permalink)  
Old 06-10-2011, 7:44 PM
Member
  Audio Feed Provider
Audio Feed Provider
 
Join Date: May 2008
Location: Castle Rock, Co.
Posts: 744
Default

Thanks, I didn't even try. It is just showing off line.
__________________
Pro-2096
Pro-106
(3)Pro-197
BCD396XT

Last edited by 04Z1V6; 06-10-2011 at 7:51 PM..
Reply With Quote
Sponsored links
  #10 (permalink)  
Old 06-10-2011, 7:45 PM
Member
  Premium Subscriber
Premium Subscriber
 
Join Date: Dec 2003
Location: Earth Sector 001
Posts: 539
Default

Someone let me know when it's time to panic. i'm always up for a good panic.
Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT -5. The time now is 5:56 PM.


Powered by vBulletin® Version 3.8.2
Copyright ©2000 - 2014, vBulletin Solutions, Inc.
All information here is Copyright 2012 by RadioReference.com LLC and Lindsay C. Blanton III.Ad Management by RedTyger
Copyright 2011 by RadioReference.com LLC Privacy Policy  |  Terms and Conditions