RadioReference on Facebook   RadioReference on Twitter   RadioReference Blog
 

Go Back   The RadioReference.com Forums > Announcements and News > Community Announcements and News

Community Announcements and News Announcements and News of interest to the RadioReference.com Community. All new threads posted here will be moderated by the administrators. Members are encouraged to post news and information here for the community.

Reply
 
LinkBack Thread Tools Search this Thread Display Modes
  #1 (permalink)  
Old 10-22-2012, 5:24 PM
blantonl's Avatar
Founder and CEO
  Shack Photos
Shack photos
RadioReference Database Admininstrator
Database Admin
Audio Feed Provider
Audio Feed Provider
 
Join Date: Dec 2000
Location: Whitefish, MT
Posts: 8,236
Default Outage caused by Amazon Web Services problems

Folks,

As you are probably aware, we had an outage today that lasted almost 5 hours, due to our primary hosting provider (Amazon Web Services) experiencing significant network connectivity problems with their EBS (Elastic Block Store) which we use to store our master database server data.

Once the problems crept up at 11:30 AM CST, we brought the site offline to prevent having other major problems on our back end infrastructure. We finally made the decision to promote on of our back slave servers to the new master at about 4:30 PM CST. This means that the site is currently running in an emergency recovery configuration - therefore expect to see maintenance during the day tomorrow so we can get back to a more robust database implementation.

In the mean time, you can see Amazon Web Services status here:

AWS Service Health Dashboard - Oct 22, 2012

Also note that this issue was not just unique to RadioReference.com. Many of some of the largest Web properties were impacted.

Update: Amazon Web Services Down In North Virginia

Thanks all your patience during this outage. It was one of the largest we've had in the history of the site.
__________________
Lindsay C. Blanton III
CEO - RadioReference.com / Broadcastify
Facebook: RadioReference | Broadcastify | Twitter: @RadioReference
Reply With Quote
Sponsored links
        
  #2 (permalink)  
Old 10-22-2012, 5:30 PM
pdfdems286's Avatar
Member
   
Join Date: Mar 2006
Location: monmouth county,n.j.
Posts: 1,563
Smile

i tried to go on. saw it was not working. got the shakes !. LOL.
__________________
OY !
Reply With Quote
  #3 (permalink)  
Old 10-22-2012, 5:34 PM
Member
  Audio Feed Provider
Audio Feed Provider
Amateur Radio Operator
Amateur Radio
 
Join Date: Nov 2005
Location: Marceline, Missouri
Posts: 35
Default

These things happen, thank you for keeping us updated though!!!
Reply With Quote
  #4 (permalink)  
Old 10-22-2012, 5:54 PM
fyrrsq4's Avatar
Member
  Shack Photos
Shack photos
Audio Feed Provider
Audio Feed Provider
 
Join Date: Dec 2002
Location: RI
Posts: 4
Default

Thanks for getting the backup online and good luck with the work tomorrow!
Reply With Quote
  #5 (permalink)  
Old 10-22-2012, 6:10 PM
Mtnrider's Avatar
Member
   
Join Date: Oct 2004
Location: Greene County New Yawk
Posts: 628
Default

I was headed to therapy till i saw it was working again..Thanks for being on top of things
Reply With Quote
Sponsored links
        
  #6 (permalink)  
Old 10-22-2012, 6:13 PM
MaxTracker's Avatar
Member
   
Join Date: May 2003
Location: Texas
Posts: 2,300
Default Welcome back!

I was reading a little about Amazon hosting while RR was down. RR goes down on occasion but I can't remember when it was down for 5+ hours (with the exception of announced maintenance).

Because of what radioreference.com is to the scanning and communications world, have you considered a secondary site to keep operations flowing, even if the secondary only utilizes a reference database and not the other perks such as Forums and Broadcastify. Because radioreference.com comes pre-loaded now any downtime could have a negative impact to scanneing/scanner sales worldwide (Though higly unlikely unless the servers were down for days)

Anyways, thank you for keeping us informed and Glad your back! Sure you guys have lotsa work to do (and some rag chewing on amazon reps)

Cheers!
__________________
Scanning Psychiatric Hospitals
NCTX-Scanner
BCD536HP * BCD996XT * BCT15X * PSR-600 * PRO-164 * IC-208H * TAIT TM9155
Reply With Quote
  #7 (permalink)  
Old 10-22-2012, 6:28 PM
troymail's Avatar
Member
   
Join Date: Dec 2002
Location: Anne Arundel County, MD.
Posts: 3,667
Default

Wow - first the GRE announcement last week and then today RR goes off the air --- I was losing it.

So much for "the Cloud" saving the world....
__________________
PSR800 x 2 | PSR500 | PRO96!
BCD536 | BCD436 | BCD396T | BC296D | BC245XLT | BC600XLT | BCIV | VX-8R
Not a Radio Shack fan.....
Reply With Quote
  #8 (permalink)  
Old 10-22-2012, 7:14 PM
blantonl's Avatar
Founder and CEO
  Shack Photos
Shack photos
RadioReference Database Admininstrator
Database Admin
Audio Feed Provider
Audio Feed Provider
 
Join Date: Dec 2000
Location: Whitefish, MT
Posts: 8,236
Default

Hehehe...

RadioReference is a serious infrastructure operation. We have anywhere from 16-20 servers going at one time with Amazon Web services, 1 server with ServerBeach, and 8 servers with 1000TB.com. I'm not sure a lot of people realize what it takes to run this platform from end to end. I had almost 10 people email me today asking if I wanted space on their GoDaddy hosting platform. I appreciate the though, but....

We are backed up against disaster, but we certainly can do a better job to mitigate issues when something like this happens.

But let's not forget - RadioReference sees very little downtime - it is a rare occurrence for us to have problems. We've designed the back end very well to be resilient, but even the largest Web businesses in the world get caught by unexpected problems. We are no exception.

That doesn't excuse it for us, because ultimately we are responsible for what happened - no one else.
__________________
Lindsay C. Blanton III
CEO - RadioReference.com / Broadcastify
Facebook: RadioReference | Broadcastify | Twitter: @RadioReference
Reply With Quote
  #9 (permalink)  
Old 10-22-2012, 7:38 PM
E-Man's Avatar
Member
  Shack Photos
Shack photos
 
Join Date: Mar 2004
Location: Global
Posts: 833
Default

Amazon outage takes down Reddit, Foursquare, others:

Amazon outage takes down Reddit, Foursquare, others - CNN.com
Reply With Quote
Sponsored links
  #10 (permalink)  
Old 10-22-2012, 7:41 PM
Member
   
Join Date: Feb 2012
Location: St. Paul, MN
Posts: 2
Default Thanks for the update!

I couldn't figure out why my cell phone could get every other site but yours. I too was getting shaky but then I brought the site up on my PC and saw the notice about RR being down.

Thank you for posting the full info about what happened. It's great to see a site that gives us more than "our servers were down."

Keep up the great work and have an adult beverage tonight and relax. :-)
Reply With Quote
  #11 (permalink)  
Old 10-22-2012, 8:40 PM
mjbjr's Avatar
Member
  Audio Feed Provider
Audio Feed Provider
 
Join Date: Dec 2009
Location: Macon,Ga USA
Posts: 409
Default

Quote:
Originally Posted by blantonl View Post
Hehehe...

RadioReference is a serious infrastructure operation. We have anywhere from 16-20 servers going at one time with Amazon Web services, 1 server with ServerBeach, and 8 servers with 1000TB.com. I'm not sure a lot of people realize what it takes to run this platform from end to end. I had almost 10 people email me today asking if I wanted space on their GoDaddy hosting platform. I appreciate the though, but....

We are backed up against disaster, but we certainly can do a better job to mitigate issues when something like this happens.

But let's not forget - RadioReference sees very little downtime - it is a rare occurrence for us to have problems. We've designed the back end very well to be resilient, but even the largest Web businesses in the world get caught by unexpected problems. We are no exception.

That doesn't excuse it for us, because ultimately we are responsible for what happened - no one else.
Didnt you make a post once about the ins and outs of RR? I cant seem to find it.
__________________
Macon,Georgia Police,Fire and Ems

Running off a Pro-197
Reply With Quote
  #12 (permalink)  
Old 10-22-2012, 8:57 PM
scannerfreak's Avatar
Moderator
  RadioReference Database Admininstrator
Database Admin
 
Join Date: Jul 2003
Location: Indiana
Posts: 5,565
Default

Quote:
Originally Posted by mjbjr View Post
Didnt you make a post once about the ins and outs of RR? I cant seem to find it.
RadioReference Blog: The Monthly Bill
__________________

Forum Rules and Guidelines
Reply With Quote
  #13 (permalink)  
Old 10-22-2012, 9:04 PM
mjbjr's Avatar
Member
  Audio Feed Provider
Audio Feed Provider
 
Join Date: Dec 2009
Location: Macon,Ga USA
Posts: 409
Default

Quote:
Originally Posted by scannerfreak View Post
Thats it,Thank you!
__________________
Macon,Georgia Police,Fire and Ems

Running off a Pro-197
Reply With Quote
  #14 (permalink)  
Old 10-23-2012, 4:51 AM
MaxTracker's Avatar
Member
   
Join Date: May 2003
Location: Texas
Posts: 2,300
Default Saw those on FB

Quote:
Originally Posted by blantonl View Post
I'm not sure a lot of people realize what it takes to run this platform from end to end. I had almost 10 people email me today asking if I wanted space on their GoDaddy hosting platform. I appreciate the though, but....
I doubt most people have any idea how complex the radioreference.com backbone really is. I didn't till I got into this and it blows me away how far you can take it if you have the working capitol. I'm a happy Go Daddy subscriber but am still looking into/at Amazon hosting - tbh Amazon even has less downtime than several of the other "Big Boys" it's one of those occasional 'Stuff happens' deals.

These little blips in the system though will have you and the crew probably looking at new contingencies to implement down the road ....
__________________
Scanning Psychiatric Hospitals
NCTX-Scanner
BCD536HP * BCD996XT * BCT15X * PSR-600 * PRO-164 * IC-208H * TAIT TM9155
Reply With Quote
  #15 (permalink)  
Old 10-23-2012, 9:51 AM
gr8rcall's Avatar
Member
   
Join Date: Jun 2012
Location: Alamance County, NC
Posts: 756
Default

I got kinda scared yesterday (almost had a heart attack) when I saw:
"RadioReference.com is offline due to Amazon EC2 hosting problems"

Glad it's working again!!!

I kind of felt like "troymail's" AVATAR!
__________________
Supporter of Broadcastify!
Play "Travel Scrabble" here: Travel Scrabble (Word Game)
Pick your favorite soda!
Don't be fooled, I'm not a LEO!

Last edited by gr8rcall; 10-23-2012 at 9:53 AM..
Reply With Quote
Sponsored links
  #16 (permalink)  
Old 10-23-2012, 10:47 AM
gewecke's Avatar
Member
  Premium Subscriber
Premium Subscriber
Amateur Radio Operator
Amateur Radio
 
Join Date: Jan 2006
Location: Northwest Illinois
Posts: 5,780
Talking

Thanks for the update!
...Had serious withdrawal symptoms.

73,
n9zas
__________________
"Whatever doesn't kill you...will make you stronger"!
Reply With Quote
  #17 (permalink)  
Old 10-23-2012, 2:00 PM
One13Truck's Avatar
Member
  Audio Feed Provider
Audio Feed Provider
 
Join Date: Jul 2004
Location: N.E. Pa.
Posts: 458
Default

I was browsing the forums when it went down. At first I thought the internet went down when I couldn't get anything to load. But working in the communications industry sometimes crap happens. Not much can be done about it. Glad to see it seems like it's working batter again today though.
__________________
I STILL use mutual aid numbers & 10 codes instead of plain language. I like them better & will keep using them.

210XLT, 120XLT, BCT2, Pro-90, Pro-2048, Pro-94, BCT15X, PSR-800,BCD396XT,BCD996XT
Reply With Quote
  #18 (permalink)  
Old 10-23-2012, 5:21 PM
luke-1's Avatar
Member
  Audio Feed Provider
Audio Feed Provider
Amateur Radio Operator
Amateur Radio
 
Join Date: Jan 2003
Location: Parker, CO
Posts: 639
Default

I just received an email stating my feed was down. I checked it and it is up and running fine.

Anything to do with this?
__________________
Luke
I'm listening to you
MNN 060
KC0VPS
Aurora Online
Reply With Quote
  #19 (permalink)  
Old 10-23-2012, 5:31 PM
scannerfreak's Avatar
Moderator
  RadioReference Database Admininstrator
Database Admin
 
Join Date: Jul 2003
Location: Indiana
Posts: 5,565
Default

Quote:
Originally Posted by luke-1 View Post
I just received an email stating my feed was down. I checked it and it is up and running fine.

Anything to do with this?
From yesterday's initial post:

Quote:
Originally Posted by blantonl View Post
This means that the site is currently running in an emergency recovery configuration - therefore expect to see maintenance during the day tomorrow so we can get back to a more robust database implementation.
So the site was taken down today for a bit for said maintenance.
__________________

Forum Rules and Guidelines
Reply With Quote
  #20 (permalink)  
Old 10-23-2012, 8:11 PM
One13Truck's Avatar
Member
  Audio Feed Provider
Audio Feed Provider
 
Join Date: Jul 2004
Location: N.E. Pa.
Posts: 458
Default

Yep. I got an email as well. But I actually DID take mine down for a bit to do some upgrades to some stuff on the computer. Can I still blame Amazon for it though?

I'm joking. But wow I can't believe how seriously some people took the outage when I was checking on the FB site yesterday. You'd think it was down for weeks and happened all the time. With the size of the site and the traffic it gets I'm surprised it stays up as much as it does. Just another example of the fine work done by everybody here to keep things running smooth. The hamster in the wheel in the back room must be one heck of a rodent to keep the place going!
__________________
I STILL use mutual aid numbers & 10 codes instead of plain language. I like them better & will keep using them.

210XLT, 120XLT, BCT2, Pro-90, Pro-2048, Pro-94, BCT15X, PSR-800,BCD396XT,BCD996XT
Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are On
Pingbacks are On
Refbacks are On



All times are GMT -5. The time now is 3:12 AM.


Powered by vBulletin® Version 3.8.2
Copyright ©2000 - 2014, vBulletin Solutions, Inc.
All information here is Copyright 2012 by RadioReference.com LLC and Lindsay C. Blanton III.Ad Management by RedTyger
Copyright 2011 by RadioReference.com LLC Privacy Policy  |  Terms and Conditions