Malicious Site Warnings - An update and what happened

[blantonl]

Folks,

Now that we've been able to completely understand what occurred, we're now in a position to clearly communicate what happened with the malicious site warnings that were presented by Google and numerous anti-virus software programs.

First, yes, we were hacked in a concentrated effort by spammers in Russia to infect the site with malware. And they were good... very good.

What happened:

A hacker was able to upload a script that allowed him to compromise some of our Javascript source files, which redirected some browsers to remote sites that attempted to deliver malware to vistors to RadioReference.com. The point of entry was our internal trouble ticketing system which had not been upgraded in a while and had a vulnerability which allowed the malicious upload. They also uploaded a back door in a separate part of the site to allow them to continue to try to infect the site whenever we cleaned up the offending code.

When this occurred, Google noticed the problem, sounded the alert to Web visitors, and blacklisted our site for approximately 18 hours after we cleaned up the initial problem. We initially thought we had cleaned the site of the malware, however the hackers were able to use the 2nd backdoor they installed to infect the site with progressively harder to detect Javascript and configuration changes on our Web servers.

How we fixed the problem.

First, after many days of fighting with the hackers, we identified the trouble ticketing application as being the point of entry, so we took it completely offline. We then took the entire site offline to perform a forensic analysis to see the points of entry, what files were modified, and how they did it.

After that - we completely re-installed all of our Web servers, and restored all of our site code from a backup. We then audited the entire new infrastructure for any threats (there were none). We then brought the site back online.

Why we didn't initially identify what happened

Well, we didn't fully understand what happened. We initially thought that Google misidentified some of our Javascript due to the way that Google initially reported the problem to us (it looked harmless). Additionally, these guys were good. They specifically targeted RadioReference and spent a lot of time researching how our site and infrastructure was put together, which allowed them to prevent us from identifying what occurred.

With that said, I can report that the site is completely clean, the point of entry corrected, and processes in place to make sure that this never happens again. I can also report that we believe the only purpose of this hack was to spread malware to vistors - as the hackers only changed Javascript and code files to distribute the malware. Our audit indicates those were their only actions.

Thanks for everyone's patience while we resolved the issue.

 

[wtp]

thanks

thank you for telling us.
so many other sites (banks,big business, etc) would never pass any info to us.
they would keep quiet or deny anything.


so thanks.

 

[bear780ks]

I'll second that wtp.. Mr.B thanks for the Info..

 

[Highpockets]

Just curious, why did it seem to only bother the main page and not the rest of the site, if that makes sense and can be answered.

Nice job on clearing it up. They were good, you were better.

 

[blantonl]

Just curious, why did it seem to only bother the main page and not the rest of the site, if that makes sense and can be answered.

Nice job on clearing it up. They were good, you were better.

In one of the attack instances - the hackers configured their code to only run on the homepage and only for Internet Explorer browsers.

They spent a lot of time working on their attack vectors...

 

[QDP2012]

Thank you sir and to your team for the hard work and for the update!

Roughly, how many staff-hours and dollars get consumed by an incident like this? I expect that the cost of such forensic analysis, and related system maintenance, on a rapid-response and "work-until-its-fixed" schedule, is not "pocket-change".

Thanks again and great job,

 

[Highpockets]

In one of the attack instances - the hackers configured their code to only run on the homepage and only for Internet Explorer browsers.

They spent a lot of time working on their attack vectors...

Thank you, hope that's the last we see of them.

 

[blantonl]

Thank you sir and to your team for the hard work and for the update!

Roughly, how many staff-hours and dollars get consumed by an incident like this? I expect that the cost of such forensic analysis, and related system maintenance, on a rapid-response and "work-until-its-fixed" schedule, is not "pocket-change".

Thanks again and great job,

Unfortunately, I was on spring break at the time with my family. I probably spent at least 20 hours that week dealing with the situation.

It also cost us dearly for at least 5 days in terms of revenue.

 

[com501]

Unfortunately, this is exactly how most professional hackers work. They CAREFULLY analyze vulnerabilities in systems and exploit them.

I'll bet if you look back through server logs, you will find their IP addresses going back WEEKS before any malware was uploaded.

The front page of any site is where you want to place your malware, that is where 90% of most people end up going before they go somewhere else. With most financial institutions, for instance, the landing page IS the home page, since everything useful from there requires a log in.

Internet Exlorer is the MOST vulnerable browser, so that is logically going to be the target of choice.

Multiple vectors and back doors for high traffic sites to make it very difficult to remove malware are standard fare for most determined hackers. There is BIG money in this type of malware.

 

[Thunderknight]

Was the attempt to infect users unsuccessful if they had Java disabled?

 

[blantonl]

Was the attempt to infect users unsuccessful if they had Java disabled?

We're thinking it targeted a 0 day Java vulnerability but we don't have definitive answers as to the target vector.

 

[Jay911]

Are the attackers 'traceable', i.e. do you know who did it? Was it someone with a beef with you/RR or were you just a target of opportunity?

 

[jfhtm350]

People dont have anything better to do than sit around and try to tear up somebodys else's computer. Thats pathetic.

 

[K9DAK]

Thanks Lindsay for the update.

I worked for a major pharmaceutical company back in 2006 when our entire network of thousands of computers in 100+ sites around the world was infected with a virus. Not sure how it got in there, but it cost us about 250 MILLION dollars, considering lost revenue and the IT support costs of scanning and cleaning every single computer in the company. Customer service reps were taking orders by writing everything down on paper, and only entering the orders into the computer system after it was cleaned and brought back online.

Fortunately, my computer was not involved since I was at an offsite conference the day of the infection and never connected to the network. But my son, the live-in IT guy, somehow heard about the infection before I did, and quarantined my wireless card for our home network before I got home that day. He said "I'm not letting your laptop on my network until I know it's clean!"

Thanks for all you do Lindsay!

 

[troshs]

Glad to hear everythings okay, Lindsay! I was just discussing about how I wondered if they had been watching all of the stuff going on with Dorner in LA since live audio from here was used in news broadcasts, and they figured they could get alot of us since everyone might have been going to the site. I've been having weird problems with my computer since I logged in that day but I think I killed the source of the issues just about an hour ago when I was searching through my files. I found a file that had a bunch of numbers on it and Windows told me it had been blocked due to coming from another computer....well I don't share my files on any network unless specifically requested and I sure hadn't been sharing the day that the creation date showed...which was when everything started. Hope everything goes smoothly from here!
Best Wishes,
Heather

 

[radioalbany]

Thanks!

Many thanks for all the hard work you and your colleagues did.
Very well done, indeed.

Andrew Ellis NO6E

 

[RadioDaze]

I've never been much of a conspiracy theorist, but I wonder about attacks like these. Are they "dry runs" or practice runs for upcoming attacks on more nationally critical sites? What is the point of "picking on" a site like RR? Seems like too much effort to simply harass. Maybe someone's cutting their teeth to do something more damaging elsewhere in the future.

Good work, Lindsay.

 

[NMPhillieFan]

This is a great lesson learned. Hopefully all the people emphatically claiming "it's a false positive" learned as well. While it is possible Google blocks a clean site occasionally, it is fairly rare. At the very least, it is worth "doing your due diligence" and running a virus scan/malware scan and taking the warning you were given seriously.

 

[mlmummert]

Yes thank you for dealing with this and admitting the problems.


Sent from my iPad using Tapatalk

 

[blantonl]

Are the attackers 'traceable', i.e. do you know who did it? Was it someone with a beef with you/RR or were you just a target of opportunity?

Almost certainly not traceable - all logs and forensics pointed back to 3 IP addresses in Russia.