Random Musings of an Insane Mind

We operate a number of DNS servers, one of which we left open for recursive lookups for clients to see their sites before they’ve moved the DNS. While we normally handle that through a proxy server we maintain, DNS was a secondary method for doing this which was sometimes easier than having a client use a proxy server.

Today, after 60 days notice, we shut off external recursive lookups at 6pm EST. In the last two hours, we’ve received a number of ‘lookups’ from a few ‘high volume’ IP Addresses. The list is surprising.

Output from (annotated with whois info):

grep -E 'denied$' /var/log/syslog|cut -f 8 -d ' '|cut -f 1 -d '#'|sort|uniq -c|sort -nr|head -n 30

 506452 212.146.85.194  (GTS Telecom Romania Operations)
  38444 141.101.125.86  (Cloudflare EU)
  10490 141.101.124.86  (Cloudflare EU)
  10236 67.228.130.45  (Softlayer)
   4784 212.227.135.196  (OneandOne AG)
   1277 173.201.216.32  (Godaddy)
    673 163.121.134.170 
    620 163.121.194.154
    617 2001:2060:ffff:a01::53 (Sonera, Finland)
    528 95.142.101.5  (Cyber Technology BVBA) 
    528 95.142.100.5  (Cyber Technology BVBA)
    528 178.237.35.125
    464 66.93.87.2
    323 193.210.18.18
    306 67.15.238.64
    294 193.210.19.19
    270 94.23.147.151
    230 76.76.11.241
    213 72.53.193.43
    207 204.194.238.17
    202 72.53.193.42
    192 208.80.194.121
    190 78.28.197.6
    190 78.28.197.5
    190 212.93.150.198
    180 204.194.238.15
    168 213.157.178.54
    160 208.69.35.22
    153 163.121.128.90
    152 212.6.108.157

What are we seeing?

GTS Telecom Romania Operations:
# grep 212.146.85.194 /var/log/syslog|cut -f 2 -d "'"|sort|uniq -c|sort -nr|more
 539590 isc.org/ANY/IN

CloudFlare EU:
# grep 141.101.125.86 /var/log/syslog|cut -f 2 -d "'"|sort|uniq -c|sort -nr|more
  41660 ripe.net/ANY/IN

# grep 141.101.124.86 /var/log/syslog|cut -f 2 -d "'"|sort|uniq -c|sort -nr|more
  10490 ripe.net/ANY/IN

SoftLayer:
# grep 67.228.130.45 /var/log/syslog|cut -f 2 -d "'"|sort|uniq -c|sort -nr|more
  10924 ripe.net/ANY/IN
      1 pdkamoaaaaekt0000dkaaabaaafbadli.ripe.net/ANY/IN
      1 oobdjlaaaaekt0000dkaaabaaafbadli.ripe.net/ANY/IN
      1 onigfiaaaaekt0000dkaaabaaafbadli.ripe.net/ANY/IN
      1 ojfhfgaaaaekt0000dkaaabaaafbadli.ripe.net/ANY/IN
      1 nphhdiaaaaekt0000dkaaabaaafbadli.ripe.net/ANY/IN
      1 ngffklaaaaekt0000dkaaabaaafbadli.ripe.net/ANY/IN

OneandOne AG:
# grep 212.227.135.196 /var/log/syslog|cut -f 2 -d "'"|sort|uniq -c|sort -nr|more
   5196 isc.org/ANY/IN

Godaddy:
# grep 173.201.216.32 /var/log/syslog|cut -f 2 -d "'"|sort|uniq -c|sort -nr|more
   3031 ripe.net/ANY/IN

Lumped into that was an IPv6 resolver that actually pointed out an interesting issue. The domains 2001:2060:ffff:a01::53 is looking for should be hitting our server for an authoritative lookup, but, appears to be doing recursive lookups. I’ve made a minor change to our configs to see if I can log a bit more data as it isn’t a continuous stream.

It is interesting to see the number of hits that occurred in a two hour period which explains why the network PPS rate on that server has been higher than normal.

This traffic is part of a DNS reflection DDOS attack using spoofed UDP packets with the ‘source address’ set to the above targets. Why they’ve chosen isc.org or ripe.net as their typical entry, I don’t know. Since UDP is a connectionless protocol, there is no Syn/Ack, making DNS susceptible to spoofed packets. Our DNS resolver, which was previously publicly available, was responding based on the source IP in the UDP payload. The hackers chose a particular group of servers to send those responses to.

Oddly, I did see some TCP traffic from one of Cloudflare’s EU servers which should have been impossible as they are using anycast. Shortly after disabling recursive lookups, the attack stopped. This means that the hackers are watching the servers involved in the attack and when they saw that it was no longer affected, they stopped sending that spoofed traffic.

We are in the process of upgrading our DNS servers and swapping things around and this was the first step in redesigning our DNS infrastructure. It was only by chance we noticed that our resolver was being used in the DNS reflection attack as it was only sending out a few mb/sec more traffic than it should have been.

My apologies to the servers that were receiving DDOS traffic from our resolvers.

I’ve been dealing with a rather persistent comment spam issue with my blog. Akismet seems to ignore this particular type of trackback spam and has let hundreds slip through over the last few months. Since my blog really isn’t that busy, it is very easy to identify the spam. I can’t turn off trackbacks as I have gotten trackbacks from other people referencing my site showing how they solved a problem.

A couple weeks ago I installed Simple Trackback Validation with Topsy Blocker which has done a good job of catching the ones that Akismet seems to have problems detecting. As I run a rather complex setup for testing my plugins, I submitted a bug fix which the author promptly installed. I don’t know why Akismet doesn’t detect keyword keyword… as potential spam, but, almost every trackback Akismet has missed, follows that pattern and so far, the other plugin has caught every single one.

This morning, I noticed two comments had been posted in the last seven hours, and, when I looked, I saw:

While my site isn’t that busy, it ranks fairly well on some search phrases and social circles. The most popular post on my blog was written over two years ago and was for Mac OS/X Snow Leopard. Other posts have been more popular over a thirty day period, but, that post has stood the test of time even though it is now two generations of Operating Systems behind.

But, I do have sites that do trackback spam from that particular page quite frequently, and, Akismet misses them about 85% of the time. You wouldn’t know from the Akismet graphs, it claims a much higher success rate, almost the inverse of the fail rate.

I know when I get links from Anandtech.com, I do often check them to make sure they are linking, and, in a few spot checks in the past, they were, but, those links appear to have been cleaned from their forums and the two posts where I remember them being listed. Today however, the trackback was on a very new article which had no relation to the page it linked to, and obviously, no link from their site pointed to my site.

I looked back through the approved comments and found 71 other trackbacks, investigated a number of pages, and, as you might suspect, my link wasn’t present anywhere.

This is where the analysis turns a bit sinister. Why did they pick a page that detailed an issue with Varnish and gzip compressed pages to link to an article Anandtech wrote yesterday? Age of the post? The original post was from Dec 2009, though, it is the outlier in the stats.

Upon looking at thirty of the trackbacks, a curious pattern emerged. Since adding the social media buttons for Google+, Twitter and Facebook, I’ve had a quick metric to gauge post popularity, and, lo and behold, Anandtech is targeting posts that have high tweet counts with the exception of the original outlier.

The original post is linked to a post that deals with Social Game Design which is also a popular post. There may have been a trackback on that page which I deleted ages ago and their trackback bot just spidered it.

Or, it could be completely dumb and just taken the urls from topsy.com if they looked far enough back through my history – except for the original outlier.

But Anandtech, Welcome to the Comment Blacklist.

A number of our clients run WordPress, but, for some reason, keeping them updated is a problem. Sites are uploaded and run on autopilot and are forgotten… until they are hacked. Last week a client asked why his WordPress 2.8 site was hacked. WordPress 2.8 was released in June 2009 with 25 WordPress releases since. We checked a few of his sites and found a few different versions running, but, how many other clients were running old WordPress versions? The results were shocking.

Finding WordPress sites on shared storage

First, we need to find each of the individual client’s WordPress installations.

find /var/www -type f -wholename \*wp-includes/version.php|awk '{ print "grep -H \"wp_version =\" " $1 }' | sh > /var/tmp/wpversions

From this, with a little cut and sort trickery, we end up with the following histogram:

35 3.1
29 2.8.5
25 3.2.1
24 2.9.2
20 2.8.4
19 3.0.1
16 2.8.2
16 2.1
15 2.6
13 2.7.1
8 3.3.1
6 2.9.1
6 2.8
6 2.3
5 3.3
5 2.7
4 3.1.2
4 2.8.1
3 3.1.1
3 3.0.5
3 3.0.4
3 2.8.6
2 3.1.3
2 3.0
2 2.0.3
1 3.1.4
1 2.9
1 2.6.3
1 2.5.1
1 2.3.3
1 2.3.2
1 2.2.2
1 2.2.1
1 2.2
1 2.1.3
1 2.0.5
1 2.0.4

Yes, we have 2 2.0.3 installations in production use out of 285 sites. Of them, 8, or, less than 3% are running the current version, 3.3.1.

Clearly this is a problem.

We have a few options, one of which is to utilize the upgrade process inside WordPress which requires us to communicate with each client, or, write a quick script to give us admin privileges to do the upgrade. Or, we could use bash.

The magic

Our filesystem structure is set up so that each user has their own UID/GID, and the paths where the domains are located are fairly static. However, the script just takes the path of the wp-content/version.php file, strips off the correct pieces, copies the uncompressed WordPress .tar.gz file, changes ownership from root to the user that owns the directory.

There are two variables that need to be set:

WORDPRESS_TMPDIR – set this to the directory where you have untarred and ungzipped the WordPress archive

BASE_PATH – set this to the machine’s root path.

The script

#!/bin/bash

# cd34, 20120212
# 
# find /var/www -type f -wholename \*wp-includes/version.php|awk '{ print "grep -H \"wp_version =\" " $1 }' | sh > /var/tmp/wpversions
# 
# if you want to really save time:
# awk < /var/tmp/wpversion '{ print "/path/to/wpu.sh " $1 }' | sh -x

# set this to match your temporary directory location for WordPress
WORDPRESS_TMPDIR=/var/tmp/wordpress
# wget -O /var/tmp http://wordpress.org/latest.tar.gz
# cd /var/tmp
# tar xzf latest.tar.gz

#set this to signify the base path of your machine's web root
BASE_PATH=/var/www

if [ "X" == "$1X" ];
then
  echo "Needs a pathname for the version.php file"
  echo
  echo "$0 /var/www/domain.com/wp-includes/version.php"
  echo
  echo "You can include data after version.php, i.e. :$version from find command"
else
  WP_INCLUDE_PATH=$1
  WP_PATH=${WP_INCLUDE_PATH%%/wp-includes/version.php*}
  DOMAIN=${WP_PATH##$BASE_PATH/}

  TMP=`stat $WP_PATH|grep Uid:`
  TMP_GID=${TMP##*Gid: ( }
  DGID=${TMP_GID%%/*}
  TMP_UID=${TMP##*Uid: ( }
  DUID=${TMP_UID%%/*}

  `cp -Rp $WORDPRESS_TMPDIR/* $WP_PATH`
  `chown -R --from=root $DUID.$DGID $WP_PATH`
  `/usr/bin/wget -q -O /dev/null "http://$DOMAIN/wp-admin/upgrade.php?step=1"`
  echo "Upgraded: http://$DOMAIN"
fi

The code for this and a few other tools that I’ve written can be found at cd34-tools, hosted on code.google.com

I believe this is ready for a little external testing. While I am not extremely happy with the compression used, it does give about a 15% reduction in space with very little CPU impact.

I do intend to write my own streaming compression which should get me closer to a 55% compression ratio based on some simple testing. I need to add some features to select the ethernet port to watch and get logging rotating on a daily basis, but, it does do the original intended job.

http://code.google.com/p/ddos-log/

Thank you for any feedback.

Note: you don’t need to be under a DDOS to test it, it just logs packets going to port 25 and 80 to a logfile for later processing.

While working with a DDOS attack that has gone on for over two years, we learn that varnishncsa is not the best logging platform out there. While Varnish does a superb job at protecting the site, the logging leaves a little to be desired. A kill/varnishncsa redirect script runs every night at midnight, logrotate compresses the files and we’re left with a big set of logfiles — logfiles that don’t represent the entire picture.

Because we’re firewalling attacker IPs, our logs only show the requests that make it through the firewall – which minimizes the data that we can collect. From a forensic analysis standpoint, that makes the collected data less valuable. As a result, we need to collect the data off a span port, but, even though it is a denial of service attack against the web, it is good to log all TCP/UDP/syn traffic on the machine to make sure we register everything.

In an ideal world, the machine should have three ethernet ports, or, you should do this monitoring from another machine, but, this is a component to the ISO I’m putting together that can be used as a front-end proxy-cache that logs the attacks. The concept is to create an ISO or USB stick installation that sets up Varnish, IPSet, this logger and the blocker that adds the rules to IPSet.

Tux, a kernel mode http accelerator, used to log to a compressed file and had a tux2w3c helper that would convert the logs to an ASCII readable format that could be processed by weblog software. Since we’re not logging the actual web request, but the TCP packet received, we have a lot more information that we can look at. Our analysis software can look for markers within that data to make decisions and send to IPSet to self-protect and self-heal through the use of expiration times on the rules.

Initially I believe the log format will look something like this:

<timestamp><attacker ip><countrycode><attacked IP><port><tcp payload>

A tool to output the logfile in an ASCII readable form will be written as well so that the data can later be analyzed. Each row will be bzip2 compressed so that the daemon can run endlessly. Logfile names will be portlog.incidentid.20120102 and won’t require rotation. I suspect it might be worthwhile to later allow the logfile to include the hour, resulting in 24 files per day.