Random Musings of an Insane Mind

Today we had a DDOS attack on one of our clients. They were running prefork with mod_php5 with a rather busy application. While we initially started filtering IP addresses using iptables and a few crudely hacked rules, we knew something had to be done that was a little more permanent. Moving to MPM-Worker with PHP served with FastCGI seemed reasonable, but, looking at the history of the attacks on this machine, I believe Apache still would have been vulnerable since we cannot filter the requests early enough in Apache’s request handler.

Apache does have the ability to fight some DDOS attacks using mod_security and mod_evasive, but, this particular attack was designed to affect apache prior to the place where these modules hook into the request. This also precludes using fail2ban. We could run mod_forensic or mod_logio to assist fail2ban, but, it is still a stopgap measure.

We could have used some Snort rules and tied those to iptables, but, that is a rather bad solution to the problem.

While we could have used Varnish, their application would have had some issues. mod_rpaf can help by adjusting the REMOTE_ADDR to take the value from X-Forwarded-For that Varnish sets. mod_geoip actually inserts itself before mod_rpaf, so, we would have needed to make a modification to mod_geoip and recompiled it. I’m not sure how Varnish would have handled Slowloris and we had to fix this now.

Putting them behind a Layer 7 load balancer would have isolated the origin server and handled the brunt of the attack on the load balancer, but, again we would have needed mod_rpaf and some modifications to their code.

In the end, Lighttpd and Nginx appeared to be the only documented solution. After the conversion, we did find documentation that said Varnish and Squid were immune to Slowloris. With Nginx or Lighttpd, we didn’t have IP address issues to contend with, it would be easy enough to modify the fastcgi config to pass the GEOIP information in the same request variable that their application expected. We knew we had to run PHP under FastCGI, so, we might as well pick a tool where we can block the attack in the webserver without having to worry about firewall rules. We did put a few firewall rules in place to block the larger offenders.

in the http { } section of our nginx config, we added:

    client_body_timeout 10;
    client_header_timeout 10;
    keepalive_timeout 10;
    send_timeout 10;
    limit_zone limit_per_ip $binary_remote_addr 16m;

and in the server { } section, we added:

    limit_conn limit_per_ip 5;

Since each server was expecting to handle one or two requests from each IP, this gave us a little headroom while solving the problem in the right place.

I believe Varnish would have held the connection and wouldn’t have sent a request to the backend which makes it fairly versatile as a tool to deal with DDOS attacks. While I do like the ability to block certain requests in VCL, the methods listed to fight this type of attack appeared to favor a non-threaded webserver. Varnish in front of Apache would have worked, but, we already knew we needed to move from Apache at some point with this client and this gave us an opportunity to shift them while under the gun.

Wouldn’t have had it any other way.

While working on a maintenance script, I’ve found myself using bash a bit more than using perl for many tasks. While I don’t mind writing in perl or python, sometimes it just seems like overkill.

I needed to read one line of a /proc/loadavg for a monitoring system and was using some code that had been written by one of our programmers in 2002 or so. The old code was written using three scripts, two of which were called from cron. The first script was written in perl, actually included /proc/loadavg as chomp($value=`cat /proc/loadvg`);

Ugh!

Then some math was done, and a decision made of what values to write for nagios’s local process to report back. Suffice it to say that the math done then involved executing a shell script from time to time based on some condition checks and a separate cron that cleaned up the status for nagios.

A literal translation of the script from perl to bash ended up with a snippet of code that looked like this:

#!/bin/bash

  LOAD=`cat /proc/loadavg`
  LOADAVG=${LOAD%%.*}

echo $LOADAVG

We’ll spare the details regarding how badly the existing script converted LOADAVG to an int in perl. While this method works, it does cause a process to be forked needlessly to read /proc/loadavg.

A quick rewrite and we end up with:

#!/bin/bash

  read LOAD < /proc/loadavg
  LOADAVG=${LOAD%%.*}

echo $LOADAVG

While this doesn't seem overly bad for a process executed once a minute, it does cause a larger problem when someone copies that snippet of code and uses it elsewhere.

If we modify the code slightly to:

#!/bin/bash

for i in {1..10000}
do
  LOAD=`cat /proc/loadavg`
  LOADAVG=${LOAD%%.*}
done

echo $LOADAVG

and

#!/bin/bash

for i in {1..10000}
do
  read LOAD < /proc/loadavg
  LOADAVG=${LOAD%%.*}
done

echo $LOADAVG

and run the following benchmark:

root@tsavo:/home/cd34# time ./s.sh 
0

real	0m13.641s
user	0m0.636s
sys	0m2.304s
root@tsavo:/home/cd34# time ./t.sh 
0

real	0m0.485s
user	0m0.332s
sys	0m0.148s

we can see that there is a substantial impact to doing things.

Taking the integer portion of a floating point number also yields a number of terrible solutions involving sed, awk and cut — all forking a separate process or two.