Compression and Massive Logging to flatfiles for DDOS logging
Monday, January 2nd, 2012While working with a DDOS attack that has gone on for over two years, we learn that varnishncsa is not the best logging platform out there. While Varnish does a superb job at protecting the site, the logging leaves a little to be desired. A kill/varnishncsa redirect script runs every night at midnight, logrotate compresses the files and we’re left with a big set of logfiles — logfiles that don’t represent the entire picture.
Because we’re firewalling attacker IPs, our logs only show the requests that make it through the firewall – which minimizes the data that we can collect. From a forensic analysis standpoint, that makes the collected data less valuable. As a result, we need to collect the data off a span port, but, even though it is a denial of service attack against the web, it is good to log all TCP/UDP/syn traffic on the machine to make sure we register everything.
In an ideal world, the machine should have three ethernet ports, or, you should do this monitoring from another machine, but, this is a component to the ISO I’m putting together that can be used as a front-end proxy-cache that logs the attacks. The concept is to create an ISO or USB stick installation that sets up Varnish, IPSet, this logger and the blocker that adds the rules to IPSet.
Tux, a kernel mode http accelerator, used to log to a compressed file and had a tux2w3c helper that would convert the logs to an ASCII readable format that could be processed by weblog software. Since we’re not logging the actual web request, but the TCP packet received, we have a lot more information that we can look at. Our analysis software can look for markers within that data to make decisions and send to IPSet to self-protect and self-heal through the use of expiration times on the rules.
Initially I believe the log format will look something like this:
<timestamp><attacker ip><countrycode><attacked IP><port><tcp payload>
A tool to output the logfile in an ASCII readable form will be written as well so that the data can later be analyzed. Each row will be bzip2 compressed so that the daemon can run endlessly. Logfile names will be portlog.incidentid.20120102 and won’t require rotation. I suspect it might be worthwhile to later allow the logfile to include the hour, resulting in 24 files per day.