Random Musings of an Insane Mind

Earlier today I was debating Ajax Push and Pyramid for a project I had in mind. I ended up spending about 45 minutes writing a quick proof of concept, then, decided that perhaps something a bit more detailed with some documentation would be helpful for others.

I used Pyramid and APE and wrote a quick demo app. All of the code for the demo app can be downloaded from http://code.google.com/p/pyramid-ape-demo/.

In the html/ directory, the files, graphics and javascript files required to run the client side of the app are included. In the ape_server/ directory, the javascript that needs to be installed in the Ape Server scripts/ directory is present. You’ll want to modify the password. Also included in the html/ directory is a python script called push.py which allows you to use urllib2.urlopen to communicate with the server directly. And finally, in the ape/ directory is a very minimal Pyramid application. pyramidape.wsgi is also included as a starting point to get the site set up.

In the demo, the left hand Coke can is controlled completely by the Ape Javascript Client code. Communications between the browser and Ape server are not processed by anything but Ape. On the right hand side, the Coke can is controlled by a json post to Pyramid and then Pyramid uses urllib2.urlopen to communicate with Ape which then updates the page.

Changes made on the page are reflected among all of the other people that are currently viewing the page in realtime. Since we’re using Ajax push, the page doesn’t need to be reloaded to show those changes. In this example, an img src and the alt text is changed along with a button. You can write your script to modify any html on the page – changing the colors of the page, elements, etc.

Using Ajax push and long polling with Pyramid isn’t difficult and this simple demo and example code should be a good starting point.

A while back I wrote a plugin to take care of a particular client traffic problem. As the traffic came in very quickly and unexpectedly, I had only minutes to come up with a solution. As I knew Varnish pretty well, my initial reaction was to put the site behind Varnish. But, there’s a problem with Varnish and WordPress.

WordPress is a cookie monster. It uses and depends on cookies for almost everything – and Varnish doesn’t cache assets that contain cookies. VCL was modified and tweaked, but, the site was still having problems.

So, a plugin was born. Since I was familiar with ESI, I opted to write a quick plugin to cache the sidebar and the content would be handled by Varnish. On each request, Varnish would assemble the Edge Side Include and serve the page – saving the server from a meltdown.

The plugin was never really production ready, though, I have used it for a year or so when particular client needs came up. When Varnish released 3.0, ESI could work with GZipped/Deflated content which significantly increased the utility of the plugin.

If you would like to read a detailed explanation of how the plugin works and why, here’s the original presentation I gave in Florida.

You can find the plugin on WordPress’s plugin hosting at http://wordpress.org/extend/plugins/cd34-varnish-esi/.

I believe this is ready for a little external testing. While I am not extremely happy with the compression used, it does give about a 15% reduction in space with very little CPU impact.

I do intend to write my own streaming compression which should get me closer to a 55% compression ratio based on some simple testing. I need to add some features to select the ethernet port to watch and get logging rotating on a daily basis, but, it does do the original intended job.

http://code.google.com/p/ddos-log/

Thank you for any feedback.

Note: you don’t need to be under a DDOS to test it, it just logs packets going to port 25 and 80 to a logfile for later processing.

Requests from:

User-Agent: BlackBerry8530/5.0.0.886 Profile/MIDP-2.1 Configuration/CLDC-1.1 VendorID/105

Accept: application/vnd.rim.html,text/html,application/xhtml+xml,
application/vnd.wap.xhtml+xml,text/vnd.sun.j2me.app-descriptor,
image/vnd.rim.png,image/jpeg,application/xvnd.rim.pme.b,
application/vnd.rim.ucs,image/gif;anim=1,application/vnd.rim.jscriptc;
v=0-8-72,application/x-javascript,application/vnd.rim.css;v=2,text/css;
media=screen,application/vnd.wap.wmlc;q=0.9,application/vnd.wap.wmlscriptc;q=0.7,
text/vnd.wap.wml;q=0.7,*/*;q=0.5

We prefer a number of content-types, but, if worse comes to worse, we’ll accept everything anyhow.

440 bytes transmitted on EVERY request made from a Blackberry when you could have just done:

Accept: */*

and saved 428 bytes PER request.

This particular page had 97 assets, amounting to almost 32k in wasted bandwidth sending headers to the CDN.

While working with a DDOS attack that has gone on for over two years, we learn that varnishncsa is not the best logging platform out there. While Varnish does a superb job at protecting the site, the logging leaves a little to be desired. A kill/varnishncsa redirect script runs every night at midnight, logrotate compresses the files and we’re left with a big set of logfiles — logfiles that don’t represent the entire picture.

Because we’re firewalling attacker IPs, our logs only show the requests that make it through the firewall – which minimizes the data that we can collect. From a forensic analysis standpoint, that makes the collected data less valuable. As a result, we need to collect the data off a span port, but, even though it is a denial of service attack against the web, it is good to log all TCP/UDP/syn traffic on the machine to make sure we register everything.

In an ideal world, the machine should have three ethernet ports, or, you should do this monitoring from another machine, but, this is a component to the ISO I’m putting together that can be used as a front-end proxy-cache that logs the attacks. The concept is to create an ISO or USB stick installation that sets up Varnish, IPSet, this logger and the blocker that adds the rules to IPSet.

Tux, a kernel mode http accelerator, used to log to a compressed file and had a tux2w3c helper that would convert the logs to an ASCII readable format that could be processed by weblog software. Since we’re not logging the actual web request, but the TCP packet received, we have a lot more information that we can look at. Our analysis software can look for markers within that data to make decisions and send to IPSet to self-protect and self-heal through the use of expiration times on the rules.

Initially I believe the log format will look something like this:

<timestamp><attacker ip><countrycode><attacked IP><port><tcp payload>

A tool to output the logfile in an ASCII readable form will be written as well so that the data can later be analyzed. Each row will be bzip2 compressed so that the daemon can run endlessly. Logfile names will be portlog.incidentid.20120102 and won’t require rotation. I suspect it might be worthwhile to later allow the logfile to include the hour, resulting in 24 files per day.