md3 : active raid1 sdb8[1] sda8[0]
      962108608 blocks [2/2] [UU]
      [===>.................]  check = 15.1% (145325952/962108608) finish=60402.6min speed=224K/sec

To abandon the check, we issued:

echo idle > /sys/block/md3/md/sync_action

Which allowed the machine to skip the rest of the test. While I don’t like disabling the checks, we’ll reschedule this one to do the check after they are done doing their work.

Apache does have the ability to fight some DDOS attacks using mod_security and mod_evasive, but, this particular attack was designed to affect apache prior to the place where these modules hook into the request. This also precludes using fail2ban. We could run mod_forensic or mod_logio to assist fail2ban, but, it is still a stopgap measure.

We could have used some Snort rules and tied those to iptables, but, that is a rather bad solution to the problem.

While we could have used Varnish, their application would have had some issues. mod_rpaf can help by adjusting the REMOTE_ADDR to take the value from X-Forwarded-For that Varnish sets. mod_geoip actually inserts itself before mod_rpaf, so, we would have needed to make a modification to mod_geoip and recompiled it. I’m not sure how Varnish would have handled Slowloris and we had to fix this now.

Putting them behind a Layer 7 load balancer would have isolated the origin server and handled the brunt of the attack on the load balancer, but, again we would have needed mod_rpaf and some modifications to their code.

In the end, Lighttpd and Nginx appeared to be the only documented solution. After the conversion, we did find documentation that said Varnish and Squid were immune to Slowloris. With Nginx or Lighttpd, we didn’t have IP address issues to contend with, it would be easy enough to modify the fastcgi config to pass the GEOIP information in the same request variable that their application expected. We knew we had to run PHP under FastCGI, so, we might as well pick a tool where we can block the attack in the webserver without having to worry about firewall rules. We did put a few firewall rules in place to block the larger offenders.

in the http { } section of our nginx config, we added:

    client_body_timeout 10;
    client_header_timeout 10;
    keepalive_timeout 10;
    send_timeout 10;
    limit_zone limit_per_ip $binary_remote_addr 16m;

and in the server { } section, we added:

    limit_conn limit_per_ip 5;

Since each server was expecting to handle one or two requests from each IP, this gave us a little headroom while solving the problem in the right place.

I believe Varnish would have held the connection and wouldn’t have sent a request to the backend which makes it fairly versatile as a tool to deal with DDOS attacks. While I do like the ability to block certain requests in VCL, the methods listed to fight this type of attack appeared to favor a non-threaded webserver. Varnish in front of Apache would have worked, but, we already knew we needed to move from Apache at some point with this client and this gave us an opportunity to shift them while under the gun.

Wouldn’t have had it any other way.

Our situation reminded me of a situation I encountered years ago. A client had an inhouse web designer and a subcontracted web designer. I felt the subcontracted web designer’s design would convert better. The client wasn’t completely convinced, but agreed to running two designs head to head. However, their implementation of the test biased the results.

What went wrong?

Each design was run for a week, in series. While this provided ample time for gathering data, the inhouse designer’s design ran during a national holiday with a three day weekend, and the subcontractor’s design ran the following week. Internet traffic patterns, the holiday weekend, weather, sporting events, TV/Movie premieres, etc. added so many variables which should have invalidated the results.

Since Google’s AB Testing has session persistence and splits traffic between the AB tests, we need to emulate this behavior. When people run AB tests in series rather than parallel, or, switch pages with a cron job or some other automated method, I cringe. A test at 5pm EST and 6pm EST will yield different results. At 5pm EST, your target audience could be driving home from work. At 6pm EST they could be sitting down for dinner.

How can Varnish help?

If we allow Varnish to select the landing page/offer page outside the origin server’s control, we can run both tests run at the same time. An internet logjam in Seattle, WA would affect both tests evenly. Likewise, a national or worldwide event would affect both tests equally. Now that we know how to make sure the AB Test is fairly balanced, we have to implement it.

Redirection sometimes plays havoc on browsers and spiders, so, we’ll rewrite the URL within Varnish using some Inline C and VCL. Google uses javascript and a document.location call to send some visitors to the B/alternate page. Users that have javascript disabled, will only see the Primary page.

Our Varnish config file contains the following:

sub vcl_recv {
  if (req.url == "/") {
    C{
      char buff[5];
      sprintf(buff,"%d",rand()%2 + 1);
      VRT_SetHdr(sp, HDR_REQ, "\011X-ABtest:", buff);
    }C
    set req.url = "/" req.http.X-ABtest "/" req.url;
  }
}

We’ve placed our landing pages in /1/ and /2/ directories on our origin server. The only page Varnish intercepts is the index page at the root of the site. Varnish randomly chooses to serve the index.html page from /1/ or /2/, internally rewrites our URL and serves it from the cache or the origin server. Since the URL rewriting is done within vcl_recv, subsequent requests for the page don’t hit the origin. The same method can be used to test landing pages that aren’t at the root of your site by modifying the if (req.url == “”) { condition.

You can test multipage offers by placing additional pages within the /1/ and /2/ directories on your origin along with the signup form. Unlike Google’s AB Test, Varnish does not support session persistence. Reloading the root page will result in the surfer alternating between both test pages. Subsequent pages need to be loaded from /1/ or /2/ based on which landing page was selected.

When doing any AB Test, change as few variables as possible, document the changes, and analyze the difference between the results. Running at least 1000 views of each is an absolute minimum. While Google’s Multivariate test provides a lot more options, a simple AB test between two pages or site tours can give some insight into what works rather easily.

If you cannot use Google’s AB Test or the Multivariate Test, using their Funnels and Goals tool will still allow you to do AB Testing.

At the top of /etc/varnish/default.vcl

C{
  #include <stdlib.h>
  #include <stdio.h>
}C

and our vcl_recv function gets the following:

  if (req.url ~ "^/panel/") {
    C{
      char buff[5];
      sprintf(buff,"%d",rand()%4);
      VRT_SetHdr(sp, HDR_REQ, "\010X-Panel:", buff);
    }C
    set req.url = regsub(req.url, "^/panel/(.*)\.(.*)$", "/panel/\1.ZZZZ.\2");
    set req.url = regsub(req.url, "ZZZZ", req.http.X-Panel);
  }

The above code allows for us to specify the source code in the html document as:

<img src="/panel/random.jpg" width="300" height="300" alt="Panel Image"/>

Since we have modified the request uri in vcl_recv before the object is cached, subsequent requests for the same modified URI will be served from Varnish’s cache, without requiring another fetch from the origin server. Based on the other VCL and preferences, you can specify a long expire time, remove cookies, or do ESI processing. Since the regexp passes the extension through, we could also randomly choose .html, .css, .jpg or any other extension you desire.

In the directory panel, you would need to have

/panel/random.0.jpg
/panel/random.1.jpg
/panel/random.2.jpg
/panel/random.3.jpg

which would be served by Varnish when the url /panel/random.jpg is requested.

Moving that process to Varnish should cut down on the load from the origin server while making your site look active and dynamic.

Since the machine hadn’t been having issues before, the first thing we did was contact the client and ask what had changed. We already knew the site and database that was using most of the CPU time, but, the bandwidth graph didn’t suggest that it was traffic overrunning the server. Our client rescued this client from another hosting company because the site was unusable in during prime time. So, we’ve inherited a problem. During the move, the site was upgraded from 1.0 to 1.5, so, we didn’t even have a decent baseline to revert to.

The stopgap solution was to move the .htaccess mod_rewrite rules into the apache configuration which helped somewhat. We identified a few sections of the code that were getting hit really hard and wrote a mod_rewrite rule to serve those images direct from disk — bypassing Joomla serving those images through itself. This made a large impact and at least got the site responsive enough that we could leave it online and work through the admin to figure out what had gone wrong.

Some of the modules that had been enabled contributed to quite a bit of the performance headache. One chat module generated 404s every second for each person logged in to see if there were any pending messages. Since Joomla is loaded for each 404 file, this added quite a bit of extra processing. Another quick modification to the configuration eliminated dozens of bad requests. At this point, the server is responsive, the client is happy and we make notes in the trouble ticket system and our internal documentation for reference.

Three days later the machine alerts and our load problem is back. After all of the changes, something is still having problems. Upon deeper inspection, we find that portions of the system dealing with the menus are being recreated each time. There’s no built in caching, so, the decision is to try Varnish. Varnish has worked in the past for WordPress sites that have gotten hit hard, so, we figured if we could cache the images, css and some of the static pages that don’t require authentication, we can get the server to be responsive again.

Apart from the basic configuration, our varnish.vcl file looked like this:

sub vcl_recv {
  if (req.http.host ~ "^(www.)?domain.com$") {
     set req.http.host = "domain.com";
  }

 if (req.url ~ "\.(png|gif|jpg|ico|jpeg|swf|css|js)$") {
    unset req.http.cookie;
  }
}

sub vcl_fetch {
 set obj.ttl = 60s;
 if (req.url ~ "\.(png|gif|jpg|ico|jpeg|swf|css|js)$") {
      set obj.ttl = 3600s;
 }
}

To get the apache logs to report the IP, you need to modify the VirtualHost config to log the forwarded IP.

The performance of the site after running Varnish in front of Apache was quite good. Apache was left with handling only .php and the server is again responsive. It runs like this for a week or more without any issues and only a slight load spike here or there.

However, Joomla doesn’t like the fact that every request’s REMOTE_ADDR is 127.0.0.1 and some addons stop working. In particular an application that allows the client to upload .pdf files into a library requires a valid IP address for some reason. Another module to add a sub-administration panel for a manager/editor also requires an IP address other than 127.0.0.1.

With some reservation, we decide to switch to Nginx + FastCGI which removes the reverse proxy and should fix the IP address problems.

Our configuration for Nginx with Joomla:

server {
        listen 66.55.44.33:80;
	server_name  www.domain.com;
 	rewrite ^(.*) http://domain.com$1 permanent;
}
server {
        listen 66.55.44.33:80;
	server_name  domain.com;

	access_log  /var/log/nginx/domain.com-access.log;

	location / {
		root   /var/www/domain.com;
		index  index.html index.htm index.php;

           if ( !-e $request_filename ) {
             rewrite (/|\.php|\.html|\.htm|\.feed|\.pdf|\.raw|/[^.]*)$ /index.php last;
             break;
           }

	}

	error_page   500 502 503 504  /50x.html;
	location = /50x.html {
		root   /var/www/nginx-default;
	}

	location ~ \.php$ {
		fastcgi_pass   unix:/tmp/php-fastcgi.socket;
		fastcgi_index  index.php;
		fastcgi_param  SCRIPT_FILENAME  /var/www/domain.com/$fastcgi_script_name;
		include	fastcgi_params;
	}

        location = /modules/mod_oneononechat/chatfiles/ {
           if ( !-e $request_filename ) {
             return 404;
           }
        }
}

With this configuration, Joomla was handed any URL for a file that didn’t exist. This was to allow the Search Engine Friendly (SEF) links. The second 404 handler was to handle the oneononechat module which looks for messages destined for the logged in user.

With Nginx, the site is again responsive. Load spikes occur from time to time, but, the site is stable and has a lot less trouble dealing with the load. However, once in a while the load spikes, but, the server seems to recover pretty well.

However, a module called Rokmenu which was included with the template design appears to have issues. Running php behind FastCGI sometimes gives different results than running as mod_php and it appears that Rokmenu is relying on the path being passed and doesn’t normalize it properly. So, when the menu is generated, with SEF on or off, urls look like /index.php/index.php/index.php/components/com_docman/themes/default/images/icons/16×16/pdf.png.

Obviously this creates a broken link and causes more 404s. We installed a fresh Joomla on Apache, imported the data from the copy running on Nginx, and Apache with mod_php appears to work properly. However, the performance is quite poor.

In order to troubleshoot, we made a list of every addon and ran through some debugging. With apachebench, we wrote up a quick command line that could be pasted in at the ssh prompt and decided upon some metrics. Within minutes, our first test revealed 90% of our performance issue. Two of the addons required compatibility mode because they were written for 1.0 and hadn’t been updated. Turning on compatibility mode on our freshly installed site resulted in 10x worse performance. As a test, we disabled the two modules that relied on compatibility mode and turned off compatibility mode and the load dropped immensely. We had disabled SEF early on thinking it might be the issue, but, we found the performance problem almost immediately. Enabling other modules and subsequent tests showed marginal performance changes. Compatibility mode was our culprit the entire time.

The client started a search for two modules to replace the two that required compatibility mode and disabled them temporarily while we moved the site back to Apache to fix the url issue in Rokmenu. At this point, the site was responsive, though, pageloads with lots of images were not as quick as they had been with Nginx or Varnish. At a later point, images and static files will be served from Nginx or Varnish, but, the site is fairly responsive and handles the load spikes reasonably well when Googlebot or another spider hits.

In the end the site ended up running on Apache because Varnish and Nginx had minor issues with the deployment. Moving to Apache alternatives doesn’t always fix everything and may introduce side-effects that you cannot work around.

url: 'red5StreamDemo', live: true

You will need to use Flash Media Encoder version 2.5 which can be downloaded at adobe.com. Flash Media Encoder version 3.0 will not work with red5 and based on developer comments, will probably not be supported

It is against the license for Flash Media Encoder to use it with anything other than Flash Media Server.

Here is a screenshot of the config required to get Flash Media Encoder 2.5 to work with red5. Replace yourhostname with the name of your red5 server.

After you’ve configured the FMS Url and the Stream, click the Connect button, and then the Start button at the bottom to start encoding.

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
	"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> 

<script src="flowplayer-3.1.1.min.js">	

<style>
a.rtmp {
	display:block;
	width:640px;
	height:360px;
	margin:25px 0;
	text-align:center;
}

a.rtmp img {
	border:0px;
	margin-top:140px;
}
</style>

<a class="rtmp" href="#">
<img src="http://static.flowplayer.org/img/player/btn/showme.png"  />
</a>

<script type="text/javascript">

$f("a.rtmp", "flowplayer-3.1.1.swf", { 

	clip: {
		provider: 'rtmp',
// name of the .flv from /webapps/oflaDemo/streams without the .flv
                url: 'IronMan'
	}, 

	plugins: {
	  rtmp: {
		url: '/fprt/flowplayer.rtmp-3.1.0.swf',
		netConnectionUrl: 'rtmp://hostname/oflaDemo'
	  }
	}
});
</script>

Make sure you have the following files:

* flowplayer-3.1.1.swf
* flowplayer.controls-3.1.1.swf
* flowplayer.rtmp-3.1.0.swf

If you receive the error:

300: Player Initialization Failed: TypeError: Error #1009

Then you are missing flowplayer.controls-3.1.1.swf from your directory.

In the javascript,

netConnectionUrl: ‘rtmp://hostname/oflaDemo’

needs to be set the hostname of your red5 server.

url: ‘IronMan’

needs to be set to the name of the .flv in the webapps/oflaDemo/streams directory without the .flv extension

/usr/lib/red5/webapps/oflaDemo/streams# ls -1 *.flv
DarkKnight.flv
IronMan.flv
on2_flash8_w_audio.flv
test.flv

This one client has 3 machines that ran Apache2-mpm-worker with php5 running under fastcgi.  While page response time was good, the machines constantly ran at roughly 15% idle cpu time, with roughly 600mb-700mb of the ram used for cache.  All of the machines are quadcore with 4gb RAM and have been running for quite a while and have been tweaked and tuned along the way.

We started with the conversion of one site on one machine which resulted in the client being so impressed that we converted a second site on that machine which resulted in about 80mb/sec being served from nginx within minutes of deployment.  The next morning after we glanced over everything and confirmed that nginx was holding up, we converted the rest of that machine over to Nginx.  Traffic grew almost 20% after that change.

We started looking at the other machines, one of which runs phpadsnew on a relatively large network of his sites and the banners that are served from two of the main sites on one machine.  Converting those two over to nginx meant another 50mb/sec of traffic swapped from Apache.  Immediately he saw results with faster pageloads of his sites that pulled content from a central domain and with the banner ads being displayed more quickly.  After a few moments of analysis, it was decided to swap the entire machine from Apache2 to Nginx.  That process took a few hours due to the number of virtual hosts and the lack of any real script to migrate the configurations.  Response time on the sites was definitely faster.  After a little more discussion, rather than give that machine a day to settle in to see if we would find any problems, we converted his third machine.

First response in the morning:

yesterday we sent 69.1k unique surfers to sponsors, that is the highest we have ever done.

While only one of three machines was running Nginx for the entire day, one machine had about 8 hours under Nginx and the other about 2 hours under Nginx for that ‘day.’

Today, the results are somewhat clear.  Traffic is up overall, the machines are much more responsive.  Each machine is now roughly 80% idle and has roughly 2.4gb of memory reserved for cache.

Backups are scheduled at 3am on the boxes, a few rsync jobs are run to keep some content directories synced between the machines.  Overall you can see the impact on the first graph as the right hand side shows a bit more growth.  The last graph was running nginx, but, struggled to push more than 85mb/sec or so.  The middle graph shows a decline, but, they believe that is external to the process.  The sites are loading more quickly and they expect that the sites will grow quite a bit.  So far, they are reporting roughly an 18% increase in clicks to their sponsor.

First things first, we evaluated where the issue was with WordPress and installed db-cache and wp-cache-2. We had tried wp-super-cache but had seen some issues with it in some configurations. Immediately the pageload time dropped from 41 seconds to 11 seconds. Since the machine was running on a quadcore with 4gb ram and was running mostly idle, the only thing left was the 91 page elements being served. Each pageload, even with pipelining still seemed to cause some stress. Two external javascripts and one external flash object caused some delay in rendering the page. The javascripts were actually responsible for holding up the page rendering which made the site seem even slower than it was. We made some minor modifications, but, while apache2 was configured to serve things as best it could, we felt there was still some room for improvement.

While I had tested Varnish in front of Apache2, I knew it would make an impact in this situation due to the number of elements on the page and the fact that apache did a lot of work to serve each request. Varnish and its VCL eliminated a lot of the overhead Apache had and should result in the capacity for roughly 70% better performance. For this installation, we removed the one IP that was in use by the problem domain from Apache and used that for Varnish and ran Varnish on that IP, using 127.0.0.1 port 80 as the backend.

Converting a site that is in production and live is not for the fainthearted, but, here are a few notes.

For Apache you’ll want to add a line like this to make sure your logs show the remote IP rather than the IP of the Varnish server:

LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-A
gent}i\"" varnishcombined

Modify each of the VirtualHost configs to say:

<VirtualHost 127.0.0.1:80>

and change the line for the logfile to say:

CustomLog /var/log/apache2/domain.com-access.log varnishcombined

Add Listen Directives to prevent Apache from listening to port 80 on the IP address that you want varnish to answer and comment out the default Listen 80:

#Listen 80
Listen 127.0.0.1:80
Listen 66.55.44.33:80

Configuration changes for Varnish:

backend default {
.host = "127.0.0.1";
.port = "80";
}

sub vcl_recv {
  if (req.url ~ "\.(jpg|jpeg|gif|png|tiff|tif|svg|swf|ico|mp3|mp4|m4a|ogg|mov|avi|wmv)$") {
      lookup;
  }

  if (req.url ~ "\.(css|js)$") {
      lookup;
  }
}
sub vcl_fetch {
        if( req.request != "POST" )
        {
                unset obj.http.set-cookie;
        }

        set obj.ttl = 600s;
        set obj.prefetch =  -30s;
        deliver;
}

Shut down Apache, Restart Apache, Start Varnish.

tail -f the logfile for Apache for one of the domains that you have moved. Go to the site. Varnish will load everything the first time, but, successive reloads shouldn’t show requests for images, javascript, css. For this client we opted to hold things in cache for 10 minutes (600 seconds).

Overall, the process was rather seamless. Unlike converting a site to Nginx, we are not required to make changes to the rewrite config or worry about setting up a fastcgi server to answer .php requests. Overall, varnish is quite seamless to the end product. Clients will lose the ability to do some things like deny hotlinking, but, Varnish will run almost invisibly to the client. Short of the page loading considerably quicker, the client was not aware we had made any server changes and that is the true measure of success.

The conversion of one virtualhost on that machine resulted in a few minor hitches. Rewrite rules are a little different and while our conversion of those rules mostly worked, a few minor differences in the syntax cropped up and needed slight adjustments.

RewriteRule ^external/([0-9]+)/? external.php?vid=$1 [L]

changes to

rewrite "^/external/([0-9]+)/?" /external.php?vid=$1 last;

The leading / is now required in both places but the rule converts over fairly nicely.

Performance tuning is tricky at best since there aren’t too many documents that explain the different config arguments, and, very few that explain how to diagnose and tune. Most is done through trial and error watching the processes, watching logs, seeing the system react and making adjustments.

Virtual Host configuration was a challenge at first as the documentation assumes that the machine will just listen on port 80. When the machine shares the IPs with Apache which is also answering on port 80 and you’re just moving a few things over, you need to make some minor changes.

server {
        listen 66.55.44.33:80 default;
        server_name  _;

        access_log  /var/log/nginx/access.log;

        location / {
                root   /var/www/uc;
                index  index.html;
        }
}

server {
        listen 66.55.44.33:80;
        server_name  www.domain.com domain.com;

made virtual hosting work when you are only able to listen to a few IPs.

Overall, I am reasonably impressed with Nginx. The machine we upgraded was pushing about 65mb/sec, with a load of 15 and roughly 15% idle CPU. After moving 2 domains over to Nginx, the machine almost instantly climbed to 80mb/sec with a load of 2 and roughly 85% idle. System Cache went from 880mb to 2.7gb and the number of Apache processes dropped from 350 to 40. The machine is incredibly responsive now and the pages load almost instantly.

I’ll continue to monitor it, but, at this point it looks like a userland process will challenge Tux’s performance.