{"id":606,"date":"2009-04-01T11:33:06","date_gmt":"2009-04-01T15:33:06","guid":{"rendered":"http:\/\/cd34.com\/blog\/?p=606"},"modified":"2009-04-01T11:33:06","modified_gmt":"2009-04-01T15:33:06","slug":"apache-varnish-nginx-and-lighttpd","status":"publish","type":"post","link":"https:\/\/cd34.com\/blog\/infrastructure\/apache-varnish-nginx-and-lighttpd\/","title":{"rendered":"Apache, Varnish, nginx and lighttpd"},"content":{"rendered":"

I’ve never been happy with Apache’s performance.\u00a0 It seemed that it always had problems with high volume sites.\u00a0 Even extremely tweaked configurations resulted in decent performance to a point which then required more hardware to continue going.\u00a0 While I had been a huge fan of Tux, sadly, Tux doesn’t work with Linux 2.6 kernels very well.<\/p>\n

So, the search was on.\u00a0 I’ve used many webservers over the years ranging from AOLServer to Paster to Caudium looking for a robust, high-performance solution.\u00a0 I’ve debated caching servers in front of Apache, a server to handle just static files and coding the web sites to utilize that, but, I never really found the ultimate solution to handle particular requirements.<\/p>\n

This current problem is a php driven site with roughly 100 page elements plus the generated page itself.\u00a0 The site receives quite a bit of traffic and we’ve had to tweak Apache quite a bit from our default configuration to keep the machine performing well.<\/p>\n

Apache can be run many different ways.\u00a0 Generally when a site uses php, we’ll run mod_php because it is faster.\u00a0 Eaccelerator can help sometimes — though, does create a few small problems, but, in general, Apache-mpm-prefork runs quite well.\u00a0 On sites where we’ve had issues with traffic, we’ve switched over to Apache-mpm-worker with a fastcgi php process.\u00a0 This works quite well even though php scripts are slightly slower.<\/p>\n

After considerable testing, I came up with three decent metrics that I used to judge things.\u00a0 Almost all testing was done with ab (apachebench) running 10000 connections with keepalives and 50 concurrent sessions from a dual quad-core xeon machine to a gigE connected machine on the same switch running a core2quad machine.\u00a0 On the first IP was bare apache, the second IP had lighttpd, the third IP ran nginx and the fourth IP ran Varnish in front of Apache.\u00a0 Everything was set up so that no restarts of daemons would need to be made, the tests were run twice with the second result generally being the higher of the two which was used.\u00a0 The linux kernel does some caching and we’re after the performance after the kernel has done its caching, apache has forked its processes and hasn’t killed off the children, etc.<\/p>\n

First impressions from Apache-mpm-prefork were that it handled php exceedingly well, but, has never had great performance with static files.\u00a0 This is why Tux prevailed for us as Apache handled what it did best and Tux handled what it did best.\u00a0 Regrettably, Tux didn’t keep up with the 2.6 kernel and development ceased.\u00a0 With new hardware, the 2.6 kernel and the ability for userland processes to get access to sendfile, large file transfer should be almost the same for all of the processes so, startup latency of the tiny files was what really seemed to harm Apache.\u00a0 Apache-mpm-worker with php running as fastcgi has always been a fallback for us to gain a little more serving capacity as most sites have a relatively heavy static file to dynamic file construction.<\/p>\n

But, Apache seemed to have problems with the type of traffic our clients are putting through and we felt that there had to be a better way.\u00a0 I’ve read page after page of people complaining about their Drupal installation being able to take 50 users and then they upgraded to nginx or lighttpd and now their site doesn’t run into swap issues.\u00a0 If your server is having problems with 50 simultaneous users with apache, you have serious problems with your setup.\u00a0 It is not uncommon for us to push a P4\/3.0ghz with 2gb ram with 80mb\/sec traffic and MySQL running 1000 queries per second.\u00a0 Where your apache logfile reaches 6gb\/day for a domain not including the other 30 domains configured on the machine.\u00a0 VBulletin will easily run 350 online users and 250 guests on the same hardware without any difficulties.\u00a0 The same with Joomla, Drupal and the other CMS products out there.\u00a0 If you can’t run 50 simultaneous users, with any of those products, dig into the configs FIRST so that you are comparing a tuned configuration to a tuned configuration.<\/p>\n

Uptime: 593254\u00a0 Threads: 571\u00a0 Questions: 609585858\u00a0 Slow queries: 1680967\u00a0 Opens: 27182\u00a0 Flush tables: 1\u00a0 Open tables: 2337\u00a0 Queries per second avg: 1027.529<\/p><\/blockquote>\n

\"86\"<\/a><\/p>\n

Based on all of my reading, I expected Varnish -> Apache2 to be the fastest followed by nginx, lighttpd and bare Apache.\u00a0 Lighttpd has some interesting design issues that I believed would put it behind nginx, I really expected Varnish would do really well.\u00a0 For this client, we needed the FLV streaming so, I knew I would be running nginx or lighttpd for a backend for the .flv files and contemplated running Varnish in front of whichever of those performed best.\u00a0 Splitting things so that the .flv files were served from a different domain was no problem for this client, so, we weren’t having to put a solution in place where we couldn’t make changes.<\/p>\n

The testing methodology was based on numerous runs of ab where I tested and tweaked each setup.\u00a0 I am reasonably sure that someone with vast knowledge of Varnish, nginx or lighttpd would not be able to substantially change the results.\u00a0 Picking out the three or four valid pieces of information from all of the testing to give me a generalized result was difficult.<\/p>\n

The first thing I was concerned with was the raw speed on a small 6.3kb file.\u00a0 With keepalives enabled, that was a good starting point.\u00a0 The second test was to run a page that called phpinfo();.\u00a0 Not an exceedingly difficult test, it does at least start the php engine, process a page and return the result.\u00a0 The third test was to download a 21mb flv file.\u00a0 All of the tests were run with 10000 iterations and 50 concurrent threads except the 21mb flv file which ran 100 iterations and 10 concurrent threads due to the time it took.<\/p>\n\n\n\n\n\n\n\n\n
Server<\/td>\nSmall File Requests Per Second<\/td>\nphpinfo() Requests Per Second<\/td>\n.flv MB\/Sec<\/td>\nMin\/Max time to serve .flv<\/td>\nTime to run ab for .flv test<\/td>\n<\/tr>\n
Apache-mpm-prefork<\/td>\n1000<\/td>\n164<\/td>\n11.5MB\/sec<\/td>\n10-26 seconds<\/td>\n182 seconds<\/td>\n<\/tr>\n
Apache-mpm-worker<\/td>\n1042<\/td>\n132<\/td>\n11.5MB<\/td>\n11-25 seconds<\/td>\n181 seconds<\/td>\n<\/tr>\n
Lighttpd<\/td>\n1333<\/td>\n181<\/td>\n11.4MB<\/td>\n13-23 seconds<\/td>\n190 seconds<\/td>\n<\/tr>\n
nginx<\/td>\n1800<\/td>\n195<\/td>\n11.5MB<\/td>\n14-24 seconds<\/td>\n187 seconds<\/td>\n<\/tr>\n
Varnish<\/td>\n1701<\/td>\n198<\/td>\n11.3MB<\/td>\n18-30 seconds<\/td>\n188 seconds<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Granted, I expected more from Varnish and it’s caching nature does shine through.\u00a0 It is considerably more powerful than nginx due to some of the internal features it has for load balancing, multiple backends, etc.\u00a0 However, based on the results above, I have to believe that in this case, nginx wins.<\/p>\n

There are a number of things about the nginx documentation that were confusing.\u00a0 First was that they used inet rather than a local socket for communication with the php-cgi process.\u00a0 That alone bumped up php almost 30 transactions per second.\u00a0 The documentation for nginx is sometimes very terse and it required a bit more time to get configured correctly.\u00a0 While I do have both php and perl cgi working with nginx natively, some perl cgi scripts do have minor issues which I’m still working out.<\/p>\n

Lighttpd performed about as well as I expected.\u00a0 Due to some backend design issues, there are some things that made me believe it wouldn’t be the top performer.\u00a0 It is also older and more mature than Nginx and Varnish which use today’s tricks to accomplish their magic.\u00a0 File transfer speed is going to be somewhat capped because the Linux kernel opens up some APIs that allow a userspace application to ask the kernel to handle the transfer.\u00a0 Every application tested takes advantage of this.<\/p>\n

Given the choice of Varnish or Nginx for a project that didn’t require .flv streaming, I might consider Varnish.\u00a0 Lighttpd did have one very interesting module that prevented hotlinking of files in a much different manner than normal — I’ll be testing that for another application. If you are used to Apache mod_rewrite rules, Nginx and Lighttpd have a completely different structure for these.\u00a0 They work in almost the same manner with some minor syntax changes.\u00a0 Varnish runs as a cache to the frontend of your site, so, everything works with it the same way it does under Apache since Varnish merely connects to your Apache backend and caches what it can.\u00a0 Its configuration language allows considerable control over the process.<\/p>\n

Short of a few minor configuration tweaks, this particular client will be getting nginx.<\/p>\n