Google Voice Call Log Time Warp

May 24th, 2010

While I have been using Google Voice more regularly, I haven’t made the switch to using it as my primary number. It is a little odd to place a call from the web, have my office phone ring, then hear the ring on the other side, but, having my numbers in one place, a call log and many other useful features is quite handy. While I’ve only ever given out one phone number for the last ten years, I’ve maintained that with creative use of busy call forwarding, ultra call forwarding and a number of other calling features over the years. While it works, Google Voice does handle things a little better.

However, on a recent call, it appears that I made a call that was 13 minutes long, 12 minutes ago. Google knew when my call would end.

While there are quite a few odd things about Google Voice, I am getting closer to using it for my permanent number.

Nginx to Apache?

May 23rd, 2010

A few months ago we had a client that wanted to run Nginx/FastCGI rather than Apache because it was known to be faster. While we’ve had extensive experience performance tuning various webserver combinations, the workload proposed would really have been better served with Apache. While we inherited this problem from another hosting company — he moved because they couldn’t fix the performance issues — he maintained that Nginx/FastCGI for PHP was the fastest because of all of the benchmarks that had been run on the internet.

While the conversion to or from one server to another is usually painful, much of the pain can be avoided by running Apache on an alternate port, testing, then, swapping the configuration around. The graph below shows when we changed from Nginx to Apache:

We made the conversion from Nginx to Apache on Friday. Once we made the conversion, there were issues with the machine which was running an older kernel. After reviewing the workload, we migrated from 2.6.31.1 with the Anticipatory Scheduler to 2.6.34 with the Deadline Scheduler. Three other machines had been running 2.6.33.1 with the CFQ scheduler and showed no issues at the 10mb/sec mark, but, we felt that we might benchmark his workload using deadline. We’ve run a number of high-end webservers with both Anticipatory and CFQ prior to 2.6.33 and for most of our workloads, Anticipatory seemed to win. With 2.6.33, Anticipatory was removed, leaving NOOP, CFQ and Deadline. While we have a few MySQL servers running Deadline, this is probably the first heavy-use webserver that we’ve moved from CFQ/AS to Deadline.

The dips in the daily graph were during times where a cron job was running. The two final dips were during the kernel installation.

All in all, the conversion went well. The machine never really appeared to be slow, but, it is obvious that it is now handling more traffic. The load averages are roughly the same as they were before. CPU utilization is roughly the same, but, more importantly, Disk I/O is about half what it was and System now hovers around 3-4%. During the hourly cron job, the machine is not having issues like it was before.

Nginx isn’t always the best solution. In this case, 100% of the traffic is serving an 8k-21k php script to each visitor. Static content is served from another machine running Nginx.

While I do like Nginx, it is always best to use the right tool for the job. In this case, Apache happened to be the right tool.

DDOS attack mitigation

April 26th, 2010

Today we had a DDOS attack on one of our clients. They were running prefork with mod_php5 with a rather busy application. While we initially started filtering IP addresses using iptables and a few crudely hacked rules, we knew something had to be done that was a little more permanent. Moving to MPM-Worker with PHP served with FastCGI seemed reasonable, but, looking at the history of the attacks on this machine, I believe Apache still would have been vulnerable since we cannot filter the requests early enough in Apache’s request handler.

Apache does have the ability to fight some DDOS attacks using mod_security and mod_evasive, but, this particular attack was designed to affect apache prior to the place where these modules hook into the request. This also precludes using fail2ban. We could run mod_forensic or mod_logio to assist fail2ban, but, it is still a stopgap measure.

We could have used some Snort rules and tied those to iptables, but, that is a rather bad solution to the problem.

While we could have used Varnish, their application would have had some issues. mod_rpaf can help by adjusting the REMOTE_ADDR to take the value from X-Forwarded-For that Varnish sets. mod_geoip actually inserts itself before mod_rpaf, so, we would have needed to make a modification to mod_geoip and recompiled it. I’m not sure how Varnish would have handled Slowloris and we had to fix this now.

Putting them behind a Layer 7 load balancer would have isolated the origin server and handled the brunt of the attack on the load balancer, but, again we would have needed mod_rpaf and some modifications to their code.

In the end, Lighttpd and Nginx appeared to be the only documented solution. After the conversion, we did find documentation that said Varnish and Squid were immune to Slowloris. With Nginx or Lighttpd, we didn’t have IP address issues to contend with, it would be easy enough to modify the fastcgi config to pass the GEOIP information in the same request variable that their application expected. We knew we had to run PHP under FastCGI, so, we might as well pick a tool where we can block the attack in the webserver without having to worry about firewall rules. We did put a few firewall rules in place to block the larger offenders.

in the http { } section of our nginx config, we added:

    client_body_timeout 10;
    client_header_timeout 10;
    keepalive_timeout 10;
    send_timeout 10;
    limit_zone limit_per_ip $binary_remote_addr 16m;

and in the server { } section, we added:

    limit_conn limit_per_ip 5;

Since each server was expecting to handle one or two requests from each IP, this gave us a little headroom while solving the problem in the right place.

I believe Varnish would have held the connection and wouldn’t have sent a request to the backend which makes it fairly versatile as a tool to deal with DDOS attacks. While I do like the ability to block certain requests in VCL, the methods listed to fight this type of attack appeared to favor a non-threaded webserver. Varnish in front of Apache would have worked, but, we already knew we needed to move from Apache at some point with this client and this gave us an opportunity to shift them while under the gun.

Wouldn’t have had it any other way.

Reading a file in bash

April 15th, 2010

While working on a maintenance script, I’ve found myself using bash a bit more than using perl for many tasks. While I don’t mind writing in perl or python, sometimes it just seems like overkill.

I needed to read one line of a /proc/loadavg for a monitoring system and was using some code that had been written by one of our programmers in 2002 or so. The old code was written using three scripts, two of which were called from cron. The first script was written in perl, actually included /proc/loadavg as chomp($value=`cat /proc/loadvg`);

Ugh!

Then some math was done, and a decision made of what values to write for nagios’s local process to report back. Suffice it to say that the math done then involved executing a shell script from time to time based on some condition checks and a separate cron that cleaned up the status for nagios.

A literal translation of the script from perl to bash ended up with a snippet of code that looked like this:

#!/bin/bash

  LOAD=`cat /proc/loadavg`
  LOADAVG=${LOAD%%.*}

echo $LOADAVG

We’ll spare the details regarding how badly the existing script converted LOADAVG to an int in perl. While this method works, it does cause a process to be forked needlessly to read /proc/loadavg.

A quick rewrite and we end up with:

#!/bin/bash

  read LOAD < /proc/loadavg
  LOADAVG=${LOAD%%.*}

echo $LOADAVG

While this doesn't seem overly bad for a process executed once a minute, it does cause a larger problem when someone copies that snippet of code and uses it elsewhere.

If we modify the code slightly to:

#!/bin/bash

for i in {1..10000}
do
  LOAD=`cat /proc/loadavg`
  LOADAVG=${LOAD%%.*}
done

echo $LOADAVG

and

#!/bin/bash

for i in {1..10000}
do
  read LOAD < /proc/loadavg
  LOADAVG=${LOAD%%.*}
done

echo $LOADAVG

and run the following benchmark:

root@tsavo:/home/cd34# time ./s.sh 
0

real	0m13.641s
user	0m0.636s
sys	0m2.304s
root@tsavo:/home/cd34# time ./t.sh 
0

real	0m0.485s
user	0m0.332s
sys	0m0.148s

we can see that there is a substantial impact to doing things.

Taking the integer portion of a floating point number also yields a number of terrible solutions involving sed, awk and cut — all forking a separate process or two.

WordPress Cache Plugin Benchmarks

March 4th, 2010

A lot of time and effort goes into keeping a WordPress site alive when it starts to accumulate traffic. While not every site has the same goals, keeping a site responsive and online is the number one priority. When a surfer requests the page, it should load quickly and be responsive. Each addon handles caching a little differently and should be used in different cases.

For many sites, page caching will provide decent performance. Once your sites starts receiving comments, or people log in, many cache solutions cache too heavily or not enough. As many solutions as there are, it is obvious that WordPress underperforms in higher traffic situations.

The list of caching addons that we’re testing:

* DB Cache (version 0.6)
* DB Cache Reloaded (version 2.0.2)
* W3 Total Cache (version 0.8.5.1)
* WP Cache (version 2.1.2)
* WP Super Cache (version 0.9.9)
* WP Widget Cache (version 0.25.2)
* WP File Cache(version 1.2.5)
* WP Varnish (in beta)
* WP Varnish ESI Widget (in beta)

What are we testing?

* Frontpage hits
* httpload through a series of urls

We take two measurements. The cold start measurement is taken after any plugin cache has been cleared and Apache2 and MySQL have been restarted. A 30 second pause is inserted prior to starting the tests. We perform a frontpage hit 1000 times with 10 parallel connections. We then repeat that test after Apache2 and the caching solution have had time to cache that page. Afterwards, http_load requests a series of 30 URLs to simulate people surfing other pages. Between those two measurements, we should have a pretty good indicator of how well a site is going to perform in real life.

What does the Test Environment look like?

* Debian 3.1/Squeeze VPS
* Linux Kernel 2.6.33
* Single core of a Xen Virtualized Xeon X3220 (2.40ghz)
* 2gb RAM
* CoW file is written on a Raid-10 System using 4x1tb 7200RPM Drives
* Apache 2.2.14 mpm-prefork
* PHP 5.3.1
* WordPress Theme Test Data
* Tests are performed from a Quadcore Xeon machine connected via 1000 Base T on the same switch and /24 as the VPS machine

This setup is designed to replicate what most people might choose to host a reasonably popular wordpress site.

tl;dr Results

If you aren’t using Varnish in front of your web site, the clear winner is W3 Total Cache using Page Caching – Disk (Enhanced), Minify Caching – Alternative PHP Cache (APC), Database Caching – Alternative PHP Cache (APC).

If you can use Varnish, WP Varnish would be a very simple way to gain quite a bit of performance while maintaining interactivity. WP Varnish purges the cache when posts are made, allowing the site to be more dynamic and not suffer from the long cache delay before a page is updated.

W3 Total Cache has a number of options and sometimes settings can be quite detrimental to site performance. If you can’t use APC caching or Memcached for caching Database queries or Minification, turn both off. W3 Total Cache’s interface is overwhelming but the plugin author has indicated that he’ll be making a new ‘Wizard’ configuration menu in the next version along with Fragment Caching.

WP Super Cache isn’t too far behind and is also a reasonable alternative.

Either way, if you want your site to survive, you need to use a cache addon. Going from 2.5 requests per second to 800+ requests per second makes a considerable difference in the usability of your site for visitors. Logged in users and search engine bots still see uncached/live results, so, you don’t need to worry that your site won’t be indexed properly.

Results

Sorted in Ascending order in terms of higher overall performance

Addon Apachebench Cold Start
Warm Start
http_load Cold Start
Warm Start
Req/Second Time/Request 50% within x ms Fetches/Second Min First Response Avg First Response
Baseline 4.97 201.006 2004 15.1021 335.708 583.363
5.00 200.089 2000 15.1712 304.446 583.684
DB Cache 4.80 208.436 2087 15.1021 335.708 583.363
Cached all SQL queries 4.81 207.776 2091 15.1712 304.446 583.684
DB Cache 4.87 205.250 2035 14.1992 302.335 621.092
Out of Box config 4.94 202.624 2026 14.432 114.983 618.434
WP File Cache 4.95 201.890 2009 15.8869 158.597 549.176
4.99 200.211 2004 16.1758 99.728 544.107
DB Cache Reloaded 5.02 199.387 1983 15.0167 187.343 589.196
All SQL Queries Cached 5.03 200.089 1985 14.9233 150.145 586.443
DB Cache Reloaded 5.06 197.636 1968 14.9697 174.857 589.161
Out of Box config 5.08 196.980 1968 15.181 257.533 587.737
Widgetcache 6.667 149.903 1492 15.0264 245.332 602.039
6.72 148.734 1487 15.1887 299.65 598.017
W3 Total Cache 153.45 65.167 60 133.1898 8.916 85.7177
DB Cache off, Page Caching with Memcached 169.46 59.011 57 188.4 9.107 50.142
W3 Total Cache 173.49 57.639 52 108.898 7.668 86.4077
DB Cache off, Minify Cache with Memcached 189.76 52.698 48 203.522 8.122 43.8795
W3 Total Cache 171.34 58.364 50 203.718 8.097 44.1234
DB Cache using Memcached 190.01 52.269 48 206.187 8.186 42.4438
W3 Total Cache 175.29 57.048 48 87.423 7.515 107.973
Out of Box config 191.15 52.314 47 204.387 8.288 43.217
W3 Total Cache 175.29 57.047 51 204.557 8.199 42.9365
Database Cache using APC 191.19 52.304 48 200.612 8.11 44.6691
W3 Total Cache 114.02 87.703 49 114.393 8.206 82.0678
Database Cache Disabled 191.76 52.150 49 203.781 8.095 42.558
W3 Total Cache 175.80 56.884 51 107.842 7.281 87.2761
Database Cache Disabled, Minify Cache using APC 192.01 52.082 50 205.66 8.244 43.1231
W3 Total Cache 104.90 95.325 51 123.041 7.868 74.5887
Database Cache Disabled, Page Caching using APC 197.55 50.620 46 210.445 7.907 41.4102
WP Super Cache 336.88 2.968 16 15.1021 335.708 583.363
Out of Box config, Half On 391.59 2.554 16 15.1712 304.446 583.684
WP Cache 161.63 6.187 12 15.1021 335.708 583.363
482.29 20.735 11 15.1712 304.446 583.684
WP Super Cache 919.11 1.088 3 190.117 1.473 47.9367
Full on, Lockdown mode 965.69 1.036 3 975.979 1.455 9.67185
WP Super Cache 928.45 1.077 3 210.106 1.468 43.8167
Full on 970.45 1.030 3 969.256 1.488 9.78753
W3 Total Cache 1143.94 8.742 2 165.547 0.958 56.7702
Page Cache using Disk Enhanced 1222.16 8.182 3 1290.43 0.961 7.15632
W3 Total Cache 1153.50 8.669 3 165.725 0.916 56.5004
Page Caching – Disk Enhanced, Minify/Database using APC 1211.22 8.256 2 1305.94 0.948 6.97114
Varnish ESI 2304.18 0.434 4 349.351 0.221 28.1079
2243.33 0.44689 4 4312.78 0.152 2.09931
WP Varnish 1683.89 0.594 3 369.543 0.155 26.8906
3028.41 0.330 3 4318.48 0.148 2.15063

Test Script

#!/bin/sh

FETCHES=1000
PARALLEL=10

/usr/sbin/apache2ctl stop
/etc/init.d/mysql restart
apache2ctl start
echo Sleeping
sleep 30
time ( \
echo First Run; \
ab -n $FETCHES -c $PARALLEL http://example.com/; \
echo Second Run; \
ab -n $FETCHES -c $PARALLEL http://example.com/; \
\
echo First Run; \
./http_load -parallel $PARALLEL -fetches $FETCHES wordpresstest; \
echo Second Run; \
./http_load -parallel $PARALLEL -fetches $FETCHES wordpresstest; \
)

URL File for http_load

http://example.com/
http://example.com/2010/03/hello-world/
http://example.com/2008/09/layout-test/
http://example.com/2008/04/simple-gallery-test/
http://example.com/2007/12/category-name-clash/
http://example.com/2007/12/test-with-enclosures/
http://example.com/2007/11/block-quotes/
http://example.com/2007/11/many-categories/
http://example.com/2007/11/many-tags/
http://example.com/2007/11/tags-a-and-c/
http://example.com/2007/11/tags-b-and-c/
http://example.com/2007/11/tags-a-and-b/
http://example.com/2007/11/tag-c/
http://example.com/2007/11/tag-b/
http://example.com/2007/11/tag-a/
http://example.com/2007/09/tags-a-b-c/
http://example.com/2007/09/raw-html-code/
http://example.com/2007/09/simple-markup-test/
http://example.com/2007/09/embedded-video/
http://example.com/2007/09/contributor-post-approved/
http://example.com/2007/09/one-comment/
http://example.com/2007/09/no-comments/
http://example.com/2007/09/many-trackbacks/
http://example.com/2007/09/one-trackback/
http://example.com/2007/09/comment-test/
http://example.com/2007/09/a-post-with-multiple-pages/
http://example.com/2007/09/lorem-ipsum/
http://example.com/2007/09/cat-c/
http://example.com/2007/09/cat-b/
http://example.com/2007/09/cat-a/
http://example.com/2007/09/cats-a-and-c/

Entries (RSS) and Comments (RSS).
Cluster host: li