WordPress Cache Plugin Benchmarks

A lot of time and effort goes into keeping a WordPress site alive when it starts to accumulate traffic. While not every site has the same goals, keeping a site responsive and online is the number one priority. When a surfer requests the page, it should load quickly and be responsive. Each addon handles caching a little differently and should be used in different cases.

For many sites, page caching will provide decent performance. Once your sites starts receiving comments, or people log in, many cache solutions cache too heavily or not enough. As many solutions as there are, it is obvious that WordPress underperforms in higher traffic situations.

The list of caching addons that we’re testing:

* DB Cache (version 0.6)
* DB Cache Reloaded (version 2.0.2)
* W3 Total Cache (version 0.8.5.1)
* WP Cache (version 2.1.2)
* WP Super Cache (version 0.9.9)
* WP Widget Cache (version 0.25.2)
* WP File Cache(version 1.2.5)
* WP Varnish (in beta)
* WP Varnish ESI Widget (in beta)

What are we testing?

* Frontpage hits
* httpload through a series of urls

We take two measurements. The cold start measurement is taken after any plugin cache has been cleared and Apache2 and MySQL have been restarted. A 30 second pause is inserted prior to starting the tests. We perform a frontpage hit 1000 times with 10 parallel connections. We then repeat that test after Apache2 and the caching solution have had time to cache that page. Afterwards, http_load requests a series of 30 URLs to simulate people surfing other pages. Between those two measurements, we should have a pretty good indicator of how well a site is going to perform in real life.

What does the Test Environment look like?

* Debian 3.1/Squeeze VPS
* Linux Kernel 2.6.33
* Single core of a Xen Virtualized Xeon X3220 (2.40ghz)
* 2gb RAM
* CoW file is written on a Raid-10 System using 4x1tb 7200RPM Drives
* Apache 2.2.14 mpm-prefork
* PHP 5.3.1
* WordPress Theme Test Data
* Tests are performed from a Quadcore Xeon machine connected via 1000 Base T on the same switch and /24 as the VPS machine

This setup is designed to replicate what most people might choose to host a reasonably popular wordpress site.

tl;dr Results

If you aren’t using Varnish in front of your web site, the clear winner is W3 Total Cache using Page Caching – Disk (Enhanced), Minify Caching – Alternative PHP Cache (APC), Database Caching – Alternative PHP Cache (APC).

If you can use Varnish, WP Varnish would be a very simple way to gain quite a bit of performance while maintaining interactivity. WP Varnish purges the cache when posts are made, allowing the site to be more dynamic and not suffer from the long cache delay before a page is updated.

W3 Total Cache has a number of options and sometimes settings can be quite detrimental to site performance. If you can’t use APC caching or Memcached for caching Database queries or Minification, turn both off. W3 Total Cache’s interface is overwhelming but the plugin author has indicated that he’ll be making a new ‘Wizard’ configuration menu in the next version along with Fragment Caching.

WP Super Cache isn’t too far behind and is also a reasonable alternative.

Either way, if you want your site to survive, you need to use a cache addon. Going from 2.5 requests per second to 800+ requests per second makes a considerable difference in the usability of your site for visitors. Logged in users and search engine bots still see uncached/live results, so, you don’t need to worry that your site won’t be indexed properly.

Results

Sorted in Ascending order in terms of higher overall performance

Addon Apachebench Cold Start
Warm Start
http_load Cold Start
Warm Start
Req/Second Time/Request 50% within x ms Fetches/Second Min First Response Avg First Response
Baseline 4.97 201.006 2004 15.1021 335.708 583.363
5.00 200.089 2000 15.1712 304.446 583.684
DB Cache 4.80 208.436 2087 15.1021 335.708 583.363
Cached all SQL queries 4.81 207.776 2091 15.1712 304.446 583.684
DB Cache 4.87 205.250 2035 14.1992 302.335 621.092
Out of Box config 4.94 202.624 2026 14.432 114.983 618.434
WP File Cache 4.95 201.890 2009 15.8869 158.597 549.176
4.99 200.211 2004 16.1758 99.728 544.107
DB Cache Reloaded 5.02 199.387 1983 15.0167 187.343 589.196
All SQL Queries Cached 5.03 200.089 1985 14.9233 150.145 586.443
DB Cache Reloaded 5.06 197.636 1968 14.9697 174.857 589.161
Out of Box config 5.08 196.980 1968 15.181 257.533 587.737
Widgetcache 6.667 149.903 1492 15.0264 245.332 602.039
6.72 148.734 1487 15.1887 299.65 598.017
W3 Total Cache 153.45 65.167 60 133.1898 8.916 85.7177
DB Cache off, Page Caching with Memcached 169.46 59.011 57 188.4 9.107 50.142
W3 Total Cache 173.49 57.639 52 108.898 7.668 86.4077
DB Cache off, Minify Cache with Memcached 189.76 52.698 48 203.522 8.122 43.8795
W3 Total Cache 171.34 58.364 50 203.718 8.097 44.1234
DB Cache using Memcached 190.01 52.269 48 206.187 8.186 42.4438
W3 Total Cache 175.29 57.048 48 87.423 7.515 107.973
Out of Box config 191.15 52.314 47 204.387 8.288 43.217
W3 Total Cache 175.29 57.047 51 204.557 8.199 42.9365
Database Cache using APC 191.19 52.304 48 200.612 8.11 44.6691
W3 Total Cache 114.02 87.703 49 114.393 8.206 82.0678
Database Cache Disabled 191.76 52.150 49 203.781 8.095 42.558
W3 Total Cache 175.80 56.884 51 107.842 7.281 87.2761
Database Cache Disabled, Minify Cache using APC 192.01 52.082 50 205.66 8.244 43.1231
W3 Total Cache 104.90 95.325 51 123.041 7.868 74.5887
Database Cache Disabled, Page Caching using APC 197.55 50.620 46 210.445 7.907 41.4102
WP Super Cache 336.88 2.968 16 15.1021 335.708 583.363
Out of Box config, Half On 391.59 2.554 16 15.1712 304.446 583.684
WP Cache 161.63 6.187 12 15.1021 335.708 583.363
482.29 20.735 11 15.1712 304.446 583.684
WP Super Cache 919.11 1.088 3 190.117 1.473 47.9367
Full on, Lockdown mode 965.69 1.036 3 975.979 1.455 9.67185
WP Super Cache 928.45 1.077 3 210.106 1.468 43.8167
Full on 970.45 1.030 3 969.256 1.488 9.78753
W3 Total Cache 1143.94 8.742 2 165.547 0.958 56.7702
Page Cache using Disk Enhanced 1222.16 8.182 3 1290.43 0.961 7.15632
W3 Total Cache 1153.50 8.669 3 165.725 0.916 56.5004
Page Caching – Disk Enhanced, Minify/Database using APC 1211.22 8.256 2 1305.94 0.948 6.97114
Varnish ESI 2304.18 0.434 4 349.351 0.221 28.1079
2243.33 0.44689 4 4312.78 0.152 2.09931
WP Varnish 1683.89 0.594 3 369.543 0.155 26.8906
3028.41 0.330 3 4318.48 0.148 2.15063

Test Script

#!/bin/sh

FETCHES=1000
PARALLEL=10

/usr/sbin/apache2ctl stop
/etc/init.d/mysql restart
apache2ctl start
echo Sleeping
sleep 30
time ( \
echo First Run; \
ab -n $FETCHES -c $PARALLEL http://example.com/; \
echo Second Run; \
ab -n $FETCHES -c $PARALLEL http://example.com/; \
\
echo First Run; \
./http_load -parallel $PARALLEL -fetches $FETCHES wordpresstest; \
echo Second Run; \
./http_load -parallel $PARALLEL -fetches $FETCHES wordpresstest; \
)

URL File for http_load

http://example.com/
http://example.com/2010/03/hello-world/
http://example.com/2008/09/layout-test/
http://example.com/2008/04/simple-gallery-test/
http://example.com/2007/12/category-name-clash/
http://example.com/2007/12/test-with-enclosures/
http://example.com/2007/11/block-quotes/
http://example.com/2007/11/many-categories/
http://example.com/2007/11/many-tags/
http://example.com/2007/11/tags-a-and-c/
http://example.com/2007/11/tags-b-and-c/
http://example.com/2007/11/tags-a-and-b/
http://example.com/2007/11/tag-c/
http://example.com/2007/11/tag-b/
http://example.com/2007/11/tag-a/
http://example.com/2007/09/tags-a-b-c/
http://example.com/2007/09/raw-html-code/
http://example.com/2007/09/simple-markup-test/
http://example.com/2007/09/embedded-video/
http://example.com/2007/09/contributor-post-approved/
http://example.com/2007/09/one-comment/
http://example.com/2007/09/no-comments/
http://example.com/2007/09/many-trackbacks/
http://example.com/2007/09/one-trackback/
http://example.com/2007/09/comment-test/
http://example.com/2007/09/a-post-with-multiple-pages/
http://example.com/2007/09/lorem-ipsum/
http://example.com/2007/09/cat-c/
http://example.com/2007/09/cat-b/
http://example.com/2007/09/cat-a/
http://example.com/2007/09/cats-a-and-c/

Tags: , ,

25 Responses to “WordPress Cache Plugin Benchmarks”

  1. Visit my site and it tries to download a GZ file - Page 2 - WordPress Tavern Forum Says:

    […] up to answer my own question: http://cd34.com/blog/scalability/wor…in-benchmarks/ Basically what I (still) don't like about W3 Total is that there's so MANY options, and I'm not […]

  2. Rootz.de > Wordpress: Benchmarking von Caching-Plugins Says:

    […] http://cd34.com/blog/scalability/wordpress-cache-plugin-benchmarks […]

  3. Maximize WordPress and BuddyPress Performance With W3 Total Cache - WordPress MU and BuddyPress plugins, themes, support, tips and how to's Says:

    […] and yoast.com. If you’d like to compare it to some other caching plugins, have a look at some WordPress cache plugin benchmarks for a thorough analysis of current WordPress scalability […]

  4. WordPressハッカーズ Says:

    WordPress Cache Plugin Benchmarks…

    この記事は以下サイトに掲載されています。 This article has been featured on WordPressハッカーズ…

  5. Information Technology Education | Blog | Maximize WordPress and BuddyPress Performance With W3 Total Cache Says:

    […] and yoast.com. If you’d like to compare it to some other caching plugins, have a look at some WordPress cache plugin benchmarks for a thorough analysis of current WordPress scalability […]

  6. High Performance WordPress — Konstruktors Says:

    […] WordPress Cache Plugin Benchmarks […]

  7. Peter Says:

    Very informative. It would have been nice to see how W3 with memcache performs as well, but you can’t have it all.

  8. cd34 Says:

    @peter, the results are sorted in order by performance. W3/memcache test results are within the results, results 7, 8 and 9.

  9. Peter Says:

    Sorry, I see it now.

  10. blaenk Says:

    Excellent post. It motivated me to try out varnish and it is as blazing fast as you prove it to be in your post. I’m still pretty confused on the whole removing cookies part though, and I was wondering if you would ever be interested in writing a post which goes over your configuration of it for working with WordPress. There are scattered posts around the web but they all seem pretty inconsistent, particularly the part about unsetting cookies.

    Also, another question: If you’re using Varnish, is there a point to also use W3 Total Cache (for opcode caching)? Or should one just use Varnish and leave it at that?

    Thanks again for the very informative post.

  11. cd34 Says:

    Varnish doesn’t cache content if a cookie is sent within the request, so, to cache static assets, you need to remove the cookie in vcl_recv so that varnish will cache it.

    Ideally, any caching solution needs fragment caching, something that Frederick Townes (the author of W3TC) and I talked about at length a while back and is the reason I wrote the WordPress Varnish ESI plugin. Varnish doesn’t compress ESI after assembling it, which results in a convoluted installation with nginx -> varnish -> apache/nginx -> wordpress. The reason you need fragment caching is the sidebar is the same on almost every page, but, generating that sidebar has two of the most expensive queries in WordPress (Categories and Tag Cloud).

    Caching itself is difficult. Cache too much and your site runs quickly, but loses interactivity. Cache too little and your site is sluggish. While anyone can write caching, purging is the difficult part. If you were to run Varnish -> W3TC -> WordPress, using the WP-Varnish plugin, modified posts would have to purge the entry from Varnish and W3TC, and when Varnish gets a request for that new page, W3TC would have to cache it, then hand it to Varnish – resulting in double caching and probably not a lot of net benefit. You really want to run one or the other. If you can run Varnish with wp-varnish, I think that provides the best solution for now. If you aren’t able to run Varnish because you’re on a shared host, then, W3TC or WP-Super-Cache are about the best you can get. And Frederick Townes doesn’t sit still for long – since these benchmarks were run, he’s made the plugin much more intelligent and enhanced it quite a bit.

    I’ll see if I can make a more generic WordPress .vcl and post it.

  12. Wordpress optimieren: Varnish und mehr Says:

    […] Einen sehr guten Überblick über die verschiedenen Caching-Lösungen gibt dieser Benchmark-Artikel.Die Installation ist recht einfach und kann mittels ein paar Befehlen unter Debian ausgeführt […]

  13. matt Says:

    Just came across this post. You mention in regards to W3TC that “If you can’t use APC caching or Memcached for caching Database queries or Minification, turn both off. ” – I’m using eaccelerator for those. I didn’t see that option in the benchmark..I’m wondering if I should assume that eaccelerator isn’t as good as just leaving the db caching and minification off?

    Also, I’m using the development version of W3TC and I notice it has a purge Varnish option – I’m wondering if that would eliminate the need for WP-Varnish.

  14. cd34 Says:

    As this post was written about a year ago, I don’t recall whether eaccelerator was an option. If it was, I’m sure I would have tested it. W3TC has gone through a number of changes since I tested it, so, it is also possible some of the areas where it didn’t benchmark as well as one might have expected, may have been fixed. The other issue I ran into after this test is that both W3TC and WP-Super-Cache installations were very dependent on the machine it was installed on, so, you really need to have a test methodology to see if it is improving things with all of the different options available.

    When I did the test, Frederick Townes was talking about adding a Varnish component to it, but, since he’s doing page caching, if his purge does what I think it should, it would work similar to wp-varnish with the added bonus of minification, css combining, etc. Basically, with the other features, I think W3TC with Varnish would probably trump WP-Varnish overall if you consider the total benefit of those other features.

  15. WordPress Caching Plugins and Observations with W3 Total Cache | Techno Thoughts! Says:

    […] WordPress Cache Plugin Benchmarks […]

  16. Short-term Todo List/Plans Says:

    […] WP-Varnish (Can handle a lot more requests than W3 Total Cache, and I feel like getting my geek on and trying it […]

  17. Ivete Tecedor Creative » Wordpress Caching Benchmarks Says:

    […] just came across an impressive post benchmarking the performance of various WordPress caching plugins and had to share. The post was written over a year ago but it’s so thorough and useful that […]

  18. Плагин для кэширования Wordpress или адронный коллайдер для блога | seozip Says:

    […] – Для выбора рекомендую ознакомится с бэнчмаркоми плагинов кэширования […]

  19. Usando Varnish como proxy [WordPress + Apache] Says:

    […] algunos posts que me fueron útiles, ya que contienen instrucciones sencillas:Blog de Daniel MesserArtículo en CD34.comHow-to ForgeEuperia blogA posteriori: Comparando el rendimientoEs muy importante medir el efecto que […]

  20. todsul Says:

    Hey Chris,

    I’ve just tested the following on a default WordPress 3 install:
    – bare bones Nginx
    – Nginx + W3TC (page cache and minify using disk enhanced)
    – Nginx + Varnish (removed all cookies) + WP Varnish plugin

    All of these have APC opcode cache enabled.

    The bottom 2 configs are producing very similar results, but W3TC is more stable. My ab test for Varnish hangs on the first try and also slows down after I repeat the test many times. I’ve checked memory and there’s plenty free.

    Also, in terms of user experience, there’s a big banner image on the default WordPress theme that rotates randomly. W3TC caches that page and the same banner image shows. Varnish is letting the home page image reload even though I’ve removed all cookies in the VCL. I wonder if it’s to do with browser cache settings (will investigate).

    I saw a comment from Frederick Townes somewhere that he’s lost interest in Varnish. I wonder if that’s based on his own tests.

    As for my results:
    – bare bones Nginx = 57 req/s
    – Nginx + W3TC = 5180 req/s
    – Nginx + Varnish = 5124 req/s

    I wanted to have a nice clean Nginx + Varnish solution (incl Plugin), maybe with a minify plugin, but that’s it. But the Varnish results aren’t especially impressive for the additionally complexity it adds.

    Any thoughts on anything I’m missing regarding Varnish? My configs are really bare bones. No funny business.

  21. todsul Says:

    Just tested – Nginx + WP Super Cache

    Doubled the performance of W3TC and Varnish with 10,602 requests/s.
    The load for non cached pages feels noticeably quicker.

    This is all on the smallest Linode VPS.

  22. cd34 Says:

    As the benchmarks here are over a year and a half old, I would not be surprised that you couldn’t get numbers like that. In brief testing even with Varnish 3.0, we’ve got machines pushing 40k rps and Kristian Lyngstol has benchmarked Varnish at 275k rps (http://kristianlyng.wordpress.com/2010/10/23/275k-req/). There are also considerable performance scheduling changes in 2.6.39 and network improvements in 2.6.37 that would have considerable impact. Also, I don’t believe the shm file was in ram on these tests but, I’m sure I used malloc. libc6 has also received considerable improvement with malloc.

    W3TC is a page cache which uses obstart() and grabs the entire page, writes it to disk or memcached and then outputs it. The header rotator outputs html which is inserted in the page. When you’re logged in, Varnish wouldn’t actually cache the index page, allowing the img src which rotates the header to be generated on each pageload.

    W3TC’s config options are very confusing and without testing and retesting, it is very difficult to find the best configuration. WP Super Cache’s out of the box setup is really much better than W3TC’s. With enough time, you could find the magic combination of settings to be faster than WP Super Cache. There are other benefits to W3TC that can be duplicated with other plugins, but, if you wanted everything in one place, W3TC contains a number of useful plugins all in one place.

    The other thing you need to consider with your benchmark is that Linode uses virtio on the interface and benchmarking on the machine without actually using the network is going to allow you to have considerably higher numbers than if you’re actually moving traffic across the network. You can tell W3TC to not cache for logged in users which more closely replicates how Varnish is handling things. Also, I would suspect that on non-logged in users, the images wouldn’t be rotating until the cache time expired or the page was purged from the cache on both W3TC and Varnish – and, as your latest comment suggests, WP Super Cache.

  23. W3 Total Cache or WP Super Cache? - Admins Goodies Says:

    […] http://cd34.com/blog/scalability/wordpress-cache-plugin-benchmarks/ […]

  24. WordPress 缓存插件性能对比 » 域名宝盒 – DNCase.com Says:

    […] 原文:《WordPress Cache Plugin Benchmarks》 […]

  25. 性能优化插件–坑爹的W3 Total Cache | 风水的英文网赚和黑帽SEO博客 Says:

    […] not W3 Total Cache (评论闪闪发光,很多技术细节) WordPress load test part 2 – amendment WordPress Cache Plugin Benchmarks W3 Total Cache versus WP Super […]

Leave a Reply

You must be logged in to post a comment.

Entries (RSS) and Comments (RSS).
Cluster host: li