Varnish saves the day…. maybe
We had a client that had a machine where apache was being overrun… or so we thought. Everything pointed at this one set of domains owned by a client and in particular two sites with 100+ elements on the page. Images, css, javascript and iframes composed their main page. Apache was handling things reasonably well, but, it was immediately obvious that it could be better.
The conversion to Varnish was quite simple to do even on a live server. Slight modifications to the Apache config file to listen to port 81 on the set of domains in question, and a quick restart. Varnish was configured to listen to port 80 on that particular IP and some minor modifications were made to the startup.vcl file to modify things slightly:
sub vcl_fetch {
if (req.url ~ “\.(png|gif|jpg|swf|css|js)$”) {
set obj.ttl = 3600s;
}
}
A one hour cache should be granular enough to do a bit more good on these sites, overriding the default of two minutes. After an hour, it was evident that the sites did peform much more quickly, but, we still had a load issue. Some modifications of the apache config alleviated some of the other load problems after we dug further into things.
After 5 hours, we ended up with the following statistics from varnish:
0+05:18:24 xxxxxx Hitrate ratio: 10 100 1000 Hitrate avg: 0.9368 0.9231 0.9156 62576 1.00 3.28 Client connections accepted 466684 57.88 24.43 Client requests received 411765 48.90 21.55 Cache hits 148 0.00 0.01 Cache hits for pass 32018 7.98 1.68 Cache misses 54761 8.98 2.87 Backend connections success 0 0.00 0.00 Backend connections failures 45411 7.98 2.38 Backend connections reuses 48598 7.98 2.54 Backend connections recycles
Varnish is doing a great job. The site does load considerably faster, but, it didn’t solve the entire problem. It did reduce the number of apache processes on that machine from 450 to 170 or so, freed up some ram for cache, and did make the server more responsive, but, it probably only contributed to 50% of the issue. The rest of it was cleaning up some poorly written php code, modifying a few mysql tables and adding some indexes to make things work more quickly.
After we fixed the code problems, we debated removing Varnish from their configuration. Varnish did buy us time to fix the problem and does result in a better experience for surfers on the sites, but, after the backend changes, it is hard to tell whether it makes enough impact to keep a non-standard configuration running. Since it is not caching the main page of the site and is only serving the static elements (the site sets an expire time on each generated page), the only real benefit is that we are removing the need for apache to serve the static elements.
While testing another application, we were able to override hardcoded expire times and forcing a minimally cached page. Even if we cached a generated page for two minutes, it could be the difference between a responsive server and a machine struggling to keep up. Since WordPress, Joomla, Drupal and others set expire times using dates that have passed, they ensure that the site html being output is not cached. Varnish allows us to ignore that, and to set our own cache time which could save a site hit with a lot of traffic.
sub vcl_fetch {
if (obj.ttl < 120s) {
set obj.ttl = 120s;
}
}
would give us a minimum two minute cache which would cut the requests to a dynamically generated page considerably.
It is a juggling act. Where do you make the tradeoff and what do you accelerate? Too many times the solution to a website’s performance problem is to throw more hardware at it. At some point you have to split the load on multiple servers, adding new bottlenecks. An application designed to run on a single machine becomes difficult to split to two or more machines, so, many times we do what we can to keep things running on a single machine.