Gracefully Degrading Site with Varnish and High Load
Saturday, July 16th, 2011If you run Varnish, you might want to gracefully degrade your site when traffic comes unexpectedly. There are other solutions listed on the net which maintain a Three State Throttle, but, it seemed like this could be done easily within Varnish without needing too many external dependencies.
The first challenge was to figure out how we wanted to handle state. Our backend director is set up with a ‘level1’ backend which doesn’t do any health checks. We need at least one node to never fail the health check since the ‘level2’ and ‘level3’ backends will go offline to signify to Varnish that we need to take action. While this scenario considers the failure mode cascades, i.e. level2 fails, then if things continue to increase load, level3 fails, there is nothing preventing you from having separate failure modes and different VCL for those conditions.
You could have VCL that replaced the front page of your site with ‘top news’ during an event which links to your secondary page. You can rewrite your VCL to handle almost any condition and you don’t need to worry about doing a VCL load to update the configuration.
While maintaining three configurations is easier, there are a few extra points of failure added in that system. If the load on the machine gets too high and the cron job or daemon that is supposed to update the VCL doesn’t run quickly enough or has issues with network congestion talking with Varnish, your site could run in a degraded mode much longer than needed. With this solution, in the event that there is too much network congestion or too much load for the backend to respond, Varnish automatically considers that a level3 failure and enacts those rules – without the backend needing to acknowledge the problem.
The basics
First, we set up the script that Varnish will probe. The script doesn’t need to be php and only needs to respond with an error 404 to signify to Varnish that probe request has failed.
<?php
$level = $_SERVER['QUERY_STRING'];
$load = file_get_contents('/proc/loadavg') * 1;
if ( ($level == 2) and ($load > 10) ) {
header("HTTP/1.0 404 Get the bilge pumps working!");
}
if ( ($level == 3) and ($load > 20) ) {
header("HTTP/1.0 404 All hands abandon ship");
}
?>
Second, we need to have our backend pool configured to call our probe script:
backend level1 {
.host = "66.55.44.216";
.port = "80";
}
backend level2 {
.host = "66.55.44.216";
.port = "80";
.probe = {
.url = "/load.php?2";
.timeout = 0.3 s;
.window = 3;
.threshold = 3;
.initial = 3;
}
}
backend level3 {
.host = "66.55.44.216";
.port = "80";
.probe = {
.url = "/load.php?3";
.timeout = 0.3 s;
.window = 3;
.threshold = 3;
.initial = 3;
}
}
director crisis random {
{
# base that should always respond so we don't get an Error 503
.backend = level1;
.weight = 1;
}
{
.backend = level2;
.weight = 1;
}
{
.backend = level3;
.weight = 1;
}
}
Since both of our probes go to the same backend, it doesn’t matter which director we use or what weight we assign. We just need to have one backend configured that won’t fail the probe along with our level2 and level3 probes. In this example, when the load on the server is greater than 10, it triggers a level2 failure. If the load is greater than 20, it triggers a level3 failure.
In this case, when the backend probe request fails, we just rewrite the URL. Any VCL can be added, but, you will have some duplication. Since the VCL is compiled into the Varnish server, it should have negligible performance impact.
sub vcl_recv {
set req.backend = level2;
if (!req.backend.healthy) {
unset req.http.cookie;
set req.url = "/level2.php";
}
set req.backend = level3;
if (!req.backend.healthy) {
unset req.http.cookie;
set req.url = "/level3.php";
}
set req.backend = crisis;
}
In this case, when we have a level2 failure, we change any URL requested to serve the file /level2.php. In vcl_fetch, we make a few changes to the object ttl so that we prevent the backend from getting hit too hard. We also change the server name so that we can look at the headers to see what level our server is currently running. In Firefox, there is an extension called Header Spy which will allow you to keep track of a header. Often times I’ll track X-Cache which I set to HIT or MISS to make sure Varnish is caching, but, you could also track Server and be aware of whether things are running properly.
sub vcl_fetch {
set beresp.ttl = 0s;
set req.backend = level2;
if (!req.backend.healthy) {
set beresp.ttl = 5m;
unset beresp.http.set-cookie;
set beresp.http.Server = "(Level 2 - Warning)";
}
set req.backend = level3;
if (!req.backend.healthy) {
set beresp.ttl = 30m;
unset beresp.http.set-cookie;
set beresp.http.Server = "(Level 3 - Critical)";
}
At this point, we’ve got a system that degrades gracefully, even if the backend cannot respond or update Varnish’s VCL and it self-heals based on the load checks. Ideally you’ll also want to put Grace timers and possibly run Saint mode to handle significant failures, but, this should help your system protect itself from meltdown.
Complete VCL
backend level1 {
.host = "66.55.44.216";
.port = "80";
}
backend level2 {
.host = "66.55.44.216";
.port = "80";
.probe = {
.url = "/load.php?2";
.timeout = 0.3 s;
.window = 3;
.threshold = 3;
.initial = 3;
}
}
backend level3 {
.host = "66.55.44.216";
.port = "80";
.probe = {
.url = "/load.php?3";
.timeout = 0.3 s;
.window = 3;
.threshold = 3;
.initial = 3;
}
}
director crisis random {
{
# base that should always respond so we don't get an Error 503
.backend = level1;
.weight = 1;
}
{
.backend = level2;
.weight = 1;
}
{
.backend = level3;
.weight = 1;
}
}
sub vcl_recv {
set req.backend = level2;
if (!req.backend.healthy) {
unset req.http.cookie;
set req.url = "/level2.php";
}
set req.backend = level3;
if (!req.backend.healthy) {
unset req.http.cookie;
set req.url = "/level3.php";
}
set req.backend = crisis;
}
sub vcl_fetch {
set beresp.ttl = 0s;
set req.backend = level2;
if (!req.backend.healthy) {
set beresp.ttl = 5m;
unset beresp.http.set-cookie;
set beresp.http.Server = "(Level 2 - Warning)";
}
set req.backend = level3;
if (!req.backend.healthy) {
set beresp.ttl = 30m;
unset beresp.http.set-cookie;
set beresp.http.Server = "(Level 3 - Critical)";
}
if (req.url ~ "\.(gif|jpe?g|png|swf|css|js|flv|mp3|mp4|pdf|ico)(\?.*|)$") {
set beresp.ttl = 365d;
}
}
