Random Musings of an Insane Mind

Many years ago I ran into a situation with a client where the amount of traffic they were receiving was crushing their dynamically created site. Computation is always the enemy of a quick pageload, so, it is very important to do as little computation as possible when delivering a page.

While there are many ways to put together a CMS, high traffic CMS sites usually involve caching or lots of hardware. Some write static files which are much less strenuous, but, you lose some of the dynamic capabilities. Fragment caching becomes a method to make things a bit more dynamic as MasonHQ does with their page and block structure. Django-blocks was surely influenced by this or reinvented this method.

In order to get the highest performance out of a CMS with a page and block method, I had considered writing a filesystem or inode linklist that would allow the webserver to assemble the page by following the inodes on the disk to build the page. Obviously there are some issues here, but, if a block was updated by a process, it would automatically be reassembled. This emulates a write-through cache and would have provisions for dynamic content to be mixed in with the static content on disk. Assembly of the page still takes more compute cycles than a static file but is significantly less than dynamically creating the page from multiple queries.

That design seriously limits the ability to deploy the system widely. While I can control the hosting environment for personal projects, the CMS couldn’t gain wide acceptance. While Varnish is a rather simple piece of software to install, it does limit deploy-ability, but, provides a significant piece of the puzzle due to Edge Side Includes (ESI). If the CMS gets used beyond personal and small deployments, Akamai supports Edge Side Includes as well.

Rather than explain ESI, ESI Explained Simply contains about the best writeup I’ve seen to date to explain how ESI can be used.

The distinction here is using fragment caching controlled by ESI to represent different zones on the page. As a simple example, lets consider our page template contains an article and a block with the top five articles on the site. When a new post is added, we can expire the block that contains the top five articles so that it is requested on the next page fetch. Since the existing article didn’t change, the interior ESI included block doesn’t need to be purged. This allows the page to be constructed on the Edge rather than on the Origin server.

As I have worked with a number of PHP frameworks, none really met my needs so I started using Python frameworks roughly two years ago. For this CMS, I debated using Pylons or Django and ended up choosing Django. Since both can be run behind WSGI compliant servers, we’ve opened ourselves up to a number of potential solutions. Since we are running Varnish in front of our Origin server, we can run Apache2 with mod_wsgi, but, we’re not limited to that configuration. At this point, we have a relatively generic configuration the CMS can run on, but, there are many other places we can adapt the configuration for our preferences.

Some of the potential caveats:
* With Varnish or Akamai as a frontend, we need to pay closer attention to X-Forwarded-For:
* Web logs won’t exist because Varnish is serving and assembling the pages (There is a trick using ESI that could be employed if logging was critical)
* ESI processed pages with Varnish are not compressed. This is on their wishlist.

Features:
* Content can exist in multiple categories or tags
* Flexible URL mapping
* Plugin architecture for Blocks and Elements
* Content will maintain revisions and by default allow comments and threaded comments

Terms:
* Template – the graphical layout of the page with minimal CMS markup
* Element – the graphical template that is used to render a Block
* Block – a module that generates the data rendered by an Element
* Page – a Page determined by a Title, Slug and elements
* Content – The actual data that rendered by a block

Goals:
* Flexible enough to handle something as simple as a personal blog, but, also capable of powering a highly trafficed site.
* Data storage of common elements to handle publishing of content and comments with the ability to store information to allow threaded comments. This would allow the CMS to handle a blog application, a CMS, or, a forum.
* A method to store ancillary data in a model so that upgrades to the existing database model will not affect developed plugins.
* Block system to allow prepackaged css/templating while allowing local replacement without affecting the default package.
* Upgrades through pypy or easy_install.
* Ability to add CDN/ESI without needing to modify templates. The system will run without needing to be behind Varnish, but, its full power won’t be realized without Varnish or Akamai in front of the origin server.
* Seamless integration of affiliate referral tracking and conversion statistics

At this point, the question in my mind was whether or not to start with an existing project and adapt it or start from scratch. At this point, the closest Django CMS I could find was Django-Blocks and I do intend to look it over fairly closely, but, a cursory look showed the authors were taking it in a slightly different direction than I anticipated. I’ll certainly look through the code again, but, the way I’ve envisioned this, I think there are some fundamental points that clash.

As I already have much of the database model written for an older PHP CMS that I wrote, I’m addressing some of the shortcomings I ran across with that design and modifying the models to be a little more generic. While I am sure there are proprietary products that currently utilize ESI, I believe my approach is unique and flexible enough to power everything from a blog to a site or forums or even a classified ads site.

A week or two ago, a story broke regarding a security upgrade in Windows. In the race to scoop the story first, facts were not checked, the validity of the story was based on a blog post at a security company.

Ed Bott @ Ziff Davis covered it in What the “Black screen of death” story says about tech journalism.

Even TechCrunch falls into this with a spoofed Eric Schmidt joins Twitter. Post first, ask later. Rather than correct the incorrect article, let it run for the adviews.

Since the introduction of the Internet, journalistic accuracy has dropped substantially. While spell-check should eliminate most of the errors, typographic errors occur frequently. The number of journalists that get your and you’re confused or their and there is staggering. Tribune Media, CNN/Turner, ABC, Fox and MSNBC are not immune. Associated Press, Reuters and United Press International remain news leaders with accurate, verified and grammatically correct articles. With the downturn in paper journalism, competent writers have been replaced with less expensive writers that are more interested in the number of bylines they can generate than the quality of their work.

To test a theory, a mock-up of a Facebook Beta application, a ruse posted on a few news sites with corroborating evidence and a ‘hot tip’ to two media outlets resulted in 31 different locations picking up on the post, 2700 or so retweets and precisely one site validating the facts.

The first site it was posted to, Hacker News, suspected it was fake almost immediately. However, they missed the significance of the names chosen, the times that the other comments were posted and the sequence of names. Hackers indeed. A spoof post about a hamster falling into the LHC stayed within the top 210 posts for almost four days before enough ‘news’ displaced it.

In the end, it took a security person from Facebook to post and the thread was subsequently killed. Did Facebook violate someone’s privacy to get to the bottom of this? There sure wasn’t much red tape for the Facebook engineer to peer into someone’s profile to get to the bottom of it.

TheNextWeb suspected something was amiss and updated their post throughout the day clearly indicating the updates. Martin Bryant contacted me via email to ask quite directly whether the information was true. This is good journalism.

I suppose most of the sites that ran the story are just pulling RSS feeds from somewhere with no editorial oversight. A trusted syndicated source could distribute a hoax fairly widely and the remnants would be available on the web and search engines for years.

Do sites knowingly run with incorrect headlines in search of ad dollars associated with a hot story — hoax or not? Three sites that picked up the story clearly wanted the the hysteria and hype to drive adviews.

In the end, the glut of news available at our fingertips means that the overall quality of news has diminished. Is there a solution? With automation moving at breakneck speed, it is a problem we’re going to have to deal with for quite some time. Even Google’s news site presents stories without any editorial control and would be a difficult, but not impossible vector to exploit.

Peer reviewed news isn’t the answer as so many sites have proven and editorially controlled sites contain bias no matter how independent they claim to be.

Want to design the killer app of 2010? Fix news distribution.

I’ve always been an early adopter of technology, social media and new websites that had a technological edge. I read quite a few of the tech news websites and love to get in on early beta and beta offerings from companies. One of my recent favorite betas that I was invited to was lite.facebook.com. On the surface, it seemed to lack a certain finesse, but, the biggest feature it had was that it was extremely quick, lacked the application spam and let me see 99% of what I was interested in.

I’ve loved Google Voice and was a fairly early adopter. I had tried Grand Central, but, it didn’t replace enough functionality with what I had currently set up with the local phone company. Google Wave and their Sandbox is another product that I find very intriguing. I have worked with Wave Federations and I think once someone develops a killer app for Wave, it’ll gain wide acceptance.

But, this isn’t about Google, this is about Facebook.

I was an early adopter of FB Connect. I’ve written a number of applications that I’ve not released to experiment with their API and have been generally impressed by their openness. Some of the information an application is able to access is a privacy nightmare. People complain day in and day out about Google and Privacy – perhaps because Google has to collect all of its market intelligence based on your surfing habits, and then Facebook finds a way to have you spend hours customizing your profile – giving Facebook precisely the information that makes their advertising system 10x more intrusive than Google could ever be. Back to the point.

In August I received an email from Facebook asking if I would participate in another beta project. I was warned that this one would entail a purchase from their store, but, in exchange, I would receive credit towards advertising. It makes perfect sense to test the payment system ahead of time on a major release – something many new electronic stores fail to do. I clicked the link saying I would be a part of their beta and waited.

And waited.

Last night, a very cryptic email arrived with a link to follow to read about this exciting new product Facebook had to offer. As I read the page, I was already pulling out my wallet to get my credit card because the service seemed perfect for me. Having to maintain a LinkedIn profile and a Facebook Profile has always been an exercise in duplication. Facebook doesn’t ask enough questions to really be useful in business and I suspect if they put their heads together, they could develop a new angle.

It appears they listened.

The page was very basic, it talked about the benefits of a ‘Facebook Pro’ account, pricing hadn’t been established but they had set a test price of $29.95 for a 6 month recurring membership.

Some of the benefits listed included:

* Ability to store Work History
* Ability to write Recommendations on profiles
* Tighter control over Profile Security
* Additional Contact Method fields
* Certification badges
* Digital Business cards

Once you get in, there is a small NDA that prevents screenshots of the interface, but, it is obvious that there are hundreds of people in the beta. Even as I have set up some business interests, it is listing profiles in a ‘Business Network’ that are staggeringly accurate. A refreshing change from the People You May Know lottery.

So far, the new options are quite intriguing and if the quality of the business contacts I’ve made in the beta are indicative of the trend, I think Facebook has a real winner here.

I found it interesting that the beta was released which allows tighter control over privacy the day after they release new privacy options that the masses are hailing as anti-privacy. Perhaps this is why Facebook chose this week to release the beta.

With a few words of warning, we upgraded one of our clusters from 2.2 to 3.0.4. While this is normally a seamless project, it needed to be coordinated with both storage nodes in the cluster since the changes from 2.2 to 3.0 in openais were incompatible. Some minor changes to the cluster config file were needed which results in a cleaner file, and, a new dependency for rgmanager was added for the upgrade to 3.0.

This meant some downtime while openais was upgraded. Since we run behind a pair of load balancers, we were able to shut down the first filesystem, disconnect it from cman, upgrade one side, shut off the services on the other, bring this side up, bring up services, then upgrade the second node.

While this should have worked, cman on the primary node had no problem, but the secondary node refused to start dlm_controld.

Dec 10 12:29:20 dlm_controld dlm_controld 3.0.4 started
Dec 10 12:29:30 dlm_controld cannot find device /dev/misc/lock_dlm_plock with minor 58

For some odd reason, lock_dlm_plock was created in /dev rather than /dev/misc after the udev upgrade. Moving it into place allowed cman to start on the second node, and, allowed the cluster to run in non-degraded mode.

Why lock_dlm_plock was in the wrong place on one node and in the correct place on the other node, I’m not sure. I think prior to rgmanager being installed, the init script for cman didn’t stop when dlm couldn’t be loaded, and since the /dev/misc folder hadn’t been created, it created the lock file in /dev. Subsequent restarts of the machine have resulted in it coming up without an issue, so, it appears to be an issue somewhere in one of the startup scripts.

After installing Varnish 2.0.5 on a machine, ESI Includes didn’t work. When using varnishlog, the first error that occurred when debugging was:

No ESI processing, first char not ‘< '

   12 SessionClose – timeout
   12 StatSess     – 124.177.181.149 50662 4 0 0 0 0 0 0 0
   12 SessionOpen  c 68.212.183.136 60087 66.244.147.44:80
   12 ReqStart     c 68.212.183.136 60087 409391565
   12 RxRequest    c GET
   12 RxURL        c /esi.html
   12 RxProtocol   c HTTP/1.1
   12 RxHeader     c Host: cd34.colocdn.com
   12 RxHeader     c User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.6; en-US; rv:1.9.2b4) Gecko/20091124 Firefox/3.6b4
   12 RxHeader     c Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
   12 RxHeader     c Accept-Language: en-us,en;q=0.5
   12 RxHeader     c Accept-Encoding: gzip,deflate
   12 RxHeader     c Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
   12 RxHeader     c Keep-Alive: 115
   12 RxHeader     c Connection: keep-alive
   12 RxHeader     c X-lori-time-1: 1259718658980
   12 RxHeader     c Cache-Control: max-age=0
   12 VCL_call     c recv
   12 VCL_return   c lookup
   12 VCL_call     c hash
   12 VCL_return   c hash
   12 VCL_call     c miss
   12 VCL_return   c fetch
   12 Backend      c 14 cd34_com cd34_com
   12 ObjProtocol  c HTTP/1.1
   12 ObjStatus    c 200
   12 ObjResponse  c OK
   12 ObjHeader    c Date: Wed, 02 Dec 2009 01:50:59 GMT
   12 ObjHeader    c Server: Apache
   12 ObjHeader    c Vary: Accept-Encoding
   12 ObjHeader    c Content-Encoding: gzip
   12 ObjHeader    c Content-Type: text/html
   12 TTL          c 409391565 RFC 120 1259718659 0 0 0 0
   12 VCL_call     c fetch
   12 TTL          c 409391565 VCL 43200 1259718659
   12 ESI_xmlerror c No ESI processing, first char not ‘< '
   12 TTL          c 409391565 VCL 0 1259718659
   12 VCL_info     c XID 409391565: obj.prefetch (-30) less than ttl (-1), ignored.
   12 VCL_return   c deliver
   12 Length       c 68
   12 VCL_call     c deliver
   12 VCL_return   c deliver
   12 TxProtocol   c HTTP/1.1
   12 TxStatus     c 200
   12 TxResponse   c OK
   12 TxHeader     c Server: Apache
   12 TxHeader     c Vary: Accept-Encoding
   12 TxHeader     c Content-Encoding: gzip
   12 TxHeader     c Content-Type: text/html
   12 TxHeader     c Content-Length: 68
   12 TxHeader     c Date: Wed, 02 Dec 2009 01:50:59 GMT
   12 TxHeader     c X-Varnish: 409391565
   12 TxHeader     c Age: 0
   12 TxHeader     c Via: 1.1 varnish
   12 TxHeader     c Connection: keep-alive
   12 TxHeader     c X-Cache: MISS
   12 ReqEnd       c 409391565 1259718659.088263512 1259718659.127703667 0.000059366 0.039401770 0.000038385
   12 Debug        c "herding"

ESI received significant performance enhancements in 2.0.4 and 2.0.5 so, it seemed something was incompatible. Downgrading to 2.0.3 and using the VCL from another machine still resulted in ESI not working.

In this case, mod_deflate was running on the backend which was causing the issue. However, in reading the source code, it appears that message could also occur if your ESI include wasn’t handing back properly formed XML/HTML content. If your include doesn’t contain valid content and is only returning a small snippet, you might consider passing:

-p esi_syntax=0x1

on the command line that starts Varnish.

The changes in Varnish address the issue of ESI being enabled on binary content. Since the first character isn’t an < in almost all binary files (jpg, mpg, gif) and isn't the start of most .css/.js files, varnish doesn't need to spend extra time checking those files for includes. While you can and should selectively enable esi processing, this is just an added safeguard and a performance boost to compensate for vcl that might have an esi directive on static/binary content. Since Varnish 2.0.3 now worked properly with the new machine, we upgraded to Varnish 2.0.5 which introduced a very odd issue:

[Tue Dec 01 20:58:11 2009] [error] [client 66.244.147.40] File does not exist: /gfs/www/cd/cd34.com/index.htmlt
[Tue Dec 01 20:58:13 2009] [error] [client 66.244.147.40] File does not exist: /gfs/www/cd/cd34.com/index.html7
[Tue Dec 01 20:58:24 2009] [error] [client 66.244.147.40] File does not exist: /gfs/www/cd/cd34.com/index.html\xfa
[Tue Dec 01 20:59:01 2009] [error] [client 66.244.147.40] File does not exist: /gfs/www/cd/cd34.com/index.html\xb5
[Tue Dec 01 20:59:06 2009] [error] [client 66.244.147.40] File does not exist: /gfs/www/cd/cd34.com/index.html\xe7
[Tue Dec 01 20:59:07 2009] [error] [client 66.244.147.40] File does not exist: /gfs/www/cd/cd34.com/index.html\xd4
[Tue Dec 01 20:59:08 2009] [error] [client 66.244.147.40] File does not exist: /gfs/www/cd/cd34.com/index.html\x1c

This generated 404s on the piece of the page that contained the ESI include. Downgrading to 2.0.4 fixed the issue and the issue appears to already be fixed in Trunk. Varnish Ticket #585

Varnish 2.0.4 and mod_deflate disabled addressed the two issues that prevented ESI from working correctly on this new installation.