Random Musings of an Insane Mind

Many years ago I ran into a situation with a client where the amount of traffic they were receiving was crushing their dynamically created site. Computation is always the enemy of a quick pageload, so, it is very important to do as little computation as possible when delivering a page.

While there are many ways to put together a CMS, high traffic CMS sites usually involve caching or lots of hardware. Some write static files which are much less strenuous, but, you lose some of the dynamic capabilities. Fragment caching becomes a method to make things a bit more dynamic as MasonHQ does with their page and block structure. Django-blocks was surely influenced by this or reinvented this method.

In order to get the highest performance out of a CMS with a page and block method, I had considered writing a filesystem or inode linklist that would allow the webserver to assemble the page by following the inodes on the disk to build the page. Obviously there are some issues here, but, if a block was updated by a process, it would automatically be reassembled. This emulates a write-through cache and would have provisions for dynamic content to be mixed in with the static content on disk. Assembly of the page still takes more compute cycles than a static file but is significantly less than dynamically creating the page from multiple queries.

That design seriously limits the ability to deploy the system widely. While I can control the hosting environment for personal projects, the CMS couldn’t gain wide acceptance. While Varnish is a rather simple piece of software to install, it does limit deploy-ability, but, provides a significant piece of the puzzle due to Edge Side Includes (ESI). If the CMS gets used beyond personal and small deployments, Akamai supports Edge Side Includes as well.

Rather than explain ESI, ESI Explained Simply contains about the best writeup I’ve seen to date to explain how ESI can be used.

The distinction here is using fragment caching controlled by ESI to represent different zones on the page. As a simple example, lets consider our page template contains an article and a block with the top five articles on the site. When a new post is added, we can expire the block that contains the top five articles so that it is requested on the next page fetch. Since the existing article didn’t change, the interior ESI included block doesn’t need to be purged. This allows the page to be constructed on the Edge rather than on the Origin server.

As I have worked with a number of PHP frameworks, none really met my needs so I started using Python frameworks roughly two years ago. For this CMS, I debated using Pylons or Django and ended up choosing Django. Since both can be run behind WSGI compliant servers, we’ve opened ourselves up to a number of potential solutions. Since we are running Varnish in front of our Origin server, we can run Apache2 with mod_wsgi, but, we’re not limited to that configuration. At this point, we have a relatively generic configuration the CMS can run on, but, there are many other places we can adapt the configuration for our preferences.

Some of the potential caveats:
* With Varnish or Akamai as a frontend, we need to pay closer attention to X-Forwarded-For:
* Web logs won’t exist because Varnish is serving and assembling the pages (There is a trick using ESI that could be employed if logging was critical)
* ESI processed pages with Varnish are not compressed. This is on their wishlist.

Features:
* Content can exist in multiple categories or tags
* Flexible URL mapping
* Plugin architecture for Blocks and Elements
* Content will maintain revisions and by default allow comments and threaded comments

Terms:
* Template – the graphical layout of the page with minimal CMS markup
* Element – the graphical template that is used to render a Block
* Block – a module that generates the data rendered by an Element
* Page – a Page determined by a Title, Slug and elements
* Content – The actual data that rendered by a block

Goals:
* Flexible enough to handle something as simple as a personal blog, but, also capable of powering a highly trafficed site.
* Data storage of common elements to handle publishing of content and comments with the ability to store information to allow threaded comments. This would allow the CMS to handle a blog application, a CMS, or, a forum.
* A method to store ancillary data in a model so that upgrades to the existing database model will not affect developed plugins.
* Block system to allow prepackaged css/templating while allowing local replacement without affecting the default package.
* Upgrades through pypy or easy_install.
* Ability to add CDN/ESI without needing to modify templates. The system will run without needing to be behind Varnish, but, its full power won’t be realized without Varnish or Akamai in front of the origin server.
* Seamless integration of affiliate referral tracking and conversion statistics

At this point, the question in my mind was whether or not to start with an existing project and adapt it or start from scratch. At this point, the closest Django CMS I could find was Django-Blocks and I do intend to look it over fairly closely, but, a cursory look showed the authors were taking it in a slightly different direction than I anticipated. I’ll certainly look through the code again, but, the way I’ve envisioned this, I think there are some fundamental points that clash.

As I already have much of the database model written for an older PHP CMS that I wrote, I’m addressing some of the shortcomings I ran across with that design and modifying the models to be a little more generic. While I am sure there are proprietary products that currently utilize ESI, I believe my approach is unique and flexible enough to power everything from a blog to a site or forums or even a classified ads site.

Tags: akamai, cms, django, esi, pylons, Varnish

This entry was posted on Friday, December 18th, 2009 at 6:50 pm and is filed under Framework. You can follow any responses to this entry through the RSS 2.0 feed. You can skip to the end and leave a response. Pinging is currently not allowed.