Random Musings of an Insane Mind

Many years ago I ran into a situation with a client where the amount of traffic they were receiving was crushing their dynamically created site. Computation is always the enemy of a quick pageload, so, it is very important to do as little computation as possible when delivering a page.

While there are many ways to put together a CMS, high traffic CMS sites usually involve caching or lots of hardware. Some write static files which are much less strenuous, but, you lose some of the dynamic capabilities. Fragment caching becomes a method to make things a bit more dynamic as MasonHQ does with their page and block structure. Django-blocks was surely influenced by this or reinvented this method.

In order to get the highest performance out of a CMS with a page and block method, I had considered writing a filesystem or inode linklist that would allow the webserver to assemble the page by following the inodes on the disk to build the page. Obviously there are some issues here, but, if a block was updated by a process, it would automatically be reassembled. This emulates a write-through cache and would have provisions for dynamic content to be mixed in with the static content on disk. Assembly of the page still takes more compute cycles than a static file but is significantly less than dynamically creating the page from multiple queries.

That design seriously limits the ability to deploy the system widely. While I can control the hosting environment for personal projects, the CMS couldn’t gain wide acceptance. While Varnish is a rather simple piece of software to install, it does limit deploy-ability, but, provides a significant piece of the puzzle due to Edge Side Includes (ESI). If the CMS gets used beyond personal and small deployments, Akamai supports Edge Side Includes as well.

Rather than explain ESI, ESI Explained Simply contains about the best writeup I’ve seen to date to explain how ESI can be used.

The distinction here is using fragment caching controlled by ESI to represent different zones on the page. As a simple example, lets consider our page template contains an article and a block with the top five articles on the site. When a new post is added, we can expire the block that contains the top five articles so that it is requested on the next page fetch. Since the existing article didn’t change, the interior ESI included block doesn’t need to be purged. This allows the page to be constructed on the Edge rather than on the Origin server.

As I have worked with a number of PHP frameworks, none really met my needs so I started using Python frameworks roughly two years ago. For this CMS, I debated using Pylons or Django and ended up choosing Django. Since both can be run behind WSGI compliant servers, we’ve opened ourselves up to a number of potential solutions. Since we are running Varnish in front of our Origin server, we can run Apache2 with mod_wsgi, but, we’re not limited to that configuration. At this point, we have a relatively generic configuration the CMS can run on, but, there are many other places we can adapt the configuration for our preferences.

Some of the potential caveats:
* With Varnish or Akamai as a frontend, we need to pay closer attention to X-Forwarded-For:
* Web logs won’t exist because Varnish is serving and assembling the pages (There is a trick using ESI that could be employed if logging was critical)
* ESI processed pages with Varnish are not compressed. This is on their wishlist.

Features:
* Content can exist in multiple categories or tags
* Flexible URL mapping
* Plugin architecture for Blocks and Elements
* Content will maintain revisions and by default allow comments and threaded comments

Terms:
* Template – the graphical layout of the page with minimal CMS markup
* Element – the graphical template that is used to render a Block
* Block – a module that generates the data rendered by an Element
* Page – a Page determined by a Title, Slug and elements
* Content – The actual data that rendered by a block

Goals:
* Flexible enough to handle something as simple as a personal blog, but, also capable of powering a highly trafficed site.
* Data storage of common elements to handle publishing of content and comments with the ability to store information to allow threaded comments. This would allow the CMS to handle a blog application, a CMS, or, a forum.
* A method to store ancillary data in a model so that upgrades to the existing database model will not affect developed plugins.
* Block system to allow prepackaged css/templating while allowing local replacement without affecting the default package.
* Upgrades through pypy or easy_install.
* Ability to add CDN/ESI without needing to modify templates. The system will run without needing to be behind Varnish, but, its full power won’t be realized without Varnish or Akamai in front of the origin server.
* Seamless integration of affiliate referral tracking and conversion statistics

At this point, the question in my mind was whether or not to start with an existing project and adapt it or start from scratch. At this point, the closest Django CMS I could find was Django-Blocks and I do intend to look it over fairly closely, but, a cursory look showed the authors were taking it in a slightly different direction than I anticipated. I’ll certainly look through the code again, but, the way I’ve envisioned this, I think there are some fundamental points that clash.

As I already have much of the database model written for an older PHP CMS that I wrote, I’m addressing some of the shortcomings I ran across with that design and modifying the models to be a little more generic. While I am sure there are proprietary products that currently utilize ESI, I believe my approach is unique and flexible enough to power everything from a blog to a site or forums or even a classified ads site.

Here’s a programmer that is saying goodbye to ORMs at Hatful of Hollow.

And another site offering a tutorial of sorts dealing with ORMs Why should you use an ORM.

While both have their points, both have missed a fundamental benefit that an ORM hands you.

Most of my development is in Pylons. Django’s ORM and template language can do the same thing. A programmer that has used PHP/Smarty to develop large scale systems will likely resist ORMs. After working with a team to develop 90k+ lines of PHP/Smarty over a six year period, making the shift required a paradigm shift.

Let’s consider the following structure. We have a cp_ticket table and a cp_ticket_detail table. A Ticket can have multiple detail records. The output we wish to have is:

ticket id, ticket header information
         ticket detail line
         ticket detail line #2
ticket id, ticket header information
         ticket detail line
         ticket detail line #2
         ticket detail line #3
ticket id, ticket header information
         ticket detail line
         ticket detail line #2

Our model:

class cp_ticket(DeclarativeBase):
    __tablename__ = 'cp_ticket'

    ticket_id = Column(mysql.MSBigInteger(20, unsigned = True), primary_key=True, autoincrement = True)
    priority = Column(mysql.MSEnum('1','2','3','4','5'), default = '3')

    ticket_detail = relation('cp_ticket_detail', order_by='cp_ticket_detail.ticket_detail_id')

class cp_ticket_detail(DeclarativeBase):
    __tablename__ = 'cp_ticket_detail'

    ticket_id = Column(mysql.MSBigInteger(20, unsigned = True), ForeignKey('cp_ticket.ticket_id'), default = '0')
    ticket_detail_id = Column(mysql.MSBigInteger(20, unsigned = True), primary_key=True, autoincrement = True)
    stamp = Column(mysql.MSTimeStamp, PassiveDefault('CURRENT_TIMESTAMP'))
    detail = Column(mysql.MSLongText, default = '')

Our query to pass to our template:

        tickets = meta.Session.query(cp_ticket).filter(cp_ticket.client_id==1).all()

Compared with the query as you would write it without an ORM:

select * from cp_ticket,cp_ticket_detail where client_id=1 and cp_ticket.ticket_id=cp_ticket_detail.ticket_id;

Both are doing the same fundamental thing, but, the ORM maps the results almost identical to the way we want to display the data. This makes template design easy.

Using Mako, we use the following code to display the results:

<table border="1">
 <tr><th>Ticket ID</th><th>Status</th><th>Detail</th></tr>
%for ticket in tmpl_context.tickets:
  <tr>
    <td><strong>${ticket.ticket_id}</strong></td>
    <td><strong>${ticket.priority}</strong></td>
  </tr>
  %for detail in ticket.ticket_detail:
  <tr>
    <td></td>
    <td>${detail.stamp}</td>
    <td>${detail.detail}</td>
  </tr>
  % endfor
% endfor
</table>

To do the same thing without using an ORM, you need to revert to a control break structure similar to the following:

current_ticket=0
for ticket in tickets:
  if (current_ticket != ticket.ticket_id):
    #new row, print the header
    print "<tr><td>first piece</td></tr>"
    current_ticket = ticket.ticket_id
  # print our detail row
  print "<tr><td></td><td>stamp and detail</td></tr>"

Control Break structures require you to be able to set a variable within your template language. Some template languages don’t allow that. If your template language (in any language) can’t do variable assignments in the template, guess where your html generation logic needs to go?

With an ORM, the template contains your display logic. Your webmaster/design team can modify the template without having to modify html contained within your code. The loops are simple to understand and designers usually have little problem avoiding the lines that start with %.

Sure, you could wrap much of this logic in your template to do the control-break structure, but, as you get more complex data, deciding how to display the data requires a define or some other functionality.

An ORM adds some insulation to the process, but, the result is a much easier page structure when displaying related data. Granted there are some performance hits and SQLAlchemy appears to create some queries that are not optimal, unless there is a tremendous performance hit, I think the benefits of the ORM for developing a web application are tremendous.

Once you move into an environment where you are dealing with multiple developers, having a defined schema with comments is much easier than using reflection to figure out what the meaning of a status field as enum(‘U’,'A’,'P’,'C’,'R’,'S’).

However, as the original poster mentions, you can do raw SQL within SQLAlchemy and do all of your work with reflection as he has done with his ORM^H^H^H, abstraction. If he’s still using SQLAlchemy, he can selectively decide when to use it and when to avoid it.

While doing some coding to move a Turbogears2 application over to Pylons, I ran into an issue with Validation and ToscaWidgets.

AttributeError: ‘function’ object has no attribute ‘replace’

Validation in Pylons listed:

@validate(form=movie_form, error_handler=index)

for the decorator syntax, but, error_handler should be a name, not a function. The correct decorator should be:

@validate(form=movie_form, error_handler=’index’)