Random Musings of an Insane Mind

Cue Inspiration

I had an idea the other day for a simple Facebook application. With Pylons, I figured it would take no more than an hour or two to get the mechanics of the application working at which point a designer could come in to handle the rest. What followed was quite a struggle.

I’ve written Facebook applications using the old PHP SDK. While this method is rather well documented, moving to Python should have been easy. PyFacebook, a project hosted at GitHub, is woefully out of date. Even after applying three of the suggested patches, and modifying one of the imports in the library, I was left with numerous issues regarding the access_token, odd redirects and a few other minor issues. Add to this the fact that Facebook really suggests that applications use the Graph API and IFrames rather than the FBML canvas and we’re setting ourselves up for a problem down the road. Facebook has maintained that they will always support FBML and the REST API, but, new features won’t be accessible to the older API.

With that in mind, I looked at Python SDK, the officially recommended library for Python and Facebook. While Pylons is supported fairly well, going through the documentation on Facebook’s site resulted in looking through the PHP-SDK, the supplied oauth access in the Python-SDK and a bit of trial and error along the way. OAuth with the Graph API is a bit more complex to understand, but, if your application is using AJAX or JSON, avoiding the FBML proxy is much quicker. Flash was used quite a bit with FBML applications so that applications could communicate directly with a game server which made things much quicker. HTML apps using FBML often exhibited pageload performance problems. With the IFrame method, your application still runs within Facebook’s canvas, but, the surfer is communicating directly with your application.

What happened?

First, I tried to replicate what I had done in PHP using PyFacebook and Pylons. While there are hooks for WSGI (which includes Paster/Pylons implemented servers), there were a number of issues. I briefly tried Django with PyFacebook and met different issues. Once you stray from PHP, you’re in uncharted territory. A statistic I read somewhere claimed that only a few hundred apps were developed in something other than PHP with Java/JSP being the most common alternate. Django, the Google App Engine and web.py appear to be the favorites among Python frameworks. While I know there are a handful of applications running Pylons, and at least one running TurboGears, I don’t believe there are many using the Graph API.

At this point, the Graph API and OAuth seemed to be the sane choice. An IFrame canvas, using the Javascript SDK to complement Python SDK appeared to be the answer.

The first stumbling block when following the OAuth guide on Facebook is the frame in a frame shaded authentication box. Clicking on the grey box opens the inner frame to the full page where you can authorize the application, but, that is a rather ugly situation. The following Javascript fixes that which isn’t great solution, but does work.

<script language="javascript">
top.location.href='https://graph.facebook.com/oauth/authorize?client_id=${config['facebook.appid']}&redirect_uri=${config['facebook.callbackurl']}&display=page&scope=publish_stream';
</script>

Error validating verification code

After working with a few other issues, another issue with the auth_token resulted in the following error (after loading the url that was being fetched):

{
   "error": {
      "type": "OAuthException",
      "message": "Error validating verification code."
   }
}

Adding &type=client_cred to your access_token url fixes that situation.

Here’s the guide

We’re going to put our project in the facebook directory and use Pylons 1.0:

git clone http://github.com/facebook/python-sdk.git
wget http://www.pylonshq.com/download/1.0/go-pylons.py
python go-pylons.py facebook
cd facebook
source bin/activate
paster create -t pylons fbapp
cd fbapp
vi development.ini
rm fbapp/public/index.html
cp ../python-sdk/src/facebook.py fbapp/fbapp/lib

We need to make a few changes to our development.ini in the [app:main] section:

facebook.callbackurl = http://apps.facebook.com/ourfbapp/
facebook.apikey = 6b5aca8bd71c1234590e697f79xxxxxx
facebook.secret = df5d928b87c0df312c8be101e5xxxxxx
facebook.appid = 124322020xxxxxx

modify config/routing.py:

    map.connect('/{action}', controller='root')
    map.connect('/', controller='root', action='index')

templates/oauth_redirect.mako:

<script language="javascript">
top.location.href='https://graph.facebook.com/oauth/authorize?client_id=${config['facebook.appid']}&redirect_uri=${config['facebook.callbackurl']}&display=page&scope=publish_stream';
</script>
<noscript>
<a href="https://graph.facebook.com/oauth/authorize?client_id=${config['facebook.appid']}&redirect_uri=${config['facebook.callbackurl']}&display=page&scope=publish_stream" target="_top">Click here to authorize this application</a>
</noscript>

templates/index.mako:

${tmpl_context.user}

controllers/root.py:

# using python 2.5
import simplejson
import cgi
import urllib

from pylons import request, response, session, tmpl_context, config
from pylons.controllers.util import abort, redirect

from fbapp.lib.base import BaseController, render
import fbapp.lib.facebook as facebook

class RootController(BaseController):

    def __before__(self):
        tmpl_context.user = None
        if request.params.has_key('session'):
            access_token = simplejson.loads(request.params['session'])['access_token']
            graph = facebook.GraphAPI(access_token)
            tmpl_context.user = graph.get_object("me")

    def index(self):
        if not tmpl_context.user:
            return render('/oauth_redirect.mako')
        return render('/index.mako')

In Facebook, you want to make the following changes:

Canvas Callback URL: http://yourdomain.com/
Connect URL: http://yourdomain.com/
Canvas URL: http://apps.facebook.com/yourappname/
FBML/Iframe: iframe
Application Type: website

Under the Migrations Tab, Verify that New Data Permissions and New SDKs are both set to enabled. When you write your application, you can refer to the extended permissions which are set in the &scope= section of oauth_redirect.mako.

What’s next?

Once you’ve retrieved the access_token and the user_id, you probably want to save that into your local database so that you don’t need to fetch the data from the Graph API on every pageload. While the Graph method is indeed faster than FBML, Facebook has lifted some of the restrictions regarding the data you can keep which allows for faster pageloads. With the IFrame method, pages using AJAX/JSON are indeed much quicker.

While I originally estimated this project to take ‘a few hours’, working through all of the possible scenarios with the Python Facebook SDK ended up taking quite a bit more time than expected. The Graph API is very well thought out and is much faster than the REST API and appears to be almost as fast as FQL.

Good Luck!

Many years ago I ran into a situation with a client where the amount of traffic they were receiving was crushing their dynamically created site. Computation is always the enemy of a quick pageload, so, it is very important to do as little computation as possible when delivering a page.

While there are many ways to put together a CMS, high traffic CMS sites usually involve caching or lots of hardware. Some write static files which are much less strenuous, but, you lose some of the dynamic capabilities. Fragment caching becomes a method to make things a bit more dynamic as MasonHQ does with their page and block structure. Django-blocks was surely influenced by this or reinvented this method.

In order to get the highest performance out of a CMS with a page and block method, I had considered writing a filesystem or inode linklist that would allow the webserver to assemble the page by following the inodes on the disk to build the page. Obviously there are some issues here, but, if a block was updated by a process, it would automatically be reassembled. This emulates a write-through cache and would have provisions for dynamic content to be mixed in with the static content on disk. Assembly of the page still takes more compute cycles than a static file but is significantly less than dynamically creating the page from multiple queries.

That design seriously limits the ability to deploy the system widely. While I can control the hosting environment for personal projects, the CMS couldn’t gain wide acceptance. While Varnish is a rather simple piece of software to install, it does limit deploy-ability, but, provides a significant piece of the puzzle due to Edge Side Includes (ESI). If the CMS gets used beyond personal and small deployments, Akamai supports Edge Side Includes as well.

Rather than explain ESI, ESI Explained Simply contains about the best writeup I’ve seen to date to explain how ESI can be used.

The distinction here is using fragment caching controlled by ESI to represent different zones on the page. As a simple example, lets consider our page template contains an article and a block with the top five articles on the site. When a new post is added, we can expire the block that contains the top five articles so that it is requested on the next page fetch. Since the existing article didn’t change, the interior ESI included block doesn’t need to be purged. This allows the page to be constructed on the Edge rather than on the Origin server.

As I have worked with a number of PHP frameworks, none really met my needs so I started using Python frameworks roughly two years ago. For this CMS, I debated using Pylons or Django and ended up choosing Django. Since both can be run behind WSGI compliant servers, we’ve opened ourselves up to a number of potential solutions. Since we are running Varnish in front of our Origin server, we can run Apache2 with mod_wsgi, but, we’re not limited to that configuration. At this point, we have a relatively generic configuration the CMS can run on, but, there are many other places we can adapt the configuration for our preferences.

Some of the potential caveats:
* With Varnish or Akamai as a frontend, we need to pay closer attention to X-Forwarded-For:
* Web logs won’t exist because Varnish is serving and assembling the pages (There is a trick using ESI that could be employed if logging was critical)
* ESI processed pages with Varnish are not compressed. This is on their wishlist.

Features:
* Content can exist in multiple categories or tags
* Flexible URL mapping
* Plugin architecture for Blocks and Elements
* Content will maintain revisions and by default allow comments and threaded comments

Terms:
* Template – the graphical layout of the page with minimal CMS markup
* Element – the graphical template that is used to render a Block
* Block – a module that generates the data rendered by an Element
* Page – a Page determined by a Title, Slug and elements
* Content – The actual data that rendered by a block

Goals:
* Flexible enough to handle something as simple as a personal blog, but, also capable of powering a highly trafficed site.
* Data storage of common elements to handle publishing of content and comments with the ability to store information to allow threaded comments. This would allow the CMS to handle a blog application, a CMS, or, a forum.
* A method to store ancillary data in a model so that upgrades to the existing database model will not affect developed plugins.
* Block system to allow prepackaged css/templating while allowing local replacement without affecting the default package.
* Upgrades through pypy or easy_install.
* Ability to add CDN/ESI without needing to modify templates. The system will run without needing to be behind Varnish, but, its full power won’t be realized without Varnish or Akamai in front of the origin server.
* Seamless integration of affiliate referral tracking and conversion statistics

At this point, the question in my mind was whether or not to start with an existing project and adapt it or start from scratch. At this point, the closest Django CMS I could find was Django-Blocks and I do intend to look it over fairly closely, but, a cursory look showed the authors were taking it in a slightly different direction than I anticipated. I’ll certainly look through the code again, but, the way I’ve envisioned this, I think there are some fundamental points that clash.

As I already have much of the database model written for an older PHP CMS that I wrote, I’m addressing some of the shortcomings I ran across with that design and modifying the models to be a little more generic. While I am sure there are proprietary products that currently utilize ESI, I believe my approach is unique and flexible enough to power everything from a blog to a site or forums or even a classified ads site.

Here’s a programmer that is saying goodbye to ORMs at Hatful of Hollow.

And another site offering a tutorial of sorts dealing with ORMs Why should you use an ORM.

While both have their points, both have missed a fundamental benefit that an ORM hands you.

Most of my development is in Pylons. Django’s ORM and template language can do the same thing. A programmer that has used PHP/Smarty to develop large scale systems will likely resist ORMs. After working with a team to develop 90k+ lines of PHP/Smarty over a six year period, making the shift required a paradigm shift.

Let’s consider the following structure. We have a cp_ticket table and a cp_ticket_detail table. A Ticket can have multiple detail records. The output we wish to have is:

ticket id, ticket header information
         ticket detail line
         ticket detail line #2
ticket id, ticket header information
         ticket detail line
         ticket detail line #2
         ticket detail line #3
ticket id, ticket header information
         ticket detail line
         ticket detail line #2

Our model:

class cp_ticket(DeclarativeBase):
    __tablename__ = 'cp_ticket'

    ticket_id = Column(mysql.MSBigInteger(20, unsigned = True), primary_key=True, autoincrement = True)
    priority = Column(mysql.MSEnum('1','2','3','4','5'), default = '3')

    ticket_detail = relation('cp_ticket_detail', order_by='cp_ticket_detail.ticket_detail_id')

class cp_ticket_detail(DeclarativeBase):
    __tablename__ = 'cp_ticket_detail'

    ticket_id = Column(mysql.MSBigInteger(20, unsigned = True), ForeignKey('cp_ticket.ticket_id'), default = '0')
    ticket_detail_id = Column(mysql.MSBigInteger(20, unsigned = True), primary_key=True, autoincrement = True)
    stamp = Column(mysql.MSTimeStamp, PassiveDefault('CURRENT_TIMESTAMP'))
    detail = Column(mysql.MSLongText, default = '')

Our query to pass to our template:

        tickets = meta.Session.query(cp_ticket).filter(cp_ticket.client_id==1).all()

Compared with the query as you would write it without an ORM:

select * from cp_ticket,cp_ticket_detail where client_id=1 and cp_ticket.ticket_id=cp_ticket_detail.ticket_id;

Both are doing the same fundamental thing, but, the ORM maps the results almost identical to the way we want to display the data. This makes template design easy.

Using Mako, we use the following code to display the results:

<table border="1">
 <tr><th>Ticket ID</th><th>Status</th><th>Detail</th></tr>
%for ticket in tmpl_context.tickets:
  <tr>
    <td><strong>${ticket.ticket_id}</strong></td>
    <td><strong>${ticket.priority}</strong></td>
  </tr>
  %for detail in ticket.ticket_detail:
  <tr>
    <td></td>
    <td>${detail.stamp}</td>
    <td>${detail.detail}</td>
  </tr>
  % endfor
% endfor
</table>

To do the same thing without using an ORM, you need to revert to a control break structure similar to the following:

current_ticket=0
for ticket in tickets:
  if (current_ticket != ticket.ticket_id):
    #new row, print the header
    print "<tr><td>first piece</td></tr>"
    current_ticket = ticket.ticket_id
  # print our detail row
  print "<tr><td></td><td>stamp and detail</td></tr>"

Control Break structures require you to be able to set a variable within your template language. Some template languages don’t allow that. If your template language (in any language) can’t do variable assignments in the template, guess where your html generation logic needs to go?

With an ORM, the template contains your display logic. Your webmaster/design team can modify the template without having to modify html contained within your code. The loops are simple to understand and designers usually have little problem avoiding the lines that start with %.

Sure, you could wrap much of this logic in your template to do the control-break structure, but, as you get more complex data, deciding how to display the data requires a define or some other functionality.

An ORM adds some insulation to the process, but, the result is a much easier page structure when displaying related data. Granted there are some performance hits and SQLAlchemy appears to create some queries that are not optimal, unless there is a tremendous performance hit, I think the benefits of the ORM for developing a web application are tremendous.

Once you move into an environment where you are dealing with multiple developers, having a defined schema with comments is much easier than using reflection to figure out what the meaning of a status field as enum(‘U’,’A’,’P’,’C’,’R’,’S’).

However, as the original poster mentions, you can do raw SQL within SQLAlchemy and do all of your work with reflection as he has done with his ORM^H^H^H, abstraction. If he’s still using SQLAlchemy, he can selectively decide when to use it and when to avoid it.

While doing some coding to move a Turbogears2 application over to Pylons, I ran into an issue with Validation and ToscaWidgets.

AttributeError: ‘function’ object has no attribute ‘replace’

Validation in Pylons listed:

@validate(form=movie_form, error_handler=index)

for the decorator syntax, but, error_handler should be a name, not a function. The correct decorator should be:

@validate(form=movie_form, error_handler=’index’)