Random Musings of an Insane Mind

On Nov 4/Nov 5 a rumor that was later substantiated regarding Pylons surfaced. While the initial message was regarding TurboGears which is based on Pylons, the rumor was that Pylons was being rolled into another framework. Ben Bangert issued a post that gave some of the reasoning behind the merger of repoze.bfg and Pylons.

Since we’ve been working on an application for the last few weeks that isn’t in production, it seemed like an ideal test case. While the documentation for Pyramid is superb, transitioning from Pylons to Pyramid still has a few rough edges as the terminology that Pylons developers are used to were changed.

One of the first issues is the way that Pylons handles routes and converting to Pyramid. In Pylons, default route entries are set in config/routing.py:

    map.connect('/{controller}/{action}')
    map.connect('/{controller}/{action}/{id}')

To emulate that behavior in Pyramid, modify __init__.py:

    config.add_handler('client', '/client/:action', handler=Client)
    config.add_handler('clientid', '/client/:action/:id', handler=Client)

Pyramid doesn’t scan controllers, so, if you have multiple controllers, you’ll need to specify each. Also, make sure that you use a unique name (client|clientid) for each handler that you’ve added to avoid any 404s. Rather than the old controller structure you had, your code is now considerably cleaner and looks like:

from pyramid.response import Response
from pyramid.view import action

class Client(object):
    def __init__(self, request):
        self.request = request
        self.dbsession = DBSession()

    @action(renderer='client_index.jinja2')
    def index(self):
        return {'views':5, 'clicks':1}

The decorator signifies the template that you want to use and variables that you want to pass to the template are returned. This ends most of the tmpl_context. or c. clutter that was present in controllers and templates.

More documentation on handlers is available here.

An early version of the pyramid templates does not contain the weberror helper that was available in Pylons. The Pylons paster templates that are included in Pyramid do have the helper. A discussion with Chris McDonough should result in the changes being made to the Pyramid templates as well.

Another area that needs attention is Flash messages. Currently they are not supported and webhelpers.flash appears to implement things in a manner that won’t work with Pyramid. Through the subscribers method in Pyramid, it looks to be somewhat trivial to implement. The existing Pylons template in Pyramid does contain passthroughs of the c./tmpl_context. globals and helper modules that were available in Pylons. Modifying that slightly should allow Flash messages to be easily enabled. If you are going to transition, it would make more sense to use one of the Pylons templates than to migrate straight to the Pyramid templates.

For reference, the file:
pyramid-1.0a1-py2.6.egg/pyramid/paster_templates/pylons_sqla/+package+/subscribers.py_tmpl
mentions the existing global handling of the special objects for which Flash messages can probably be reimplemented.

SQLAlchemy is supported, but, there is an additional extension loaded which appears to do an autocommit on SQL queries. A brief readthrough mentions that you need to use s.join() to join your two database queries to be handled. It appears that you can join two db handles on separate databases which makes this a bit more powerful than using normal transactions as you could ensure a record was written to mongoDB and MySQL. I need to spend a little more time reading through this.

pagination appears to depend on routes, and even with routes installed, an error is thrown with thread._local requiring a mapper which is probably not going to work with Pyramid and will require some rewriting. This appears to be the same issue (thread._local) with Flash messages and it was mentioned that both items were relatively high priority and easy fixes.

forms – By default, formish is installed. While their site was down, it was stated that formish was in no way an endorsement, it was just included as it was part of bfg and is not a dependency in Pyramid*. Some preliminary work with FormAlchemy showed that it should work without too much difficulty, but, I decided to give Deform a try. Since I had already looked at Deform in the past for Pylons, and the screencast demonstration was done in Pylons, I was somewhat familiar with the methods. Converting over to Deform was a matter of reworking a few schemas. I’ve had some difficulties getting Deform to work with output from SQLAlchemy. Basically, Deform works very well with ZODB which is a schemaless database. Using it to edit rows returned from SQLAlchemy requires one to manually iterate through the returned row to create an appstruct to hand to Deform. For a number of simple forms this probably wouldn’t be difficult. As our project has a number of GridSets, converting over to Deform would have been considerably more difficult. To get FormAlchemy to work, we required the following changes:

form.py:

from mako.template import Template

from formalchemy import config as fa_config
from formalchemy import templates
from formalchemy import validators
from formalchemy import fields
from formalchemy import forms
from formalchemy import tables
from formalchemy.ext.fsblob import FileFieldRenderer
from formalchemy.ext.fsblob import ImageFieldRenderer

fa_config.encoding = 'utf-8'

class TemplateEngine(templates.TemplateEngine):
    def render(self, name, **kwargs):
        return Template(filename='/var/www/pyr/atg/atg/templates/forms/%s.mako' % name).render(**kwargs)

fa_config.engine = TemplateEngine()

class FieldSet(forms.FieldSet):
    pass

class Grid(tables.Grid):
    pass

our file using formalchemy:

from pyramid.httpexceptions import HTTPRedirection

from formalchemy import validators
from formalchemy.fields import Field
from atg.form import FieldSet

dbsession = DBSession()

User = FieldSet(auth.AuthUser, session=dbsession)
User.configure(
    include = [
        User.contact,
        User.email,
        User.company,
        User.addr1,
        User.addr2,
        User.city,
        User.state,
        User.zip,
        User.phone,
    ],
    options=[User.email.set(validate=validators.email)]
)

    @action(renderer='client_account.jinja2')
    def account(self):
        record = self.dbsession.query(auth.AuthUser).filter(auth.AuthUser.id==self.uid).first()
        fs = User.bind(record, data=self.request.POST or None)
        if self.request.POST and fs.validate():   
            fs.sync()
            self.dbsession.merge(record)
            self.dbsession.flush()
            HTTPRedirection(location='/client/account')
        return {'fs':fs}

model from auth.py:

class AuthUser(Base):
    __tablename__ = 'auth_users'

    id = Column(mysql.BIGINT(20, unsigned=True), primary_key=True, autoincrement
=True)
    username = Column(Unicode(80), nullable=False)
    _password = Column('password', Unicode(80), nullable=False)
    email = Column(Unicode(80), nullable=False)
    contact = Column(Unicode(80), nullable=False)
    company = Column(Unicode(80), nullable=False)
    addr1 = Column(Unicode(80), nullable=False)
    addr2 = Column(Unicode(80))
    city = Column(Unicode(80), nullable=False)
    state = Column(Unicode(80), nullable=False)
    zip = Column(Unicode(80), nullable=False)
    phone = Column(Unicode(80), nullable=False)

client_account.jinja2:

<form method="post">
{{ fs.render()|safe }}
<input type="submit" value="save">
</form>

There is a minor problem with validation with FormEncode that we’re still working with dealing with validation on a form that has been bound.

webhelpers – Most webhelpers appear to work fine. It was refreshing to see that many of the webhelpers.constants work fine without having to swap the order of the tuples with Deform/Formish. Currently, Flash and Paginate are broken as mentioned above, but, those will be fixed relatively quickly.

Authentication is built in. While the permissions system is quite well thought out, getting it to work in a basic fashion required quite a bit of tweaking. Basically:

__init__.py:

from pyramid.authentication import AuthTktAuthenticationPolicy
from pyramid.authorization import ACLAuthorizationPolicy

    authn_policy = AuthTktAuthenticationPolicy(
        'sosecret', callback=groupfinder)
    authz_policy = ACLAuthorizationPolicy()
    config = Configurator(settings=settings,
                          root_factory='atg.models.RootFactory',
                          authentication_policy=authn_policy,
                          authorization_policy=authz_policy)

    config.add_route('login', '/login',
                     view='atg.login.login',
                     view_renderer='atg:templates/login.pt')
    config.add_route('admin2', '/admin/', view='atg.admin.index', view_permission='edit', 
                     view_renderer='admin_index.jinja2')
    config.add_handler('admin', '/admin/:action', handler=Admin, permission='edit')

login.py:

from pyramid.httpexceptions import HTTPFound
from pyramid.security import remember
from pyramid.security import forget
from pyramid.url import route_url

from atg.security import USERS

def login(request):
    login_url = route_url('login', request)
    referrer = request.url
    if referrer == login_url:
        referrer = '/' # never use the login form itself as came_from
    came_from = request.params.get('came_from', referrer)
    message = ''
    login = ''
    password = ''
    if 'form.submitted' in request.params:
        login = request.params['login']
        password = request.params['password']
        if USERS.get(login) == password:
            headers = remember(request, login)
            return HTTPFound(location = came_from,
                             headers = headers)
        message = 'Failed login'

    return dict(
        message = message,
        url = request.application_url + '/login',
        came_from = came_from,
        login = login,
        password = password,
        )
    
def logout(request):
    headers = forget(request)
    return HTTPFound(location = route_url('view_wiki', request),
                     headers = headers)

security.py:

USERS = {'editor':'editor',
          'viewer':'viewer'}
GROUPS = {'editor':['group:editors']}

def groupfinder(userid, request):
    if userid in USERS:
        return GROUPS.get(userid, [])

templates/login.pt:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html
    xmlns="http://www.w3.org/1999/xhtml"
    xmlns:tal="http://xml.zope.org/namespaces/tal">

<head>
  <meta content="text/html; charset=utf-8" http-equiv="Content-Type"/>
  <title>bfg tutorial wiki (based on TurboGears 20-Minute Wiki)</title>
  <link rel="stylesheet" type="text/css"
        href="${request.application_url}/static/style.css" />
</head>

<body>

<h1>Log In</h1>

<div tal:replace="message"/>

<div class="main_content">
  <form action="${url}" method="post">
    <input type="hidden" name="came_from" value="${came_from}"/>
    <input type="text" name="login" value="${login}"/>
    <br/>
    <input type="password" name="password" value="${password}"/>
    <br/>
    <input type="submit" name="form.submitted" value="Log In"/>
  </form>
</div>  

</body>
</html>

I’ve not gotten the 401/403 error to prompt for a login, but, I believe that is something minor that I’m missing. In addition, I need to modify the schema to use an SQL backend.

All in all, transitioning the code took a bit longer than anticipated and I received some great advice and help from Ben Bangert and Chris McDonough on IRC. I tried to keep my questions to a minimum, but, transitioning from Pylons to Pyramid is going to be harder than moving from repoze.bfg to Pyramid.

Overall, I feel pretty good about the move. I had been a TurboGears users since 2.0-beta and moved over to Pylons for most development a few years back. After having spent 16-20 hours working with Pyramid, I believe that Pyramid is a step in the right direction. In addition to moving this app from Pylons to Pyramid, we switched from Mako to Jinja which required some rewrites of the templates. Mako is still used for FormAlchemy and is loaded in our Pyramid installation.

I don’t really see anything that would make me consider using another framework and I think it is a step in the right direction. Quick apachebench tests show some performance improvements which is also a nice benefit. While the software has an Alpha designation, from a stability standpoint, I’d say it performs more like a Release Candidate but the Alpha designation is probably maintained so that API changes can be pushed as necessary.

To the entire Pylons and Repoze.bfg teams, I say Congratulations! I see some great possibilities on the horizon with a larger community behind Pyramid.

* Chris McDonough helped clarify this (tweet)

After working through quite a bit of the documentation on the pylons site and the repoze site, I didn’t really find a step by step guide to get repoze.who/repoze.what working with Pylons.

Some of the references used:

* http://code.gustavonarea.net/repoze.what-pylons/Manual/Protecting.html
* http://wiki.pylonshq.com/display/pylonscookbook/Authorization+with+repoze.what

Thanks to my nephew Qwait for overloading ActionProtector to intercept 403s and give an additional chance to authenticate.

wget http://pylonshq.com/download/1.0/go-pylons.py
python go-pylons.py --no-site-packages pylons
cd pylons
source bin/activate
easy_install repoze.what-quickstart
easy_install repoze.what-pylons
easy_install mysql-python
paster create -t pylons project
cd project

Because of the association table and the cascade, you must use MySQL or Postgresql rather than SQLite. You also might need to modify development.ini [server:main] host/port.

cd project

config/middleware.py:

    # CUSTOM MIDDLEWARE HERE (filtered by error handling middlewares)
    from project.lib.auth import add_auth
    app = add_auth(app, config)

config/routing.py:

    map.connect('/login', controller='login', action='login')
    map.connect('/login/submit', controller='login', action='login_handler')
    map.connect('/login/continue', controller='login', action='post_login')
    map.connect('/logout/continue', controller='login', action='post_logout')
    map.connect('/logout', controller='login', action='logout_handler')

lib/auth.py: – modified to intercept 403 and provide chance to authenticate

from pylons import response, url
from pylons.controllers.util import redirect

from repoze.what.plugins.quickstart import setup_sql_auth
from repoze.what.plugins import pylonshq

import project.lib.helpers as h

from project.model.meta import Session
from project.model.auth import AuthUser, AuthGroup, AuthPermission

def add_auth(app, config):
   return setup_sql_auth(
       app, AuthUser, AuthGroup, AuthPermission, Session,
       login_handler = '/login/submit',
       logout_handler = '/logout',
       post_login_url = '/login/continue',
       post_logout_url = '/logout/continue',
       cookie_secret = 'my_secret_word',
       translations = {
           'user_name' : 'username',
           'groups' : 'auth_groups',
           'group_name' : 'name',
           'permissions' : 'auth_permissions',
           'permission_name' : 'name'
       }
   )

def redirect_auth_denial(reason):
    if response.status_int == 401:
        message = 'You are not logged in.'
        message_type = 'warning'
    else:
        message = 'You do not have the permissions to access this page.'
        message_type = 'error'

    h.flash(message, message_type)
    redirect(url('/login', came_from=url.current()))

class ActionProtector(pylonshq.ActionProtector):
    default_denial_handler = staticmethod(redirect_auth_denial)

model/auth.py:

from sqlalchemy import *
from sqlalchemy.databases import mysql
from sqlalchemy.orm import relation, backref, synonym
from sqlalchemy.orm.exc import NoResultFound

from project.model.meta import Base

import os
from hashlib import sha1
from datetime import datetime

group_permission_table = Table('auth_group_permissions', Base.metadata,
    Column('group_id', mysql.BIGINT(20, unsigned=True), ForeignKey('auth_groups.id', onupdate='CASCADE', ondelete='CASCADE')),
    Column('permission_id', mysql.BIGINT(20, unsigned=True), ForeignKey('auth_permissions.id', onupdate='CASCADE', ondelete='CASCADE'))
)
user_group_table = Table('auth_user_groups', Base.metadata,
    Column('user_id', mysql.BIGINT(20, unsigned=True), ForeignKey('auth_users.id', onupdate='CASCADE', ondelete='CASCADE')),
    Column('group_id', mysql.BIGINT(20, unsigned=True), ForeignKey('auth_groups.id', onupdate='CASCADE', ondelete='CASCADE'))
)

class AuthGroup(Base):
    __tablename__ = 'auth_groups'

    id = Column(mysql.BIGINT(20, unsigned=True), primary_key=True, autoincrement=True)
    name = Column(Unicode(80), unique=True, nullable=False)
    created = Column(mysql.DATE())

    users = relation('AuthUser', secondary=user_group_table, backref='auth_groups')

    def __repr__(self):
        return '<group: name=%s>' % self.name

    def __unicode__(self):
        return self.name

class AuthUser(Base):
    __tablename__ = 'auth_users'

    id = Column(mysql.BIGINT(20, unsigned=True), primary_key=True, autoincrement=True)
    username = Column(Unicode(80), nullable=False)
    _password = Column('password', Unicode(80), nullable=False)
 
    @property
    def permissions(self):
        perms = set()
        for g in self.groups:
            perms = perms | set(g.permissions)
        return perms

    def _set_password(self, password):
        hashed_password = password

        if isinstance(password, unicode):
            password_8bit = password.encode('UTF-8')
        else:
            password_8bit = password

        salt = sha1()
        salt.update(os.urandom(60))
        hash = sha1()
        hash.update(password_8bit + salt.hexdigest())
        hashed_password = salt.hexdigest() + hash.hexdigest()

        if not isinstance(hashed_password, unicode):
            hashed_password = hashed_password.decode('UTF-8')
        self._password = hashed_password

    def _get_password(self):
        return self._password

    password = synonym('_password', descriptor=property(_get_password, _set_password))

    def validate_password(self, password):
        hashed_pass = sha1()
        hashed_pass.update(password + self.password[:40])
        return self.password[40:] == hashed_pass.hexdigest()

    def __repr__(self):
        return '<user: id="%s" username="%s" email="%s">' % (self.id, self.username, self.email)

    def __unicode__(self):
        return self.username

class AuthPermission(Base):
    __tablename__ = 'auth_permissions'

    id = Column(mysql.BIGINT(20, unsigned=True), primary_key=True, autoincrement=True)
    name = Column(Unicode(80), unique=True, nullable=False)
    description = Column(mysql.TEXT())

    groups = relation(AuthGroup, secondary=group_permission_table, backref='auth_permissions')

    def __unicode__(self):
        return self.permission_name

controllers/login.py:

from pylons import request, response, session, tmpl_context, config, url
from pylons.controllers.util import redirect

from project.lib.base import BaseController, render
from project.lib.helpers import flash

class LoginController(BaseController):
    def login(self):
        login_counter = request.environ['repoze.who.logins']
        if login_counter > 0:
            flash('Wrong credentials')
        tmpl_context.login_counter = login_counter
        tmpl_context.came_from = request.params.get('came_from') or url('/')
        return render('login.mako')

    def login_handler(self):
        pass

    def post_login(self):
        identity = request.environ.get('repoze.who.identity')
        came_from = str(request.params.get('came_from', '')) or url('/')
        if not identity:
            login_counter = request.environ['repoze.who.logins'] + 1
            redirect(url('/login', came_from=came_from, __logins=login_counter))
        redirect(came_from)

    def logout_handler(self):
        pass
     
    def post_logout(self):
        redirect('/')

templates/login.mako:

<% messages = h.flash.pop_messages() %>
% if messages:
<div class="flash">
    % for message in messages:
      <p class="${message.category}">${message}
    % endfor
</div>
% endif

  <form action="${h.url('/login/submit', came_from=tmpl_context.came_from, __logins=tmpl_context.login_counter)}" method="POST">
    <label for="login">Username:<input type="text" id="login" name="login" /><br />
    <label for="password">Password:<input type="password" id="password" name="password" />
    <input type="submit" value="Login" />
  </form>

controllers/root.py:

from pylons import request, response, session, tmpl_context, config

#from repoze.what.plugins.pylonshq import ActionProtector, ControllerProtector
from project.lib.auth import ActionProtector
from repoze.what.predicates import is_user, has_permission, in_group

from project.lib.base import BaseController, render

class RootController(BaseController):
    def index(self):
        return render('index.mako')

    @ActionProtector(is_user('test'))
    def user(self):
        return render('loggedin.mako')

    @ActionProtector(is_user('nottest'))
    def notuser(self):
        return render('loggedin.mako')

    @ActionProtector(in_group('admin'))
    def admin(self):
        return render('loggedin.mako')

    @ActionProtector(has_permission('edit'))
    def edit(self):
        return render('loggedin.mako')

lib/helpers.py:

from pylons import url

from webhelpers.pylonslib import Flash as _Flash
flash = _Flash()

websetup.py: – after the Session, Base import:

from project.model.auth import *

Setup/Create the database, start paster

paster setup-app development.ini
paster serve --reload development.ini

createuser.py:

#!/usr/bin/python2.6

from sqlalchemy import create_engine
engine = create_engine('mysql://user:pass@localhost/dbname', echo=True)
from sqlalchemy.orm import sessionmaker
Session = sessionmaker(bind=engine)
session = Session()

from project.model.auth import *

u = AuthUser()
u.username = u'test'
u.password = u'test'
session.add(u)
g = AuthGroup()
g.name = u'admin'
g.users.append(u)
session.add(g)
p = AuthPermission()
p.name = u'edit'
p.groups.append(g)
session.add(p)
session.commit()

While discussing a project with a client it occurred to me that perhaps you don’t need to get too complex to do something simple. Since using Facebook almost mandates that your surfer has Javascript enabled, we should be able to write a very simple application that posts to someone’s wall after asking for the ‘publish’ permission. After reading the documentation and looking through a few github repositories, the solution was quite simple.

While this code doesn’t use the non-blocking async javascript method, it is a good example. I’ve seen others that use the inline publish, but, for some reason I couldn’t get it to consistently open a dialog box rather than a popup – which Chrome conveniently blocked. I also ran into an issue that Facebook appears to aggressively cache objects that don’t explicitly set an expire time. While I find that acceptable for the application’s long term goals, it did make debugging a slightly frustrating experience.

To try the application, http://apps.facebook.com/onepageapp/. A direct link to the code is available at http://fbapp.cd34.com/opa/index.txt.

<!DOCTYPE html>
<html xmlns="http://www.w3.org/1999/xhtml" xmlns:fb="http://www.facebook.com/2008/fbml">
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>
        <title>One Page App</title>
    </head>
    <body>
<div id="fb-root"></div>
Thanks for clicking.  This is just a test to make sure that the application
works as expected.
<p>
You can skip posting to the wall if you would like.
<script src="http://connect.facebook.net/en_US/all.js"></script>
<script>
appid = '154516391233318';
name = 'One Page App';
caption = 'A single page application designed to test whether it could be done';
description = 'A quick page that allows you to post to someone\'s wall';
href = 'http://apps.facebook.com/onepageapp/';
user_message_prompt = 'Post a sample to your wall';
message = 'Let\'s do a sample post to the wall.';
action_text = 'Get the code';
action_href = 'http://fbapp.cd34.com/opa/index.txt';

FB.init({appId  : appid, status : true, cookie : true, xfbml  : false });
FB.getLoginStatus(function(response) {
  if (response.session) {
    FB.ui(
      {
        method: 'stream.publish',
        display: 'dialog',
        message: message,
        attachment: {
          name: name,
          caption: caption,
          description: description,
          href: href
        },
        action_links: [
          { text: action_text, href: action_href }
        ],
        user_message_prompt: user_message_prompt
      },
      function(response) {
        self.location.href='/opa/thanks.html';
      }
    );
  } else {
    top.location.href='https://graph.facebook.com/oauth/authorize?client_id='+appid+'&redirect_uri='+href+'&display=page&scope=publish_stream';
  }
});
</script>
</body>
</html>

Cue Inspiration

I had an idea the other day for a simple Facebook application. With Pylons, I figured it would take no more than an hour or two to get the mechanics of the application working at which point a designer could come in to handle the rest. What followed was quite a struggle.

I’ve written Facebook applications using the old PHP SDK. While this method is rather well documented, moving to Python should have been easy. PyFacebook, a project hosted at GitHub, is woefully out of date. Even after applying three of the suggested patches, and modifying one of the imports in the library, I was left with numerous issues regarding the access_token, odd redirects and a few other minor issues. Add to this the fact that Facebook really suggests that applications use the Graph API and IFrames rather than the FBML canvas and we’re setting ourselves up for a problem down the road. Facebook has maintained that they will always support FBML and the REST API, but, new features won’t be accessible to the older API.

With that in mind, I looked at Python SDK, the officially recommended library for Python and Facebook. While Pylons is supported fairly well, going through the documentation on Facebook’s site resulted in looking through the PHP-SDK, the supplied oauth access in the Python-SDK and a bit of trial and error along the way. OAuth with the Graph API is a bit more complex to understand, but, if your application is using AJAX or JSON, avoiding the FBML proxy is much quicker. Flash was used quite a bit with FBML applications so that applications could communicate directly with a game server which made things much quicker. HTML apps using FBML often exhibited pageload performance problems. With the IFrame method, your application still runs within Facebook’s canvas, but, the surfer is communicating directly with your application.

What happened?

First, I tried to replicate what I had done in PHP using PyFacebook and Pylons. While there are hooks for WSGI (which includes Paster/Pylons implemented servers), there were a number of issues. I briefly tried Django with PyFacebook and met different issues. Once you stray from PHP, you’re in uncharted territory. A statistic I read somewhere claimed that only a few hundred apps were developed in something other than PHP with Java/JSP being the most common alternate. Django, the Google App Engine and web.py appear to be the favorites among Python frameworks. While I know there are a handful of applications running Pylons, and at least one running TurboGears, I don’t believe there are many using the Graph API.

At this point, the Graph API and OAuth seemed to be the sane choice. An IFrame canvas, using the Javascript SDK to complement Python SDK appeared to be the answer.

The first stumbling block when following the OAuth guide on Facebook is the frame in a frame shaded authentication box. Clicking on the grey box opens the inner frame to the full page where you can authorize the application, but, that is a rather ugly situation. The following Javascript fixes that which isn’t great solution, but does work.

<script language="javascript">
top.location.href='https://graph.facebook.com/oauth/authorize?client_id=${config['facebook.appid']}&redirect_uri=${config['facebook.callbackurl']}&display=page&scope=publish_stream';
</script>

Error validating verification code

After working with a few other issues, another issue with the auth_token resulted in the following error (after loading the url that was being fetched):

{
   "error": {
      "type": "OAuthException",
      "message": "Error validating verification code."
   }
}

Adding &type=client_cred to your access_token url fixes that situation.

Here’s the guide

We’re going to put our project in the facebook directory and use Pylons 1.0:

git clone http://github.com/facebook/python-sdk.git
wget http://www.pylonshq.com/download/1.0/go-pylons.py
python go-pylons.py facebook
cd facebook
source bin/activate
paster create -t pylons fbapp
cd fbapp
vi development.ini
rm fbapp/public/index.html
cp ../python-sdk/src/facebook.py fbapp/fbapp/lib

We need to make a few changes to our development.ini in the [app:main] section:

facebook.callbackurl = http://apps.facebook.com/ourfbapp/
facebook.apikey = 6b5aca8bd71c1234590e697f79xxxxxx
facebook.secret = df5d928b87c0df312c8be101e5xxxxxx
facebook.appid = 124322020xxxxxx

modify config/routing.py:

    map.connect('/{action}', controller='root')
    map.connect('/', controller='root', action='index')

templates/oauth_redirect.mako:

<script language="javascript">
top.location.href='https://graph.facebook.com/oauth/authorize?client_id=${config['facebook.appid']}&redirect_uri=${config['facebook.callbackurl']}&display=page&scope=publish_stream';
</script>
<noscript>
<a href="https://graph.facebook.com/oauth/authorize?client_id=${config['facebook.appid']}&redirect_uri=${config['facebook.callbackurl']}&display=page&scope=publish_stream" target="_top">Click here to authorize this application</a>
</noscript>

templates/index.mako:

${tmpl_context.user}

controllers/root.py:

# using python 2.5
import simplejson
import cgi
import urllib

from pylons import request, response, session, tmpl_context, config
from pylons.controllers.util import abort, redirect

from fbapp.lib.base import BaseController, render
import fbapp.lib.facebook as facebook

class RootController(BaseController):

    def __before__(self):
        tmpl_context.user = None
        if request.params.has_key('session'):
            access_token = simplejson.loads(request.params['session'])['access_token']
            graph = facebook.GraphAPI(access_token)
            tmpl_context.user = graph.get_object("me")

    def index(self):
        if not tmpl_context.user:
            return render('/oauth_redirect.mako')
        return render('/index.mako')

In Facebook, you want to make the following changes:

Canvas Callback URL: http://yourdomain.com/
Connect URL: http://yourdomain.com/
Canvas URL: http://apps.facebook.com/yourappname/
FBML/Iframe: iframe
Application Type: website

Under the Migrations Tab, Verify that New Data Permissions and New SDKs are both set to enabled. When you write your application, you can refer to the extended permissions which are set in the &scope= section of oauth_redirect.mako.

What’s next?

Once you’ve retrieved the access_token and the user_id, you probably want to save that into your local database so that you don’t need to fetch the data from the Graph API on every pageload. While the Graph method is indeed faster than FBML, Facebook has lifted some of the restrictions regarding the data you can keep which allows for faster pageloads. With the IFrame method, pages using AJAX/JSON are indeed much quicker.

While I originally estimated this project to take ‘a few hours’, working through all of the possible scenarios with the Python Facebook SDK ended up taking quite a bit more time than expected. The Graph API is very well thought out and is much faster than the REST API and appears to be almost as fast as FQL.

Good Luck!

Many years ago I ran into a situation with a client where the amount of traffic they were receiving was crushing their dynamically created site. Computation is always the enemy of a quick pageload, so, it is very important to do as little computation as possible when delivering a page.

While there are many ways to put together a CMS, high traffic CMS sites usually involve caching or lots of hardware. Some write static files which are much less strenuous, but, you lose some of the dynamic capabilities. Fragment caching becomes a method to make things a bit more dynamic as MasonHQ does with their page and block structure. Django-blocks was surely influenced by this or reinvented this method.

In order to get the highest performance out of a CMS with a page and block method, I had considered writing a filesystem or inode linklist that would allow the webserver to assemble the page by following the inodes on the disk to build the page. Obviously there are some issues here, but, if a block was updated by a process, it would automatically be reassembled. This emulates a write-through cache and would have provisions for dynamic content to be mixed in with the static content on disk. Assembly of the page still takes more compute cycles than a static file but is significantly less than dynamically creating the page from multiple queries.

That design seriously limits the ability to deploy the system widely. While I can control the hosting environment for personal projects, the CMS couldn’t gain wide acceptance. While Varnish is a rather simple piece of software to install, it does limit deploy-ability, but, provides a significant piece of the puzzle due to Edge Side Includes (ESI). If the CMS gets used beyond personal and small deployments, Akamai supports Edge Side Includes as well.

Rather than explain ESI, ESI Explained Simply contains about the best writeup I’ve seen to date to explain how ESI can be used.

The distinction here is using fragment caching controlled by ESI to represent different zones on the page. As a simple example, lets consider our page template contains an article and a block with the top five articles on the site. When a new post is added, we can expire the block that contains the top five articles so that it is requested on the next page fetch. Since the existing article didn’t change, the interior ESI included block doesn’t need to be purged. This allows the page to be constructed on the Edge rather than on the Origin server.

As I have worked with a number of PHP frameworks, none really met my needs so I started using Python frameworks roughly two years ago. For this CMS, I debated using Pylons or Django and ended up choosing Django. Since both can be run behind WSGI compliant servers, we’ve opened ourselves up to a number of potential solutions. Since we are running Varnish in front of our Origin server, we can run Apache2 with mod_wsgi, but, we’re not limited to that configuration. At this point, we have a relatively generic configuration the CMS can run on, but, there are many other places we can adapt the configuration for our preferences.

Some of the potential caveats:
* With Varnish or Akamai as a frontend, we need to pay closer attention to X-Forwarded-For:
* Web logs won’t exist because Varnish is serving and assembling the pages (There is a trick using ESI that could be employed if logging was critical)
* ESI processed pages with Varnish are not compressed. This is on their wishlist.

Features:
* Content can exist in multiple categories or tags
* Flexible URL mapping
* Plugin architecture for Blocks and Elements
* Content will maintain revisions and by default allow comments and threaded comments

Terms:
* Template – the graphical layout of the page with minimal CMS markup
* Element – the graphical template that is used to render a Block
* Block – a module that generates the data rendered by an Element
* Page – a Page determined by a Title, Slug and elements
* Content – The actual data that rendered by a block

Goals:
* Flexible enough to handle something as simple as a personal blog, but, also capable of powering a highly trafficed site.
* Data storage of common elements to handle publishing of content and comments with the ability to store information to allow threaded comments. This would allow the CMS to handle a blog application, a CMS, or, a forum.
* A method to store ancillary data in a model so that upgrades to the existing database model will not affect developed plugins.
* Block system to allow prepackaged css/templating while allowing local replacement without affecting the default package.
* Upgrades through pypy or easy_install.
* Ability to add CDN/ESI without needing to modify templates. The system will run without needing to be behind Varnish, but, its full power won’t be realized without Varnish or Akamai in front of the origin server.
* Seamless integration of affiliate referral tracking and conversion statistics

At this point, the question in my mind was whether or not to start with an existing project and adapt it or start from scratch. At this point, the closest Django CMS I could find was Django-Blocks and I do intend to look it over fairly closely, but, a cursory look showed the authors were taking it in a slightly different direction than I anticipated. I’ll certainly look through the code again, but, the way I’ve envisioned this, I think there are some fundamental points that clash.

As I already have much of the database model written for an older PHP CMS that I wrote, I’m addressing some of the shortcomings I ran across with that design and modifying the models to be a little more generic. While I am sure there are proprietary products that currently utilize ESI, I believe my approach is unique and flexible enough to power everything from a blog to a site or forums or even a classified ads site.