Random Musings of an Insane Mind

My first attempt at migrating a Pylons project to Pyramid was accomplished without too much difficulty. That project was relatively small, however, this project hasn’t been put into production.

A brief history of this application:

We have a legacy PHP application which consists of 78k lines of code. Originally we started rewriting the application in Turbogears 2.0 and later moved over to Pylons with ToscaWidgets. We used ToscaWidgets because all of our forms had been written for TG2. The application never made it into production, but, was substantial enough that we felt it would be a good test to convert it to Pyramid since we’re focusing our development efforts on Pyramid.

Initially I started to write a script to do the migration from Pylons to Pyramid and from ToscaWidgets to Deform but abandoned that. Changing the controllers from Pylons to Pyramid was fairly easy.

Most of the pylons imports were commented out and replaced with:

import deform
import colander

from pyramid.response import Response
from pyramid.view import action
from pyramid.security import authenticated_userid
from pyramid.threadlocal import get_current_request
import webhelpers.paginate as paginate

References to tmpl_context. were altered, our __before__ action for authkit was removed and replaced with an __init__ since we were using handlers. Routes were modified and given unique names, templates were modified to remove references to ${h. and ${tmpl_context., and we began the process of stepping through the application. Initially I had written a script to convert the ToscaWidgets form models over to Deform schemas, but, after two hours, it became obvious it would take more time to do that than to manually recreate the forms. After working through much of the process, I’m debating whether this project should have been migrated over to FormAlchemy as almost every form is a duplicate of the SQL schema. Since it isn’t in production, we still have some time to make that decision.

What follows is a summary of the errors received and what caused the errors.

TypeError: ‘NoneType’ object is not iterable

I ran into this while importing some routes from an existing project. Since routes must be uniquely named in Pyramid, a route with the same name will replace a prior route. Some of the route names have gotten quite unwieldy.

TypeError: object.__new__() takes no parameters

Solution, add the __init__ block to your class in your handler.

class YourClass(object):
    def __init__(self, request):
        self.request = request

ValueError: Non-response object returned from view named (and no renderer): {‘template’: ‘billing_index’}

Turbogears 2.0 code:

    def index(self, **kw):
        return dict(template='billing_index')

modify to:

    @action(renderer='billing_index.mako', permission='client')
    def index(self, **kw):
        return {}

AttributeError: ‘Undefined’ object has no attribute ‘form’

Pyramid doesn’t pass the tmpl_context., c. or h. through to the template, so, the return values of each of the actions needs to pass the fields required. As a result, any template code that replies on these globals needs to be modified, and the corresponding action needs to return the values in the dictionary. You can use one of the pylons templates which will instantiate a subscriber method to replicate the Pylons globals if you want.

AttributeError: ‘Undefined’ object has no attribute ‘literal’

Existing forms are relying on the pylons h. global.

${h.literal(form(value=value))}

UnboundLocalError: local variable ‘clients’ referenced before assignment

This occurs when you remove tmpl_context. and are left with a variable assigned that matches the class. For example:

tmpl_context.clients = meta.Session.query(clients).filter_by(client_id==1).all()

When tmpl_context. is removed, the class clients is cast incorrectly.

AttributeError: ‘Undefined’ object has no attribute ‘pager’

Paginate itemset isn’t being passed to the template.

TypeError: ‘NoneType’ object is not iterable

Paginate is getting the result set, rather than the paginate set passed in the return dictionary.

NameError: global name ‘request’ is not defined

request.matchdict/request.params -> self.request.matchdict/self.request.params

In a handler, request. is referred to as self.request. If you’ve converted things over to a handler rather than writing the individual routes for each action, you’ll need to preface any request. with self.

NotImplementedError: no URL generator available

Webhelpers pagination doesn’t know how to generate URLs in Pyramid, but, you can use a callable to generate the URLs. This requires access to pyramid.threadlocal which is generally not recommended, but, does allow you to use paginate until a pyramid compatible paginate is written.

The generator you need looks like this:

from webhelpers.util import update_params
from pyramid.threadlocal import get_current_request

def get_page_url(**kw):
    return update_params(get_current_request().path_info, **kw)

In your paginate block, you need to add the following:

url=get_page_url,

        paginator = paginate.Page(
            features,
            page=int(self.request.params.get('page', 1)),
            items_per_page = 40,
            url=get_page_url,
        )

RuntimeError: Caught exception rendering template. TypeError: ‘int’ object is not iterable

deform select widget requires tuple for the dropdown creation. Toscawidgets would build the right hand side if it was passed a list of IDs.

TypeError: ‘Undefined’ object is unsubscriptable

Returning a dict, removing tmpl_context., a ${value[‘asdf’]} is missing ‘value’:value being passed in the return dict.

TypeError: sequence item 10: expected string or Unicode, Undefined found

When using deform, return {‘form’:form.render()} rather than return {‘form’:form}.

Summary

This is the second application I’ve converted from Pylons 1.0 to Pyramid and most of the issues have been syntax issues. Hopefully the error summary above will save someone some time.

For one of our projects we used Pylons 1.0 with Repoze.who/Repoze.what, though, we only used groups without permissions. In our application, a member could access one of three controllers based on their membership in a group. There are a number of methods that can be used to set up authentication. One method is to create a thin table with the Primary Key, Username and Password and use an association table to add any additional information. This has the ability to be flexible when you need to add a number of fields that shouldn’t be contained within the AuthUser table. Another method is to use the AuthUser table to hold the related information. This example uses the latter method.

The first changes made are to __init__.py to add the policies and Forbidden view. This view allows us to prompt a user for a username/password when they visit a page that is protected by the ACLs in Pyramid.

auth.py contains our authentication model with the Permission models removed. We put the login/logout views, forbidden view along with the groupfinder and the RootFactory in login.py to consolidate authentication into an auth.py and login.py file for installations in other applications. Last but not least is a simple template to present the user with a login page.

For our RootFactory, we’ve defined three groups, client, manager and admin. Within our __init__.py (or our views if we use add_handler), we can restrict permissions using Pyramid’s authentication.

In our __init__.py, we can use the permission= to specify membership in a group.

config.add_route('admin2', '/admin/', view='project.admin.index', permission='admin', view_renderer='admin_index.jinja2')

If we have used add_handler:

config.add_route('admin2', '/admin/', view='project.admin.index', view_renderer='admin_index.jinja2')

We can use the permissions ACL in the @action decorator:

@action(renderer='admin_index.jinja2', permission='admin')

To access the userid in your views:

from pyramid.security import authenticated_userid

userid = authenticated_userid(request)

At this point, we’ve migrated a Repoze.who/Repoze.what authentication scheme that only used Group membership as its criteria and we have an SQL backed authentication system under Pyramid.

Most of the guidance for this came from:

* http://docs.pylonshq.com/pyramid/dev/tutorials/wiki2/authorization.html

Modifications to __init__.py:

Added to the import section:

from pyramid.authentication import AuthTktAuthenticationPolicy
from pyramid.authorization import ACLAuthorizationPolicy

from project.login import forbidden_view, groupfinder
from pyramid.exceptions import Forbidden

Added to the Configurator:

    authn_policy = AuthTktAuthenticationPolicy(
        'sosecret', callback=groupfinder)
    authz_policy = ACLAuthorizationPolicy()
    config = Configurator(settings=settings,
                          root_factory='project.login.RootFactory',
                          authentication_policy=authn_policy,
                          authorization_policy=authz_policy)

Added within the begin()/end() block of the Configurator

    config.add_view(forbidden_view, context=Forbidden)
    config.add_route('login', '/login',
                     view='project.login.login',
                     view_renderer='project:templates/login.pt')
    config.add_route('logout', '/logout',
                     view='project.login.logout',)

auth.py:

import transaction

from sqlalchemy import create_engine
from sqlalchemy import Column
from sqlalchemy import Integer
from sqlalchemy import Unicode

from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.declarative import declarative_base

from sqlalchemy.orm import scoped_session
from sqlalchemy.orm import sessionmaker

from zope.sqlalchemy import ZopeTransactionExtension

DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
Base = declarative_base()

from sqlalchemy import *
from sqlalchemy.databases import mysql
from sqlalchemy.orm import relation, backref, synonym
from sqlalchemy.orm.exc import NoResultFound

import os
from hashlib import sha1
from datetime import datetime

user_group_table = Table('auth_user_groups', Base.metadata,
    Column('user_id', mysql.BIGINT(20, unsigned=True), ForeignKey('auth_users.id', onupdate='CASCADE', ondelete='CASCADE')),
    Column('group_id', mysql.BIGINT(20, unsigned=True), ForeignKey('auth_groups.id', onupdate='CASCADE', ondelete='CASCADE'))
)

class AuthGroup(Base):
    __tablename__ = 'auth_groups'

    id = Column(mysql.BIGINT(20, unsigned=True), primary_key=True, autoincrement=True)
    name = Column(Unicode(80), unique=True, nullable=False)
    created = Column(mysql.DATE())

    users = relation('AuthUser', secondary=user_group_table, backref='auth_groups')

    def __repr__(self):
        return '' % self.name

    def __unicode__(self):
        return self.name

class AuthUser(Base):
    __tablename__ = 'auth_users'

    id = Column(mysql.BIGINT(20, unsigned=True), primary_key=True, autoincrement=True)
    username = Column(Unicode(80), nullable=False)
    _password = Column('password', Unicode(80), nullable=False)
    email = Column(Unicode(80), nullable=False)
    contact = Column(Unicode(80), nullable=False)
    company = Column(Unicode(80), nullable=False)

    groups = relation('AuthGroup', secondary=user_group_table, backref='auth_users')

    @property
    def permissions(self):
        perms = set()
        for g in self.groups:
            perms = perms | set(g.permissions)
        return perms

    def _set_password(self, password):
        hashed_password = password

        if isinstance(password, unicode):
            password_8bit = password.encode('UTF-8')
        else:
            password_8bit = password

        salt = sha1()
        salt.update(os.urandom(60))
        hash = sha1()
        hash.update(password_8bit + salt.hexdigest())
        hashed_password = salt.hexdigest() + hash.hexdigest()

        if not isinstance(hashed_password, unicode):
            hashed_password = hashed_password.decode('UTF-8')
        self._password = hashed_password

    def _get_password(self):
        return self._password

    password = synonym('_password', descriptor=property(_get_password, _set_password))

    def validate_password(self, password):
        hashed_pass = sha1()
        hashed_pass.update(password + self.password[:40])
        return self.password[40:] == hashed_pass.hexdigest()

    def __repr__(self):
        return '' % (self.id, self.username, self.email)

    def __unicode__(self):
        return self.username

login.py:

from pyramid.httpexceptions import HTTPFound
from pyramid.security import remember
from pyramid.security import forget
from pyramid.security import Allow
from pyramid.security import Everyone
from pyramid.url import route_url
from pyramid.renderers import render_to_response

from project.auth import AuthUser
from project.models import DBSession

def login(request):
    dbsession = DBSession()
    login_url = route_url('login', request)
    referrer = request.url
    if referrer == login_url:
        referrer = '/' # never use the login form itself as came_from
    came_from = request.params.get('came_from', referrer)
    message = ''
    login = ''
    password = ''
    if 'form.submitted' in request.params:
        login = request.params['login']
        password = request.params['password']
        auth = dbsession.query(AuthUser).filter(AuthUser.username==login).first()
        if auth.validate_password(password):
            headers = remember(request, login)
            """
                  We use the Primary Key as our identifier once someone has authenticated rather than the
                  username.  You can change what is returned as the userid by altering what is passed to
                  remember.
            """
            #headers = remember(request, auth.id)
            return HTTPFound(location = came_from,
                             headers = headers)
        message = 'Failed login'

    return dict(
        message = message,
        url = request.application_url + '/login',
        came_from = came_from,
        login = login,
        password = password,
        )
    
def logout(request):
    headers = forget(request)
    return HTTPFound(location = route_url('root', request),
                     headers = headers)
    
def forbidden_view(request):
    login_url = route_url('login', request)
    referrer = request.url
    if referrer == login_url:
        referrer = '/' # never use the login form itself as came_from
    came_from = request.params.get('came_from', referrer)
    return render_to_response('templates/login.pt', dict(
               message = '',
               url = request.application_url + '/login',
               came_from = came_from,
               login = '',
               password = '',
           ), request=request)

def groupfinder(userid, request):
    dbsession = DBSession()
    auth = dbsession.query(AuthUser).filter(AuthUser.id==userid).first()
    if auth:
        return [('group:%s' % group.name) for group in auth.groups]

class RootFactory(object):
    __acl__ = [ (Allow, 'group:client', 'client'),
                (Allow, 'group:manager', 'manager'),
                (Allow, 'group:admin', 'admin') ]
    def __init__(self, request):
        self.__dict__.update(request.matchdict)

templates/login.pt:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html
    xmlns="http://www.w3.org/1999/xhtml"
    xmlns:tal="http://xml.zope.org/namespaces/tal">

<head>
  <meta content="text/html; charset=utf-8" http-equiv="Content-Type"/>
  <title>Authentication Test</title>
  <link rel="stylesheet" type="text/css"
        href="${request.application_url}/static/style.css" />
</head>

<body>

<h1>Log In</h1>

<div tal:replace="message"/>

<div class="main_content">
  <form action="${url}" method="post">
    <input type="hidden" name="came_from" value="${came_from}"/>
    <input type="text" name="login" value="${login}"/>
    <br/>
    <input type="password" name="password" value="${password}"/>
    <br/>
    <input type="submit" name="form.submitted" value="Log In"/>
  </form>
</div>  

</body>
</html>

On Nov 4/Nov 5 a rumor that was later substantiated regarding Pylons surfaced. While the initial message was regarding TurboGears which is based on Pylons, the rumor was that Pylons was being rolled into another framework. Ben Bangert issued a post that gave some of the reasoning behind the merger of repoze.bfg and Pylons.

Since we’ve been working on an application for the last few weeks that isn’t in production, it seemed like an ideal test case. While the documentation for Pyramid is superb, transitioning from Pylons to Pyramid still has a few rough edges as the terminology that Pylons developers are used to were changed.

One of the first issues is the way that Pylons handles routes and converting to Pyramid. In Pylons, default route entries are set in config/routing.py:

    map.connect('/{controller}/{action}')
    map.connect('/{controller}/{action}/{id}')

To emulate that behavior in Pyramid, modify __init__.py:

    config.add_handler('client', '/client/:action', handler=Client)
    config.add_handler('clientid', '/client/:action/:id', handler=Client)

Pyramid doesn’t scan controllers, so, if you have multiple controllers, you’ll need to specify each. Also, make sure that you use a unique name (client|clientid) for each handler that you’ve added to avoid any 404s. Rather than the old controller structure you had, your code is now considerably cleaner and looks like:

from pyramid.response import Response
from pyramid.view import action

class Client(object):
    def __init__(self, request):
        self.request = request
        self.dbsession = DBSession()

    @action(renderer='client_index.jinja2')
    def index(self):
        return {'views':5, 'clicks':1}

The decorator signifies the template that you want to use and variables that you want to pass to the template are returned. This ends most of the tmpl_context. or c. clutter that was present in controllers and templates.

More documentation on handlers is available here.

An early version of the pyramid templates does not contain the weberror helper that was available in Pylons. The Pylons paster templates that are included in Pyramid do have the helper. A discussion with Chris McDonough should result in the changes being made to the Pyramid templates as well.

Another area that needs attention is Flash messages. Currently they are not supported and webhelpers.flash appears to implement things in a manner that won’t work with Pyramid. Through the subscribers method in Pyramid, it looks to be somewhat trivial to implement. The existing Pylons template in Pyramid does contain passthroughs of the c./tmpl_context. globals and helper modules that were available in Pylons. Modifying that slightly should allow Flash messages to be easily enabled. If you are going to transition, it would make more sense to use one of the Pylons templates than to migrate straight to the Pyramid templates.

For reference, the file:
pyramid-1.0a1-py2.6.egg/pyramid/paster_templates/pylons_sqla/+package+/subscribers.py_tmpl
mentions the existing global handling of the special objects for which Flash messages can probably be reimplemented.

SQLAlchemy is supported, but, there is an additional extension loaded which appears to do an autocommit on SQL queries. A brief readthrough mentions that you need to use s.join() to join your two database queries to be handled. It appears that you can join two db handles on separate databases which makes this a bit more powerful than using normal transactions as you could ensure a record was written to mongoDB and MySQL. I need to spend a little more time reading through this.

pagination appears to depend on routes, and even with routes installed, an error is thrown with thread._local requiring a mapper which is probably not going to work with Pyramid and will require some rewriting. This appears to be the same issue (thread._local) with Flash messages and it was mentioned that both items were relatively high priority and easy fixes.

forms – By default, formish is installed. While their site was down, it was stated that formish was in no way an endorsement, it was just included as it was part of bfg and is not a dependency in Pyramid*. Some preliminary work with FormAlchemy showed that it should work without too much difficulty, but, I decided to give Deform a try. Since I had already looked at Deform in the past for Pylons, and the screencast demonstration was done in Pylons, I was somewhat familiar with the methods. Converting over to Deform was a matter of reworking a few schemas. I’ve had some difficulties getting Deform to work with output from SQLAlchemy. Basically, Deform works very well with ZODB which is a schemaless database. Using it to edit rows returned from SQLAlchemy requires one to manually iterate through the returned row to create an appstruct to hand to Deform. For a number of simple forms this probably wouldn’t be difficult. As our project has a number of GridSets, converting over to Deform would have been considerably more difficult. To get FormAlchemy to work, we required the following changes:

form.py:

from mako.template import Template

from formalchemy import config as fa_config
from formalchemy import templates
from formalchemy import validators
from formalchemy import fields
from formalchemy import forms
from formalchemy import tables
from formalchemy.ext.fsblob import FileFieldRenderer
from formalchemy.ext.fsblob import ImageFieldRenderer

fa_config.encoding = 'utf-8'

class TemplateEngine(templates.TemplateEngine):
    def render(self, name, **kwargs):
        return Template(filename='/var/www/pyr/atg/atg/templates/forms/%s.mako' % name).render(**kwargs)

fa_config.engine = TemplateEngine()

class FieldSet(forms.FieldSet):
    pass

class Grid(tables.Grid):
    pass

our file using formalchemy:

from pyramid.httpexceptions import HTTPRedirection

from formalchemy import validators
from formalchemy.fields import Field
from atg.form import FieldSet

dbsession = DBSession()

User = FieldSet(auth.AuthUser, session=dbsession)
User.configure(
    include = [
        User.contact,
        User.email,
        User.company,
        User.addr1,
        User.addr2,
        User.city,
        User.state,
        User.zip,
        User.phone,
    ],
    options=[User.email.set(validate=validators.email)]
)

    @action(renderer='client_account.jinja2')
    def account(self):
        record = self.dbsession.query(auth.AuthUser).filter(auth.AuthUser.id==self.uid).first()
        fs = User.bind(record, data=self.request.POST or None)
        if self.request.POST and fs.validate():   
            fs.sync()
            self.dbsession.merge(record)
            self.dbsession.flush()
            HTTPRedirection(location='/client/account')
        return {'fs':fs}

model from auth.py:

class AuthUser(Base):
    __tablename__ = 'auth_users'

    id = Column(mysql.BIGINT(20, unsigned=True), primary_key=True, autoincrement
=True)
    username = Column(Unicode(80), nullable=False)
    _password = Column('password', Unicode(80), nullable=False)
    email = Column(Unicode(80), nullable=False)
    contact = Column(Unicode(80), nullable=False)
    company = Column(Unicode(80), nullable=False)
    addr1 = Column(Unicode(80), nullable=False)
    addr2 = Column(Unicode(80))
    city = Column(Unicode(80), nullable=False)
    state = Column(Unicode(80), nullable=False)
    zip = Column(Unicode(80), nullable=False)
    phone = Column(Unicode(80), nullable=False)

client_account.jinja2:

<form method="post">
{{ fs.render()|safe }}
<input type="submit" value="save">
</form>

There is a minor problem with validation with FormEncode that we’re still working with dealing with validation on a form that has been bound.

webhelpers – Most webhelpers appear to work fine. It was refreshing to see that many of the webhelpers.constants work fine without having to swap the order of the tuples with Deform/Formish. Currently, Flash and Paginate are broken as mentioned above, but, those will be fixed relatively quickly.

Authentication is built in. While the permissions system is quite well thought out, getting it to work in a basic fashion required quite a bit of tweaking. Basically:

__init__.py:

from pyramid.authentication import AuthTktAuthenticationPolicy
from pyramid.authorization import ACLAuthorizationPolicy

    authn_policy = AuthTktAuthenticationPolicy(
        'sosecret', callback=groupfinder)
    authz_policy = ACLAuthorizationPolicy()
    config = Configurator(settings=settings,
                          root_factory='atg.models.RootFactory',
                          authentication_policy=authn_policy,
                          authorization_policy=authz_policy)

    config.add_route('login', '/login',
                     view='atg.login.login',
                     view_renderer='atg:templates/login.pt')
    config.add_route('admin2', '/admin/', view='atg.admin.index', view_permission='edit', 
                     view_renderer='admin_index.jinja2')
    config.add_handler('admin', '/admin/:action', handler=Admin, permission='edit')

login.py:

from pyramid.httpexceptions import HTTPFound
from pyramid.security import remember
from pyramid.security import forget
from pyramid.url import route_url

from atg.security import USERS

def login(request):
    login_url = route_url('login', request)
    referrer = request.url
    if referrer == login_url:
        referrer = '/' # never use the login form itself as came_from
    came_from = request.params.get('came_from', referrer)
    message = ''
    login = ''
    password = ''
    if 'form.submitted' in request.params:
        login = request.params['login']
        password = request.params['password']
        if USERS.get(login) == password:
            headers = remember(request, login)
            return HTTPFound(location = came_from,
                             headers = headers)
        message = 'Failed login'

    return dict(
        message = message,
        url = request.application_url + '/login',
        came_from = came_from,
        login = login,
        password = password,
        )
    
def logout(request):
    headers = forget(request)
    return HTTPFound(location = route_url('view_wiki', request),
                     headers = headers)

security.py:

USERS = {'editor':'editor',
          'viewer':'viewer'}
GROUPS = {'editor':['group:editors']}

def groupfinder(userid, request):
    if userid in USERS:
        return GROUPS.get(userid, [])

templates/login.pt:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html
    xmlns="http://www.w3.org/1999/xhtml"
    xmlns:tal="http://xml.zope.org/namespaces/tal">

<head>
  <meta content="text/html; charset=utf-8" http-equiv="Content-Type"/>
  <title>bfg tutorial wiki (based on TurboGears 20-Minute Wiki)</title>
  <link rel="stylesheet" type="text/css"
        href="${request.application_url}/static/style.css" />
</head>

<body>

<h1>Log In</h1>

<div tal:replace="message"/>

<div class="main_content">
  <form action="${url}" method="post">
    <input type="hidden" name="came_from" value="${came_from}"/>
    <input type="text" name="login" value="${login}"/>
    <br/>
    <input type="password" name="password" value="${password}"/>
    <br/>
    <input type="submit" name="form.submitted" value="Log In"/>
  </form>
</div>  

</body>
</html>

I’ve not gotten the 401/403 error to prompt for a login, but, I believe that is something minor that I’m missing. In addition, I need to modify the schema to use an SQL backend.

All in all, transitioning the code took a bit longer than anticipated and I received some great advice and help from Ben Bangert and Chris McDonough on IRC. I tried to keep my questions to a minimum, but, transitioning from Pylons to Pyramid is going to be harder than moving from repoze.bfg to Pyramid.

Overall, I feel pretty good about the move. I had been a TurboGears users since 2.0-beta and moved over to Pylons for most development a few years back. After having spent 16-20 hours working with Pyramid, I believe that Pyramid is a step in the right direction. In addition to moving this app from Pylons to Pyramid, we switched from Mako to Jinja which required some rewrites of the templates. Mako is still used for FormAlchemy and is loaded in our Pyramid installation.

I don’t really see anything that would make me consider using another framework and I think it is a step in the right direction. Quick apachebench tests show some performance improvements which is also a nice benefit. While the software has an Alpha designation, from a stability standpoint, I’d say it performs more like a Release Candidate but the Alpha designation is probably maintained so that API changes can be pushed as necessary.

To the entire Pylons and Repoze.bfg teams, I say Congratulations! I see some great possibilities on the horizon with a larger community behind Pyramid.

* Chris McDonough helped clarify this (tweet)

After working through quite a bit of the documentation on the pylons site and the repoze site, I didn’t really find a step by step guide to get repoze.who/repoze.what working with Pylons.

Some of the references used:

* http://code.gustavonarea.net/repoze.what-pylons/Manual/Protecting.html
* http://wiki.pylonshq.com/display/pylonscookbook/Authorization+with+repoze.what

Thanks to my nephew Qwait for overloading ActionProtector to intercept 403s and give an additional chance to authenticate.

wget http://pylonshq.com/download/1.0/go-pylons.py
python go-pylons.py --no-site-packages pylons
cd pylons
source bin/activate
easy_install repoze.what-quickstart
easy_install repoze.what-pylons
easy_install mysql-python
paster create -t pylons project
cd project

Because of the association table and the cascade, you must use MySQL or Postgresql rather than SQLite. You also might need to modify development.ini [server:main] host/port.

cd project

config/middleware.py:

    # CUSTOM MIDDLEWARE HERE (filtered by error handling middlewares)
    from project.lib.auth import add_auth
    app = add_auth(app, config)

config/routing.py:

    map.connect('/login', controller='login', action='login')
    map.connect('/login/submit', controller='login', action='login_handler')
    map.connect('/login/continue', controller='login', action='post_login')
    map.connect('/logout/continue', controller='login', action='post_logout')
    map.connect('/logout', controller='login', action='logout_handler')

lib/auth.py: – modified to intercept 403 and provide chance to authenticate

from pylons import response, url
from pylons.controllers.util import redirect

from repoze.what.plugins.quickstart import setup_sql_auth
from repoze.what.plugins import pylonshq

import project.lib.helpers as h

from project.model.meta import Session
from project.model.auth import AuthUser, AuthGroup, AuthPermission

def add_auth(app, config):
   return setup_sql_auth(
       app, AuthUser, AuthGroup, AuthPermission, Session,
       login_handler = '/login/submit',
       logout_handler = '/logout',
       post_login_url = '/login/continue',
       post_logout_url = '/logout/continue',
       cookie_secret = 'my_secret_word',
       translations = {
           'user_name' : 'username',
           'groups' : 'auth_groups',
           'group_name' : 'name',
           'permissions' : 'auth_permissions',
           'permission_name' : 'name'
       }
   )

def redirect_auth_denial(reason):
    if response.status_int == 401:
        message = 'You are not logged in.'
        message_type = 'warning'
    else:
        message = 'You do not have the permissions to access this page.'
        message_type = 'error'

    h.flash(message, message_type)
    redirect(url('/login', came_from=url.current()))

class ActionProtector(pylonshq.ActionProtector):
    default_denial_handler = staticmethod(redirect_auth_denial)

model/auth.py:

from sqlalchemy import *
from sqlalchemy.databases import mysql
from sqlalchemy.orm import relation, backref, synonym
from sqlalchemy.orm.exc import NoResultFound

from project.model.meta import Base

import os
from hashlib import sha1
from datetime import datetime

group_permission_table = Table('auth_group_permissions', Base.metadata,
    Column('group_id', mysql.BIGINT(20, unsigned=True), ForeignKey('auth_groups.id', onupdate='CASCADE', ondelete='CASCADE')),
    Column('permission_id', mysql.BIGINT(20, unsigned=True), ForeignKey('auth_permissions.id', onupdate='CASCADE', ondelete='CASCADE'))
)
user_group_table = Table('auth_user_groups', Base.metadata,
    Column('user_id', mysql.BIGINT(20, unsigned=True), ForeignKey('auth_users.id', onupdate='CASCADE', ondelete='CASCADE')),
    Column('group_id', mysql.BIGINT(20, unsigned=True), ForeignKey('auth_groups.id', onupdate='CASCADE', ondelete='CASCADE'))
)

class AuthGroup(Base):
    __tablename__ = 'auth_groups'

    id = Column(mysql.BIGINT(20, unsigned=True), primary_key=True, autoincrement=True)
    name = Column(Unicode(80), unique=True, nullable=False)
    created = Column(mysql.DATE())

    users = relation('AuthUser', secondary=user_group_table, backref='auth_groups')

    def __repr__(self):
        return '<group: name=%s>' % self.name

    def __unicode__(self):
        return self.name

class AuthUser(Base):
    __tablename__ = 'auth_users'

    id = Column(mysql.BIGINT(20, unsigned=True), primary_key=True, autoincrement=True)
    username = Column(Unicode(80), nullable=False)
    _password = Column('password', Unicode(80), nullable=False)
 
    @property
    def permissions(self):
        perms = set()
        for g in self.groups:
            perms = perms | set(g.permissions)
        return perms

    def _set_password(self, password):
        hashed_password = password

        if isinstance(password, unicode):
            password_8bit = password.encode('UTF-8')
        else:
            password_8bit = password

        salt = sha1()
        salt.update(os.urandom(60))
        hash = sha1()
        hash.update(password_8bit + salt.hexdigest())
        hashed_password = salt.hexdigest() + hash.hexdigest()

        if not isinstance(hashed_password, unicode):
            hashed_password = hashed_password.decode('UTF-8')
        self._password = hashed_password

    def _get_password(self):
        return self._password

    password = synonym('_password', descriptor=property(_get_password, _set_password))

    def validate_password(self, password):
        hashed_pass = sha1()
        hashed_pass.update(password + self.password[:40])
        return self.password[40:] == hashed_pass.hexdigest()

    def __repr__(self):
        return '<user: id="%s" username="%s" email="%s">' % (self.id, self.username, self.email)

    def __unicode__(self):
        return self.username

class AuthPermission(Base):
    __tablename__ = 'auth_permissions'

    id = Column(mysql.BIGINT(20, unsigned=True), primary_key=True, autoincrement=True)
    name = Column(Unicode(80), unique=True, nullable=False)
    description = Column(mysql.TEXT())

    groups = relation(AuthGroup, secondary=group_permission_table, backref='auth_permissions')

    def __unicode__(self):
        return self.permission_name

controllers/login.py:

from pylons import request, response, session, tmpl_context, config, url
from pylons.controllers.util import redirect

from project.lib.base import BaseController, render
from project.lib.helpers import flash

class LoginController(BaseController):
    def login(self):
        login_counter = request.environ['repoze.who.logins']
        if login_counter > 0:
            flash('Wrong credentials')
        tmpl_context.login_counter = login_counter
        tmpl_context.came_from = request.params.get('came_from') or url('/')
        return render('login.mako')

    def login_handler(self):
        pass

    def post_login(self):
        identity = request.environ.get('repoze.who.identity')
        came_from = str(request.params.get('came_from', '')) or url('/')
        if not identity:
            login_counter = request.environ['repoze.who.logins'] + 1
            redirect(url('/login', came_from=came_from, __logins=login_counter))
        redirect(came_from)

    def logout_handler(self):
        pass
     
    def post_logout(self):
        redirect('/')

templates/login.mako:

<% messages = h.flash.pop_messages() %>
% if messages:
<div class="flash">
    % for message in messages:
      <p class="${message.category}">${message}
    % endfor
</div>
% endif

  <form action="${h.url('/login/submit', came_from=tmpl_context.came_from, __logins=tmpl_context.login_counter)}" method="POST">
    <label for="login">Username:<input type="text" id="login" name="login" /><br />
    <label for="password">Password:<input type="password" id="password" name="password" />
    <input type="submit" value="Login" />
  </form>

controllers/root.py:

from pylons import request, response, session, tmpl_context, config

#from repoze.what.plugins.pylonshq import ActionProtector, ControllerProtector
from project.lib.auth import ActionProtector
from repoze.what.predicates import is_user, has_permission, in_group

from project.lib.base import BaseController, render

class RootController(BaseController):
    def index(self):
        return render('index.mako')

    @ActionProtector(is_user('test'))
    def user(self):
        return render('loggedin.mako')

    @ActionProtector(is_user('nottest'))
    def notuser(self):
        return render('loggedin.mako')

    @ActionProtector(in_group('admin'))
    def admin(self):
        return render('loggedin.mako')

    @ActionProtector(has_permission('edit'))
    def edit(self):
        return render('loggedin.mako')

lib/helpers.py:

from pylons import url

from webhelpers.pylonslib import Flash as _Flash
flash = _Flash()

websetup.py: – after the Session, Base import:

from project.model.auth import *

Setup/Create the database, start paster

paster setup-app development.ini
paster serve --reload development.ini

createuser.py:

#!/usr/bin/python2.6

from sqlalchemy import create_engine
engine = create_engine('mysql://user:pass@localhost/dbname', echo=True)
from sqlalchemy.orm import sessionmaker
Session = sessionmaker(bind=engine)
session = Session()

from project.model.auth import *

u = AuthUser()
u.username = u'test'
u.password = u'test'
session.add(u)
g = AuthGroup()
g.name = u'admin'
g.users.append(u)
session.add(g)
p = AuthPermission()
p.name = u'edit'
p.groups.append(g)
session.add(p)
session.commit()

While I spent quite a bit of time deciphering the Graph API documentation and the OAuth guides that Facebook puts forth and submitted three documentation fixes for examples that call non-existent parameters and consequently don’t work, I came to the realization that my original layout really only works if you use a single Pylons instance per Facebook application. Since we’re focused on Don’t Repeat Yourself (DRY) Principles, some thought needs to go into things.

First, our platform needs to be designed. For this set of projects we’re going to use Nginx with uwsgi. Since we’re serving static content, we’re going to set up our directories on Nginx to allow that content to be served outside our Pylons virtual environment. Tony Landis was one of the first to provide an implementation guide for uwsgi with Nginx for Pylons which provided some of the information needed to get things working.

Our theoretical layout looks like the following:

/webroot
  |--- /static
  |--- /fb
/virtualenv
  |--- /fbappone
  |--- /fbapptwo

Later we’ll add a CDN that does origin pulls from /webroot/static. This application would have worked wonderfully with Varnish and ESI if the ESI could be compressed, but, setting up Nginx -> Varnish -> Nginx -> uwsgi seemed somewhat inefficient just to add compression. The Facebook application we’ve developed is an IFrame canvas which took roughly fifteen hours to debug after the original concept was decided. The majority of that time was spent dealing with the IFrame canvas issues. FBML was much easier to get working properly.

What we end up with is a url structure like:

http://basedomain.com/
     /static/ (xd_receiver.html, jquery support modules, CSS files)
     /fb/ (Generic facebook files, support, tos, help)
     /(fbapp)/application_one/
     /(fbapp)/application_two/

As a result of this structure, we don’t need to manipulate config/routing.py as the default’s set by Pylons map things the way we want. In the /static/ directory, we can put our CSS, js and static media files. Remember to minify the CSS and js files and combine them if possible.

Our nginx config looks like:

server {
    listen   1.2.3.4:80;
    server_name  xxxxxx.com;
    access_log /var/log/nginx/xxxxxx.com-access.log;

    location ~* (css|js|png|jpe?g|gif|ico|swf|flv)$ {
        expires max;
    }

    gzip on;
    gzip_min_length 500;
    gzip_types text/plain application/xml text/html text/javascript;
    gzip_disable "MSIE [1-6]\.";

    location ^~ /static/ {
    	alias   /var/www/xxxxxx.com/static/;
    }
    location ^~ /fb/ {
    	alias   /var/www/xxxxxx.com/fb/;
    }
    location / {
        uwsgi_pass  unix:/tmp/uwsgi.sock;
        include     uwsgi_params;
    }
}

We could modify the nginx config to pull / from the static page, but, we’re actually capturing that with a root controller that knows what applications reside below it as a directory of sorts.

We used Debian which doesn’t support uwsgi yet. A brief set of instructions follows which should work on any Debian based distribution as well:

apt-get install libxml2-dev dpkg-dev debhelper
cd /usr/src
apt-get source nginx
wget http://projects.unbit.it/downloads/uwsgi-0.9.4.4.tar.gz
tar xzf uwsgi-0.9.4.4.tar.gz
cd nginx
vi debian/rules
  add:  --add-module=/usr/src/uwsgi-0.9.4.4/nginx/ \
dpkg-buildpackage
dpkg -i ../nginx_0.7.65-5_i386.deb
mkdir /usr/local/nginx/
cp /usr/src/uwsgi-0.9.4.4/nginx/uwsgi_params /etc/nginx

/etc/nginx/uwsgi_params, add:

uwsgi_param  SCRIPT_NAME        /;

Note: I had problems with 0.9.5.1 and paster enabled wsgi applications which caused issues with Pylons.

Our uwsgi command line for development:

/usr/src/uwsgi-0.9.4.4/uwsgi -s /tmp/uwsgi.sock -C -iH /var/www/facebook/ --paste config:/var/www/facebook/fpapp/development.ini

One of the things that made Facebook integration difficult was somewhat incomplete documentation or even incorrect documentation on Facebook’s site. While the Graph API is new, it is quite a bit more powerful. While they do have official support, I think I’ll use velruse for OAuth integration next time and use the Python-SDK for the Graph API integration. See my previous post on using Pylons for a Facebook Application for a little more detailed information on how to get the application working.