Posts Tagged ‘repoze’

A different approach to Email Validation

Monday, December 13th, 2010

I ran into an issue with a library that did email validation. One of the client’s email addresses didn’t validate and while debugging the code, I realized that more than just that client’s email address could be rejected. While using a regexp is not a failsafe solution to email address validation, it does attempt to make sure you are getting a reasonably good email address.

However, the regexp can still fail and in some cases, you get a client that is turned away. The only true way to validate an email address is valid at least once is to require a client to receive an email and validate their account. With so many throwaway account services, you can never be assured that anything past that first email address will ever be received. However, to give you the best chance at getting a valid email address, you need to do some validation.

There are a number of methods for validating email addresses on the internet and most fail somewhere. Do you accept emails with ” or < > in them? You might, since some email clients paste that information when you copy an email address. Or, do you accept the bare minimum email address, accepting the fact that people could use -, +, . or even ‘ in the name portion? Trying to get the person to enter a name that is reasonably valid is the goal.

One could have a very generic rule that looks for an @ with a . after it and some characters. That would ensure almost the largest cross-section of potential email addresses but we may have cast too wide a net. We could go with a very strict regexp but potentially miss out on some corner cases.

As we can see, the error message presented here is correct, the email address we’ve entered is not correct:

But, what if we do put in a valid email address:

Our regexp fails to recognize the email address even though it is valid. Of course, this requires someone to adjust some code and we’re fine until the next email address that fails our test is entered.

What if we introduce a new error condition, a soft fail. If the address passes our very simple test and looks to have a valid email address construction, we’ll put another field asking the user to check to see if any typos were made. Users that pass our more stringent regexp wouldn’t be subjected to the soft fail error handling.

When a user is subjected to the soft fail, our software can log the email address so we can adjust our validation rules, but, we’re not restricting a client from being able to sign up because their email address didn’t fit the regexp that we’re using.

While I feel reasonably confident with some of the regexp’s I use, email validation has always been one of the most troublesome portions of any system we’ve designed.

For the above test, we used Deform/Colander and Pyramid.

Using Pyramid with Deform to Edit/Create records in SQLAlchemy backed database

Monday, November 8th, 2010

While working with Pyramid it was worth taking a good look at deform. During my brief look before, the one thing that appeared to be missing was a method to quickly integrate with SQLAlchemy backed data. For a handful of forms, manually creating the appstruct to be passed to the Form wouldn’t be difficult, but, for some more significant applications, it could be quite cumbersome. The following is a one page application that allows you to create a new user, or, edit an existing user by specifying the userid in the URL. The two pieces that do the real work are record_to_appstruct and merge_session_with_post. record_to_appstruct takes the class returned from SQLAlchemy and converts it to an appstruct that deform likes. Once we get the validated data, we merge the record with the post items using merge_session_with_post, merge the record and present a screen with the post data.

This is more a proof of concept, but, works well enough that we were able to convert quite a few forms and work more closely with Deform and Pyramid as we do more development.


from pyramid.configuration import Configurator
from paste.deploy.converters import asbool

from defo.models import initialize_sql

def app(global_config, **settings):
    """ This function returns a WSGI application.
    It is usually called by the PasteDeploy framework during 
    ``paster serve``.
    db_string = settings.get('db_string')
    if db_string is None:
        raise ValueError("No 'db_string' value in application configuration.")
    db_echo = settings.get('db_echo', 'false')
    initialize_sql(db_string, asbool(db_echo))
    config = Configurator(settings=settings)
    config.add_static_view('static', 'defo:static')
    config.add_route('home', '/', view='defo.views.edit',
    config.add_route('homeid', '/:id', view='defo.views.edit',
    return config.make_wsgi_app()

from defo.models import DBSession
from defo.models import MyModel

from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()

from sqlalchemy import *
from sqlalchemy.databases import mysql
from sqlalchemy.orm import relation, backref, synonym
from sqlalchemy.orm.exc import NoResultFound

import deform
import colander
from webhelpers import constants

class UserSchema(colander.Schema):
    username = colander.SchemaNode(colander.String())
    contact = colander.SchemaNode(colander.String())
    email = colander.SchemaNode(colander.String())
    company = colander.SchemaNode(colander.String())
    addr1 = colander.SchemaNode(colander.String())
    addr2 = colander.SchemaNode(colander.String(), missing=u'',)
    city = colander.SchemaNode(colander.String())
    state = colander.SchemaNode(colander.String())
    zip = colander.SchemaNode(colander.String())
    country = colander.SchemaNode(
        widget = deform.widget.SelectWidget(values=constants.country_codes()),
    phone = colander.SchemaNode(colander.String())

class AuthUser(Base):
    __tablename__ = 'auth_users'

    id = Column(mysql.BIGINT(20, unsigned=True), primary_key=True, autoincrement=True)
    username = Column(Unicode(80), nullable=False)
    email = Column(Unicode(80), nullable=False)
    contact = Column(Unicode(80), nullable=False)
    company = Column(Unicode(80), nullable=False)
    addr1 = Column(Unicode(80), nullable=False)
    addr2 = Column(Unicode(80))
    city = Column(Unicode(80), nullable=False)
    state = Column(Unicode(80), nullable=False)
    zip = Column(Unicode(80), nullable=False)
    country = Column(Unicode(80), nullable=False)
    phone = Column(Unicode(80), nullable=False)

def record_to_appstruct(self):
    return dict([(k, self.__dict__[k]) for k in sorted(self.__dict__) if '_sa_' != k[:4]])

def merge_session_with_post(session, post):
    for key,value in post:
        setattr(session, key, value)
    return session

def edit(request):
    dbsession = DBSession()
    if request.matchdict.has_key('id'):
        record = dbsession.query(AuthUser). \
        record = AuthUser()

    schema = UserSchema()
    form = deform.Form(schema, buttons=('submit',))
    if request.POST:
            appstruct = form.validate(request.POST.items())
        except deform.ValidationFailure, e:
            return {'form':e.render()}
        record = merge_session_with_post(record, request.POST.items())
        return {'formdata':appstruct}
        appstruct = record_to_appstruct(record)
    return {'form':form.render(appstruct=appstruct)}


    Deform Demo Site
  <!-- Meta Tags -->
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  <!-- CSS -->
  <link rel="stylesheet" href="/static/css/form.css" type="text/css" />
  <link rel="stylesheet" href="/static/css/theme.css" type="text/css" />
  <!-- JavaScript -->
  <script type="text/javascript"
  <script type="text/javascript"
this is a test template in mako
% if form:
% else:
We got our form data: ${formdata}
% endif
<script type="text/javascript">

Pylons 1.0 to Pyramid 1.0a1

Sunday, November 7th, 2010

On Nov 4/Nov 5 a rumor that was later substantiated regarding Pylons surfaced. While the initial message was regarding TurboGears which is based on Pylons, the rumor was that Pylons was being rolled into another framework. Ben Bangert issued a post that gave some of the reasoning behind the merger of repoze.bfg and Pylons.

Since we’ve been working on an application for the last few weeks that isn’t in production, it seemed like an ideal test case. While the documentation for Pyramid is superb, transitioning from Pylons to Pyramid still has a few rough edges as the terminology that Pylons developers are used to were changed.

One of the first issues is the way that Pylons handles routes and converting to Pyramid. In Pylons, default route entries are set in config/


To emulate that behavior in Pyramid, modify

    config.add_handler('client', '/client/:action', handler=Client)
    config.add_handler('clientid', '/client/:action/:id', handler=Client)

Pyramid doesn’t scan controllers, so, if you have multiple controllers, you’ll need to specify each. Also, make sure that you use a unique name (client|clientid) for each handler that you’ve added to avoid any 404s. Rather than the old controller structure you had, your code is now considerably cleaner and looks like:

from pyramid.response import Response
from pyramid.view import action

class Client(object):
    def __init__(self, request):
        self.request = request
        self.dbsession = DBSession()

    def index(self):
        return {'views':5, 'clicks':1}

The decorator signifies the template that you want to use and variables that you want to pass to the template are returned. This ends most of the tmpl_context. or c. clutter that was present in controllers and templates.

More documentation on handlers is available here.

An early version of the pyramid templates does not contain the weberror helper that was available in Pylons. The Pylons paster templates that are included in Pyramid do have the helper. A discussion with Chris McDonough should result in the changes being made to the Pyramid templates as well.

Another area that needs attention is Flash messages. Currently they are not supported and webhelpers.flash appears to implement things in a manner that won’t work with Pyramid. Through the subscribers method in Pyramid, it looks to be somewhat trivial to implement. The existing Pylons template in Pyramid does contain passthroughs of the c./tmpl_context. globals and helper modules that were available in Pylons. Modifying that slightly should allow Flash messages to be easily enabled. If you are going to transition, it would make more sense to use one of the Pylons templates than to migrate straight to the Pyramid templates.

For reference, the file:
mentions the existing global handling of the special objects for which Flash messages can probably be reimplemented.

SQLAlchemy is supported, but, there is an additional extension loaded which appears to do an autocommit on SQL queries. A brief readthrough mentions that you need to use s.join() to join your two database queries to be handled. It appears that you can join two db handles on separate databases which makes this a bit more powerful than using normal transactions as you could ensure a record was written to mongoDB and MySQL. I need to spend a little more time reading through this.

pagination appears to depend on routes, and even with routes installed, an error is thrown with thread._local requiring a mapper which is probably not going to work with Pyramid and will require some rewriting. This appears to be the same issue (thread._local) with Flash messages and it was mentioned that both items were relatively high priority and easy fixes.

forms – By default, formish is installed. While their site was down, it was stated that formish was in no way an endorsement, it was just included as it was part of bfg and is not a dependency in Pyramid*. Some preliminary work with FormAlchemy showed that it should work without too much difficulty, but, I decided to give Deform a try. Since I had already looked at Deform in the past for Pylons, and the screencast demonstration was done in Pylons, I was somewhat familiar with the methods. Converting over to Deform was a matter of reworking a few schemas. I’ve had some difficulties getting Deform to work with output from SQLAlchemy. Basically, Deform works very well with ZODB which is a schemaless database. Using it to edit rows returned from SQLAlchemy requires one to manually iterate through the returned row to create an appstruct to hand to Deform. For a number of simple forms this probably wouldn’t be difficult. As our project has a number of GridSets, converting over to Deform would have been considerably more difficult. To get FormAlchemy to work, we required the following changes:

from mako.template import Template

from formalchemy import config as fa_config
from formalchemy import templates
from formalchemy import validators
from formalchemy import fields
from formalchemy import forms
from formalchemy import tables
from formalchemy.ext.fsblob import FileFieldRenderer
from formalchemy.ext.fsblob import ImageFieldRenderer

fa_config.encoding = 'utf-8'

class TemplateEngine(templates.TemplateEngine):
    def render(self, name, **kwargs):
        return Template(filename='/var/www/pyr/atg/atg/templates/forms/%s.mako' % name).render(**kwargs)

fa_config.engine = TemplateEngine()

class FieldSet(forms.FieldSet):

class Grid(tables.Grid):

our file using formalchemy:

from pyramid.httpexceptions import HTTPRedirection

from formalchemy import validators
from formalchemy.fields import Field
from atg.form import FieldSet

dbsession = DBSession()

User = FieldSet(auth.AuthUser, session=dbsession)
    include = [,,,

    def account(self):
        record = self.dbsession.query(auth.AuthUser).filter(
        fs = User.bind(record, data=self.request.POST or None)
        if self.request.POST and fs.validate():   
        return {'fs':fs}

model from

class AuthUser(Base):
    __tablename__ = 'auth_users'

    id = Column(mysql.BIGINT(20, unsigned=True), primary_key=True, autoincrement
    username = Column(Unicode(80), nullable=False)
    _password = Column('password', Unicode(80), nullable=False)
    email = Column(Unicode(80), nullable=False)
    contact = Column(Unicode(80), nullable=False)
    company = Column(Unicode(80), nullable=False)
    addr1 = Column(Unicode(80), nullable=False)
    addr2 = Column(Unicode(80))
    city = Column(Unicode(80), nullable=False)
    state = Column(Unicode(80), nullable=False)
    zip = Column(Unicode(80), nullable=False)
    phone = Column(Unicode(80), nullable=False)


<form method="post">
{{ fs.render()|safe }}
<input type="submit" value="save">

There is a minor problem with validation with FormEncode that we’re still working with dealing with validation on a form that has been bound.

webhelpers – Most webhelpers appear to work fine. It was refreshing to see that many of the webhelpers.constants work fine without having to swap the order of the tuples with Deform/Formish. Currently, Flash and Paginate are broken as mentioned above, but, those will be fixed relatively quickly.

Authentication is built in. While the permissions system is quite well thought out, getting it to work in a basic fashion required quite a bit of tweaking. Basically:

from pyramid.authentication import AuthTktAuthenticationPolicy
from pyramid.authorization import ACLAuthorizationPolicy

    authn_policy = AuthTktAuthenticationPolicy(
        'sosecret', callback=groupfinder)
    authz_policy = ACLAuthorizationPolicy()
    config = Configurator(settings=settings,

    config.add_route('login', '/login',
    config.add_route('admin2', '/admin/', view='atg.admin.index', view_permission='edit', 
    config.add_handler('admin', '/admin/:action', handler=Admin, permission='edit')

from pyramid.httpexceptions import HTTPFound
from import remember
from import forget
from pyramid.url import route_url

from import USERS

def login(request):
    login_url = route_url('login', request)
    referrer = request.url
    if referrer == login_url:
        referrer = '/' # never use the login form itself as came_from
    came_from = request.params.get('came_from', referrer)
    message = ''
    login = ''
    password = ''
    if 'form.submitted' in request.params:
        login = request.params['login']
        password = request.params['password']
        if USERS.get(login) == password:
            headers = remember(request, login)
            return HTTPFound(location = came_from,
                             headers = headers)
        message = 'Failed login'

    return dict(
        message = message,
        url = request.application_url + '/login',
        came_from = came_from,
        login = login,
        password = password,
def logout(request):
    headers = forget(request)
    return HTTPFound(location = route_url('view_wiki', request),
                     headers = headers)

USERS = {'editor':'editor',
GROUPS = {'editor':['group:editors']}

def groupfinder(userid, request):
    if userid in USERS:
        return GROUPS.get(userid, [])


<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"

  <meta content="text/html; charset=utf-8" http-equiv="Content-Type"/>
  <title>bfg tutorial wiki (based on TurboGears 20-Minute Wiki)</title>
  <link rel="stylesheet" type="text/css"
        href="${request.application_url}/static/style.css" />


<h1>Log In</h1>

<div tal:replace="message"/>

<div class="main_content">
  <form action="${url}" method="post">
    <input type="hidden" name="came_from" value="${came_from}"/>
    <input type="text" name="login" value="${login}"/>
    <input type="password" name="password" value="${password}"/>
    <input type="submit" name="form.submitted" value="Log In"/>


I’ve not gotten the 401/403 error to prompt for a login, but, I believe that is something minor that I’m missing. In addition, I need to modify the schema to use an SQL backend.

All in all, transitioning the code took a bit longer than anticipated and I received some great advice and help from Ben Bangert and Chris McDonough on IRC. I tried to keep my questions to a minimum, but, transitioning from Pylons to Pyramid is going to be harder than moving from repoze.bfg to Pyramid.

Overall, I feel pretty good about the move. I had been a TurboGears users since 2.0-beta and moved over to Pylons for most development a few years back. After having spent 16-20 hours working with Pyramid, I believe that Pyramid is a step in the right direction. In addition to moving this app from Pylons to Pyramid, we switched from Mako to Jinja which required some rewrites of the templates. Mako is still used for FormAlchemy and is loaded in our Pyramid installation.

I don’t really see anything that would make me consider using another framework and I think it is a step in the right direction. Quick apachebench tests show some performance improvements which is also a nice benefit. While the software has an Alpha designation, from a stability standpoint, I’d say it performs more like a Release Candidate but the Alpha designation is probably maintained so that API changes can be pushed as necessary.

To the entire Pylons and Repoze.bfg teams, I say Congratulations! I see some great possibilities on the horizon with a larger community behind Pyramid.

* Chris McDonough helped clarify this (tweet)

Repoze.who/Repoze.what with Pylons (step by step)

Friday, October 29th, 2010

After working through quite a bit of the documentation on the pylons site and the repoze site, I didn’t really find a step by step guide to get repoze.who/repoze.what working with Pylons.

Some of the references used:


Thanks to my nephew Qwait for overloading ActionProtector to intercept 403s and give an additional chance to authenticate.

python --no-site-packages pylons
cd pylons
source bin/activate
easy_install repoze.what-quickstart
easy_install repoze.what-pylons
easy_install mysql-python
paster create -t pylons project
cd project

Because of the association table and the cascade, you must use MySQL or Postgresql rather than SQLite. You also might need to modify development.ini [server:main] host/port.

cd project


    # CUSTOM MIDDLEWARE HERE (filtered by error handling middlewares)
    from project.lib.auth import add_auth
    app = add_auth(app, config)


    map.connect('/login', controller='login', action='login')
    map.connect('/login/submit', controller='login', action='login_handler')
    map.connect('/login/continue', controller='login', action='post_login')
    map.connect('/logout/continue', controller='login', action='post_logout')
    map.connect('/logout', controller='login', action='logout_handler')

lib/ – modified to intercept 403 and provide chance to authenticate

from pylons import response, url
from pylons.controllers.util import redirect

from repoze.what.plugins.quickstart import setup_sql_auth
from repoze.what.plugins import pylonshq

import project.lib.helpers as h

from project.model.meta import Session
from project.model.auth import AuthUser, AuthGroup, AuthPermission

def add_auth(app, config):
   return setup_sql_auth(
       app, AuthUser, AuthGroup, AuthPermission, Session,
       login_handler = '/login/submit',
       logout_handler = '/logout',
       post_login_url = '/login/continue',
       post_logout_url = '/logout/continue',
       cookie_secret = 'my_secret_word',
       translations = {
           'user_name' : 'username',
           'groups' : 'auth_groups',
           'group_name' : 'name',
           'permissions' : 'auth_permissions',
           'permission_name' : 'name'

def redirect_auth_denial(reason):
    if response.status_int == 401:
        message = 'You are not logged in.'
        message_type = 'warning'
        message = 'You do not have the permissions to access this page.'
        message_type = 'error'

    h.flash(message, message_type)
    redirect(url('/login', came_from=url.current()))

class ActionProtector(pylonshq.ActionProtector):
    default_denial_handler = staticmethod(redirect_auth_denial)


from sqlalchemy import *
from sqlalchemy.databases import mysql
from sqlalchemy.orm import relation, backref, synonym
from sqlalchemy.orm.exc import NoResultFound

from project.model.meta import Base

import os
from hashlib import sha1
from datetime import datetime

group_permission_table = Table('auth_group_permissions', Base.metadata,
    Column('group_id', mysql.BIGINT(20, unsigned=True), ForeignKey('', onupdate='CASCADE', ondelete='CASCADE')),
    Column('permission_id', mysql.BIGINT(20, unsigned=True), ForeignKey('', onupdate='CASCADE', ondelete='CASCADE'))
user_group_table = Table('auth_user_groups', Base.metadata,
    Column('user_id', mysql.BIGINT(20, unsigned=True), ForeignKey('', onupdate='CASCADE', ondelete='CASCADE')),
    Column('group_id', mysql.BIGINT(20, unsigned=True), ForeignKey('', onupdate='CASCADE', ondelete='CASCADE'))

class AuthGroup(Base):
    __tablename__ = 'auth_groups'

    id = Column(mysql.BIGINT(20, unsigned=True), primary_key=True, autoincrement=True)
    name = Column(Unicode(80), unique=True, nullable=False)
    created = Column(mysql.DATE())

    users = relation('AuthUser', secondary=user_group_table, backref='auth_groups')

    def __repr__(self):
        return '<group: name=%s>' %

    def __unicode__(self):

class AuthUser(Base):
    __tablename__ = 'auth_users'

    id = Column(mysql.BIGINT(20, unsigned=True), primary_key=True, autoincrement=True)
    username = Column(Unicode(80), nullable=False)
    _password = Column('password', Unicode(80), nullable=False)
    def permissions(self):
        perms = set()
        for g in self.groups:
            perms = perms | set(g.permissions)
        return perms

    def _set_password(self, password):
        hashed_password = password

        if isinstance(password, unicode):
            password_8bit = password.encode('UTF-8')
            password_8bit = password

        salt = sha1()
        hash = sha1()
        hash.update(password_8bit + salt.hexdigest())
        hashed_password = salt.hexdigest() + hash.hexdigest()

        if not isinstance(hashed_password, unicode):
            hashed_password = hashed_password.decode('UTF-8')
        self._password = hashed_password

    def _get_password(self):
        return self._password

    password = synonym('_password', descriptor=property(_get_password, _set_password))

    def validate_password(self, password):
        hashed_pass = sha1()
        hashed_pass.update(password + self.password[:40])
        return self.password[40:] == hashed_pass.hexdigest()

    def __repr__(self):
        return '<user: id="%s" username="%s" email="%s">' % (, self.username,

    def __unicode__(self):
        return self.username

class AuthPermission(Base):
    __tablename__ = 'auth_permissions'

    id = Column(mysql.BIGINT(20, unsigned=True), primary_key=True, autoincrement=True)
    name = Column(Unicode(80), unique=True, nullable=False)
    description = Column(mysql.TEXT())

    groups = relation(AuthGroup, secondary=group_permission_table, backref='auth_permissions')

    def __unicode__(self):
        return self.permission_name


from pylons import request, response, session, tmpl_context, config, url
from pylons.controllers.util import redirect

from project.lib.base import BaseController, render
from project.lib.helpers import flash

class LoginController(BaseController):
    def login(self):
        login_counter = request.environ['repoze.who.logins']
        if login_counter > 0:
            flash('Wrong credentials')
        tmpl_context.login_counter = login_counter
        tmpl_context.came_from = request.params.get('came_from') or url('/')
        return render('login.mako')

    def login_handler(self):

    def post_login(self):
        identity = request.environ.get('repoze.who.identity')
        came_from = str(request.params.get('came_from', '')) or url('/')
        if not identity:
            login_counter = request.environ['repoze.who.logins'] + 1
            redirect(url('/login', came_from=came_from, __logins=login_counter))

    def logout_handler(self):
    def post_logout(self):


<% messages = h.flash.pop_messages() %>
% if messages:
<div class="flash">
    % for message in messages:
      <p class="${message.category}">${message}
    % endfor
% endif

  <form action="${h.url('/login/submit', came_from=tmpl_context.came_from, __logins=tmpl_context.login_counter)}" method="POST">
    <label for="login">Username:<input type="text" id="login" name="login" /><br />
    <label for="password">Password:<input type="password" id="password" name="password" />
    <input type="submit" value="Login" />


from pylons import request, response, session, tmpl_context, config

#from repoze.what.plugins.pylonshq import ActionProtector, ControllerProtector
from project.lib.auth import ActionProtector
from repoze.what.predicates import is_user, has_permission, in_group

from project.lib.base import BaseController, render

class RootController(BaseController):
    def index(self):
        return render('index.mako')

    def user(self):
        return render('loggedin.mako')

    def notuser(self):
        return render('loggedin.mako')

    def admin(self):
        return render('loggedin.mako')

    def edit(self):
        return render('loggedin.mako')


from pylons import url

from webhelpers.pylonslib import Flash as _Flash
flash = _Flash() – after the Session, Base import:

from project.model.auth import *

Setup/Create the database, start paster

paster setup-app development.ini
paster serve --reload development.ini


from sqlalchemy import create_engine
engine = create_engine('mysql://user:pass@localhost/dbname', echo=True)
from sqlalchemy.orm import sessionmaker
Session = sessionmaker(bind=engine)
session = Session()

from project.model.auth import *

u = AuthUser()
u.username = u'test'
u.password = u'test'
g = AuthGroup() = u'admin'
p = AuthPermission() = u'edit'

Entries (RSS) and Comments (RSS).
Cluster host: li