Archive for the ‘Framework’ Category

A different approach to Email Validation

Monday, December 13th, 2010

I ran into an issue with a library that did email validation. One of the client’s email addresses didn’t validate and while debugging the code, I realized that more than just that client’s email address could be rejected. While using a regexp is not a failsafe solution to email address validation, it does attempt to make sure you are getting a reasonably good email address.

However, the regexp can still fail and in some cases, you get a client that is turned away. The only true way to validate an email address is valid at least once is to require a client to receive an email and validate their account. With so many throwaway account services, you can never be assured that anything past that first email address will ever be received. However, to give you the best chance at getting a valid email address, you need to do some validation.

There are a number of methods for validating email addresses on the internet and most fail somewhere. Do you accept emails with ” or < > in them? You might, since some email clients paste that information when you copy an email address. Or, do you accept the bare minimum email address, accepting the fact that people could use -, +, . or even ‘ in the name portion? Trying to get the person to enter a name that is reasonably valid is the goal.

One could have a very generic rule that looks for an @ with a . after it and some characters. That would ensure almost the largest cross-section of potential email addresses but we may have cast too wide a net. We could go with a very strict regexp but potentially miss out on some corner cases.

As we can see, the error message presented here is correct, the email address we’ve entered is not correct:

But, what if we do put in a valid email address:

Our regexp fails to recognize the email address even though it is valid. Of course, this requires someone to adjust some code and we’re fine until the next email address that fails our test is entered.

What if we introduce a new error condition, a soft fail. If the address passes our very simple test and looks to have a valid email address construction, we’ll put another field asking the user to check to see if any typos were made. Users that pass our more stringent regexp wouldn’t be subjected to the soft fail error handling.

When a user is subjected to the soft fail, our software can log the email address so we can adjust our validation rules, but, we’re not restricting a client from being able to sign up because their email address didn’t fit the regexp that we’re using.

While I feel reasonably confident with some of the regexp’s I use, email validation has always been one of the most troublesome portions of any system we’ve designed.

For the above test, we used Deform/Colander and Pyramid.

Using FormEncode for validation with Colander and Deform

Sunday, December 12th, 2010

While working on a project, I ran across a number of emails that didn’t properly validate using Colander. Digging into Colander’s code, the regexp used was rather basic. Chris McDonough confirmed this and said he would welcome a newer regexp.

However, FormEncode’s email validation also allows one to optionally check the DNS to see if there is a valid A or MX record – validation that might be handy. Additionally, we need to do credit card verification which is not currently included in Colander. The quick solution is to use FormEncode’s validation routines through Colander.

Our Form Schema:

def email_validate(address):
        valid = formencode.validators.Email().to_python(address)
        return True
    except Exception as message:
        return unicode(message)
class EmailTestSchema(colander.Schema):
    email = colander.SchemaNode(colander.String(),
        widget = deform.widget.TextInputWidget(size=60),
        validator = colander.Function(email_validate),

Our view:

    def test(self):
        schema = EmailTestSchema()
        form = deform.Form(schema, buttons=('Add Email',))
        if self.request.POST:
                appstruct = form.validate(self.request.POST.items())
            except deform.ValidationFailure, e:
                return {'form':e.render()}

        return {'form':form.render()}

Now, our email validation uses FormEncode which contains some fairly detailed error messages which are passed through Colander.

After more coding, I’ll post another solution that might answer some other issues I ran into with email validation.

Diagnosing errors in Pylons 1.0 to Pyramid 1.0a1+ Transition

Friday, November 26th, 2010

My first attempt at migrating a Pylons project to Pyramid was accomplished without too much difficulty. That project was relatively small, however, this project hasn’t been put into production.

A brief history of this application:

We have a legacy PHP application which consists of 78k lines of code. Originally we started rewriting the application in Turbogears 2.0 and later moved over to Pylons with ToscaWidgets. We used ToscaWidgets because all of our forms had been written for TG2. The application never made it into production, but, was substantial enough that we felt it would be a good test to convert it to Pyramid since we’re focusing our development efforts on Pyramid.

Initially I started to write a script to do the migration from Pylons to Pyramid and from ToscaWidgets to Deform but abandoned that. Changing the controllers from Pylons to Pyramid was fairly easy.

Most of the pylons imports were commented out and replaced with:

import deform
import colander

from pyramid.response import Response
from pyramid.view import action
from import authenticated_userid
from pyramid.threadlocal import get_current_request
import webhelpers.paginate as paginate

References to tmpl_context. were altered, our __before__ action for authkit was removed and replaced with an __init__ since we were using handlers. Routes were modified and given unique names, templates were modified to remove references to ${h. and ${tmpl_context., and we began the process of stepping through the application. Initially I had written a script to convert the ToscaWidgets form models over to Deform schemas, but, after two hours, it became obvious it would take more time to do that than to manually recreate the forms. After working through much of the process, I’m debating whether this project should have been migrated over to FormAlchemy as almost every form is a duplicate of the SQL schema. Since it isn’t in production, we still have some time to make that decision.

What follows is a summary of the errors received and what caused the errors.

TypeError: ‘NoneType’ object is not iterable

I ran into this while importing some routes from an existing project. Since routes must be uniquely named in Pyramid, a route with the same name will replace a prior route. Some of the route names have gotten quite unwieldy.

TypeError: object.__new__() takes no parameters

Solution, add the __init__ block to your class in your handler.

class YourClass(object):
    def __init__(self, request):
        self.request = request

ValueError: Non-response object returned from view named (and no renderer): {‘template’: ‘billing_index’}

Turbogears 2.0 code:

    def index(self, **kw):
        return dict(template='billing_index')

modify to:

    @action(renderer='billing_index.mako', permission='client')
    def index(self, **kw):
        return {}

AttributeError: ‘Undefined’ object has no attribute ‘form’

Pyramid doesn’t pass the tmpl_context., c. or h. through to the template, so, the return values of each of the actions needs to pass the fields required. As a result, any template code that replies on these globals needs to be modified, and the corresponding action needs to return the values in the dictionary. You can use one of the pylons templates which will instantiate a subscriber method to replicate the Pylons globals if you want.

AttributeError: ‘Undefined’ object has no attribute ‘literal’

Existing forms are relying on the pylons h. global.


UnboundLocalError: local variable ‘clients’ referenced before assignment

This occurs when you remove tmpl_context. and are left with a variable assigned that matches the class. For example:

tmpl_context.clients = meta.Session.query(clients).filter_by(client_id==1).all()

When tmpl_context. is removed, the class clients is cast incorrectly.

AttributeError: ‘Undefined’ object has no attribute ‘pager’

Paginate itemset isn’t being passed to the template.

TypeError: ‘NoneType’ object is not iterable

Paginate is getting the result set, rather than the paginate set passed in the return dictionary.

NameError: global name ‘request’ is not defined

request.matchdict/request.params -> self.request.matchdict/self.request.params

In a handler, request. is referred to as self.request. If you’ve converted things over to a handler rather than writing the individual routes for each action, you’ll need to preface any request. with self.

NotImplementedError: no URL generator available

Webhelpers pagination doesn’t know how to generate URLs in Pyramid, but, you can use a callable to generate the URLs. This requires access to pyramid.threadlocal which is generally not recommended, but, does allow you to use paginate until a pyramid compatible paginate is written.

The generator you need looks like this:

from webhelpers.util import update_params
from pyramid.threadlocal import get_current_request

def get_page_url(**kw):
    return update_params(get_current_request().path_info, **kw)

In your paginate block, you need to add the following:

        paginator = paginate.Page(
            page=int(self.request.params.get('page', 1)),
            items_per_page = 40,

RuntimeError: Caught exception rendering template. TypeError: ‘int’ object is not iterable

deform select widget requires tuple for the dropdown creation. Toscawidgets would build the right hand side if it was passed a list of IDs.

TypeError: ‘Undefined’ object is unsubscriptable

Returning a dict, removing tmpl_context., a ${value[‘asdf’]} is missing ‘value’:value being passed in the return dict.

TypeError: sequence item 10: expected string or Unicode, Undefined found

When using deform, return {‘form’:form.render()} rather than return {‘form’:form}.


This is the second application I’ve converted from Pylons 1.0 to Pyramid and most of the issues have been syntax issues. Hopefully the error summary above will save someone some time.

Converting Pylons 1.0 Repoze.who/Repoze.what to Pyramid 1.0a1 Authentication backed with MySQL

Thursday, November 11th, 2010

For one of our projects we used Pylons 1.0 with Repoze.who/Repoze.what, though, we only used groups without permissions. In our application, a member could access one of three controllers based on their membership in a group. There are a number of methods that can be used to set up authentication. One method is to create a thin table with the Primary Key, Username and Password and use an association table to add any additional information. This has the ability to be flexible when you need to add a number of fields that shouldn’t be contained within the AuthUser table. Another method is to use the AuthUser table to hold the related information. This example uses the latter method.

The first changes made are to to add the policies and Forbidden view. This view allows us to prompt a user for a username/password when they visit a page that is protected by the ACLs in Pyramid. contains our authentication model with the Permission models removed. We put the login/logout views, forbidden view along with the groupfinder and the RootFactory in to consolidate authentication into an and file for installations in other applications. Last but not least is a simple template to present the user with a login page.

For our RootFactory, we’ve defined three groups, client, manager and admin. Within our (or our views if we use add_handler), we can restrict permissions using Pyramid’s authentication.

In our, we can use the permission= to specify membership in a group.

config.add_route('admin2', '/admin/', view='project.admin.index', permission='admin', view_renderer='admin_index.jinja2')

If we have used add_handler:

config.add_route('admin2', '/admin/', view='project.admin.index', view_renderer='admin_index.jinja2')

We can use the permissions ACL in the @action decorator:

@action(renderer='admin_index.jinja2', permission='admin')

To access the userid in your views:

from import authenticated_userid

userid = authenticated_userid(request)

At this point, we’ve migrated a Repoze.who/Repoze.what authentication scheme that only used Group membership as its criteria and we have an SQL backed authentication system under Pyramid.

Most of the guidance for this came from:


Modifications to

Added to the import section:

from pyramid.authentication import AuthTktAuthenticationPolicy
from pyramid.authorization import ACLAuthorizationPolicy

from project.login import forbidden_view, groupfinder
from pyramid.exceptions import Forbidden

Added to the Configurator:

    authn_policy = AuthTktAuthenticationPolicy(
        'sosecret', callback=groupfinder)
    authz_policy = ACLAuthorizationPolicy()
    config = Configurator(settings=settings,

Added within the begin()/end() block of the Configurator

    config.add_view(forbidden_view, context=Forbidden)
    config.add_route('login', '/login',
    config.add_route('logout', '/logout',

import transaction

from sqlalchemy import create_engine
from sqlalchemy import Column
from sqlalchemy import Integer
from sqlalchemy import Unicode

from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.declarative import declarative_base

from sqlalchemy.orm import scoped_session
from sqlalchemy.orm import sessionmaker

from zope.sqlalchemy import ZopeTransactionExtension

DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
Base = declarative_base()

from sqlalchemy import *
from sqlalchemy.databases import mysql
from sqlalchemy.orm import relation, backref, synonym
from sqlalchemy.orm.exc import NoResultFound

import os
from hashlib import sha1
from datetime import datetime

user_group_table = Table('auth_user_groups', Base.metadata,
    Column('user_id', mysql.BIGINT(20, unsigned=True), ForeignKey('', onupdate='CASCADE', ondelete='CASCADE')),
    Column('group_id', mysql.BIGINT(20, unsigned=True), ForeignKey('', onupdate='CASCADE', ondelete='CASCADE'))

class AuthGroup(Base):
    __tablename__ = 'auth_groups'

    id = Column(mysql.BIGINT(20, unsigned=True), primary_key=True, autoincrement=True)
    name = Column(Unicode(80), unique=True, nullable=False)
    created = Column(mysql.DATE())

    users = relation('AuthUser', secondary=user_group_table, backref='auth_groups')

    def __repr__(self):
        return '' %

    def __unicode__(self):

class AuthUser(Base):
    __tablename__ = 'auth_users'

    id = Column(mysql.BIGINT(20, unsigned=True), primary_key=True, autoincrement=True)
    username = Column(Unicode(80), nullable=False)
    _password = Column('password', Unicode(80), nullable=False)
    email = Column(Unicode(80), nullable=False)
    contact = Column(Unicode(80), nullable=False)
    company = Column(Unicode(80), nullable=False)

    groups = relation('AuthGroup', secondary=user_group_table, backref='auth_users')

    def permissions(self):
        perms = set()
        for g in self.groups:
            perms = perms | set(g.permissions)
        return perms

    def _set_password(self, password):
        hashed_password = password

        if isinstance(password, unicode):
            password_8bit = password.encode('UTF-8')
            password_8bit = password

        salt = sha1()
        hash = sha1()
        hash.update(password_8bit + salt.hexdigest())
        hashed_password = salt.hexdigest() + hash.hexdigest()

        if not isinstance(hashed_password, unicode):
            hashed_password = hashed_password.decode('UTF-8')
        self._password = hashed_password

    def _get_password(self):
        return self._password

    password = synonym('_password', descriptor=property(_get_password, _set_password))

    def validate_password(self, password):
        hashed_pass = sha1()
        hashed_pass.update(password + self.password[:40])
        return self.password[40:] == hashed_pass.hexdigest()

    def __repr__(self):
        return '' % (, self.username,

    def __unicode__(self):
        return self.username

from pyramid.httpexceptions import HTTPFound
from import remember
from import forget
from import Allow
from import Everyone
from pyramid.url import route_url
from pyramid.renderers import render_to_response

from project.auth import AuthUser
from project.models import DBSession

def login(request):
    dbsession = DBSession()
    login_url = route_url('login', request)
    referrer = request.url
    if referrer == login_url:
        referrer = '/' # never use the login form itself as came_from
    came_from = request.params.get('came_from', referrer)
    message = ''
    login = ''
    password = ''
    if 'form.submitted' in request.params:
        login = request.params['login']
        password = request.params['password']
        auth = dbsession.query(AuthUser).filter(AuthUser.username==login).first()
        if auth.validate_password(password):
            headers = remember(request, login)
                  We use the Primary Key as our identifier once someone has authenticated rather than the
                  username.  You can change what is returned as the userid by altering what is passed to
            #headers = remember(request,
            return HTTPFound(location = came_from,
                             headers = headers)
        message = 'Failed login'

    return dict(
        message = message,
        url = request.application_url + '/login',
        came_from = came_from,
        login = login,
        password = password,
def logout(request):
    headers = forget(request)
    return HTTPFound(location = route_url('root', request),
                     headers = headers)
def forbidden_view(request):
    login_url = route_url('login', request)
    referrer = request.url
    if referrer == login_url:
        referrer = '/' # never use the login form itself as came_from
    came_from = request.params.get('came_from', referrer)
    return render_to_response('templates/', dict(
               message = '',
               url = request.application_url + '/login',
               came_from = came_from,
               login = '',
               password = '',
           ), request=request)

def groupfinder(userid, request):
    dbsession = DBSession()
    auth = dbsession.query(AuthUser).filter(
    if auth:
        return [('group:%s' % for group in auth.groups]

class RootFactory(object):
    __acl__ = [ (Allow, 'group:client', 'client'),
                (Allow, 'group:manager', 'manager'),
                (Allow, 'group:admin', 'admin') ]
    def __init__(self, request):


<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"

  <meta content="text/html; charset=utf-8" http-equiv="Content-Type"/>
  <title>Authentication Test</title>
  <link rel="stylesheet" type="text/css"
        href="${request.application_url}/static/style.css" />


<h1>Log In</h1>

<div tal:replace="message"/>

<div class="main_content">
  <form action="${url}" method="post">
    <input type="hidden" name="came_from" value="${came_from}"/>
    <input type="text" name="login" value="${login}"/>
    <input type="password" name="password" value="${password}"/>
    <input type="submit" name="form.submitted" value="Log In"/>


Using Pyramid with Deform to Edit/Create records in SQLAlchemy backed database

Monday, November 8th, 2010

While working with Pyramid it was worth taking a good look at deform. During my brief look before, the one thing that appeared to be missing was a method to quickly integrate with SQLAlchemy backed data. For a handful of forms, manually creating the appstruct to be passed to the Form wouldn’t be difficult, but, for some more significant applications, it could be quite cumbersome. The following is a one page application that allows you to create a new user, or, edit an existing user by specifying the userid in the URL. The two pieces that do the real work are record_to_appstruct and merge_session_with_post. record_to_appstruct takes the class returned from SQLAlchemy and converts it to an appstruct that deform likes. Once we get the validated data, we merge the record with the post items using merge_session_with_post, merge the record and present a screen with the post data.

This is more a proof of concept, but, works well enough that we were able to convert quite a few forms and work more closely with Deform and Pyramid as we do more development.


from pyramid.configuration import Configurator
from paste.deploy.converters import asbool

from defo.models import initialize_sql

def app(global_config, **settings):
    """ This function returns a WSGI application.
    It is usually called by the PasteDeploy framework during 
    ``paster serve``.
    db_string = settings.get('db_string')
    if db_string is None:
        raise ValueError("No 'db_string' value in application configuration.")
    db_echo = settings.get('db_echo', 'false')
    initialize_sql(db_string, asbool(db_echo))
    config = Configurator(settings=settings)
    config.add_static_view('static', 'defo:static')
    config.add_route('home', '/', view='defo.views.edit',
    config.add_route('homeid', '/:id', view='defo.views.edit',
    return config.make_wsgi_app()

from defo.models import DBSession
from defo.models import MyModel

from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()

from sqlalchemy import *
from sqlalchemy.databases import mysql
from sqlalchemy.orm import relation, backref, synonym
from sqlalchemy.orm.exc import NoResultFound

import deform
import colander
from webhelpers import constants

class UserSchema(colander.Schema):
    username = colander.SchemaNode(colander.String())
    contact = colander.SchemaNode(colander.String())
    email = colander.SchemaNode(colander.String())
    company = colander.SchemaNode(colander.String())
    addr1 = colander.SchemaNode(colander.String())
    addr2 = colander.SchemaNode(colander.String(), missing=u'',)
    city = colander.SchemaNode(colander.String())
    state = colander.SchemaNode(colander.String())
    zip = colander.SchemaNode(colander.String())
    country = colander.SchemaNode(
        widget = deform.widget.SelectWidget(values=constants.country_codes()),
    phone = colander.SchemaNode(colander.String())

class AuthUser(Base):
    __tablename__ = 'auth_users'

    id = Column(mysql.BIGINT(20, unsigned=True), primary_key=True, autoincrement=True)
    username = Column(Unicode(80), nullable=False)
    email = Column(Unicode(80), nullable=False)
    contact = Column(Unicode(80), nullable=False)
    company = Column(Unicode(80), nullable=False)
    addr1 = Column(Unicode(80), nullable=False)
    addr2 = Column(Unicode(80))
    city = Column(Unicode(80), nullable=False)
    state = Column(Unicode(80), nullable=False)
    zip = Column(Unicode(80), nullable=False)
    country = Column(Unicode(80), nullable=False)
    phone = Column(Unicode(80), nullable=False)

def record_to_appstruct(self):
    return dict([(k, self.__dict__[k]) for k in sorted(self.__dict__) if '_sa_' != k[:4]])

def merge_session_with_post(session, post):
    for key,value in post:
        setattr(session, key, value)
    return session

def edit(request):
    dbsession = DBSession()
    if request.matchdict.has_key('id'):
        record = dbsession.query(AuthUser). \
        record = AuthUser()

    schema = UserSchema()
    form = deform.Form(schema, buttons=('submit',))
    if request.POST:
            appstruct = form.validate(request.POST.items())
        except deform.ValidationFailure, e:
            return {'form':e.render()}
        record = merge_session_with_post(record, request.POST.items())
        return {'formdata':appstruct}
        appstruct = record_to_appstruct(record)
    return {'form':form.render(appstruct=appstruct)}


    Deform Demo Site
  <!-- Meta Tags -->
  <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
  <!-- CSS -->
  <link rel="stylesheet" href="/static/css/form.css" type="text/css" />
  <link rel="stylesheet" href="/static/css/theme.css" type="text/css" />
  <!-- JavaScript -->
  <script type="text/javascript"
  <script type="text/javascript"
this is a test template in mako
% if form:
% else:
We got our form data: ${formdata}
% endif
<script type="text/javascript">

Entries (RSS) and Comments (RSS).
Cluster host: li