Posts Tagged ‘turbogears’

Diagnosing errors in Pylons 1.0 to Pyramid 1.0a1+ Transition

Friday, November 26th, 2010

My first attempt at migrating a Pylons project to Pyramid was accomplished without too much difficulty. That project was relatively small, however, this project hasn’t been put into production.

A brief history of this application:

We have a legacy PHP application which consists of 78k lines of code. Originally we started rewriting the application in Turbogears 2.0 and later moved over to Pylons with ToscaWidgets. We used ToscaWidgets because all of our forms had been written for TG2. The application never made it into production, but, was substantial enough that we felt it would be a good test to convert it to Pyramid since we’re focusing our development efforts on Pyramid.

Initially I started to write a script to do the migration from Pylons to Pyramid and from ToscaWidgets to Deform but abandoned that. Changing the controllers from Pylons to Pyramid was fairly easy.

Most of the pylons imports were commented out and replaced with:

import deform
import colander

from pyramid.response import Response
from pyramid.view import action
from pyramid.security import authenticated_userid
from pyramid.threadlocal import get_current_request
import webhelpers.paginate as paginate

References to tmpl_context. were altered, our __before__ action for authkit was removed and replaced with an __init__ since we were using handlers. Routes were modified and given unique names, templates were modified to remove references to ${h. and ${tmpl_context., and we began the process of stepping through the application. Initially I had written a script to convert the ToscaWidgets form models over to Deform schemas, but, after two hours, it became obvious it would take more time to do that than to manually recreate the forms. After working through much of the process, I’m debating whether this project should have been migrated over to FormAlchemy as almost every form is a duplicate of the SQL schema. Since it isn’t in production, we still have some time to make that decision.

What follows is a summary of the errors received and what caused the errors.

TypeError: ‘NoneType’ object is not iterable

I ran into this while importing some routes from an existing project. Since routes must be uniquely named in Pyramid, a route with the same name will replace a prior route. Some of the route names have gotten quite unwieldy.

TypeError: object.__new__() takes no parameters

Solution, add the __init__ block to your class in your handler.

class YourClass(object):
    def __init__(self, request):
        self.request = request

ValueError: Non-response object returned from view named (and no renderer): {‘template’: ‘billing_index’}

Turbogears 2.0 code:

    def index(self, **kw):
        return dict(template='billing_index')

modify to:

    @action(renderer='billing_index.mako', permission='client')
    def index(self, **kw):
        return {}

AttributeError: ‘Undefined’ object has no attribute ‘form’

Pyramid doesn’t pass the tmpl_context., c. or h. through to the template, so, the return values of each of the actions needs to pass the fields required. As a result, any template code that replies on these globals needs to be modified, and the corresponding action needs to return the values in the dictionary. You can use one of the pylons templates which will instantiate a subscriber method to replicate the Pylons globals if you want.

AttributeError: ‘Undefined’ object has no attribute ‘literal’

Existing forms are relying on the pylons h. global.

${h.literal(form(value=value))}

UnboundLocalError: local variable ‘clients’ referenced before assignment

This occurs when you remove tmpl_context. and are left with a variable assigned that matches the class. For example:

tmpl_context.clients = meta.Session.query(clients).filter_by(client_id==1).all()

When tmpl_context. is removed, the class clients is cast incorrectly.

AttributeError: ‘Undefined’ object has no attribute ‘pager’

Paginate itemset isn’t being passed to the template.

TypeError: ‘NoneType’ object is not iterable

Paginate is getting the result set, rather than the paginate set passed in the return dictionary.

NameError: global name ‘request’ is not defined

request.matchdict/request.params -> self.request.matchdict/self.request.params

In a handler, request. is referred to as self.request. If you’ve converted things over to a handler rather than writing the individual routes for each action, you’ll need to preface any request. with self.

NotImplementedError: no URL generator available

Webhelpers pagination doesn’t know how to generate URLs in Pyramid, but, you can use a callable to generate the URLs. This requires access to pyramid.threadlocal which is generally not recommended, but, does allow you to use paginate until a pyramid compatible paginate is written.

The generator you need looks like this:

from webhelpers.util import update_params
from pyramid.threadlocal import get_current_request

def get_page_url(**kw):
    return update_params(get_current_request().path_info, **kw)

In your paginate block, you need to add the following:

url=get_page_url,
        paginator = paginate.Page(
            features,
            page=int(self.request.params.get('page', 1)),
            items_per_page = 40,
            url=get_page_url,
        )

RuntimeError: Caught exception rendering template. TypeError: ‘int’ object is not iterable

deform select widget requires tuple for the dropdown creation. Toscawidgets would build the right hand side if it was passed a list of IDs.

TypeError: ‘Undefined’ object is unsubscriptable

Returning a dict, removing tmpl_context., a ${value[‘asdf’]} is missing ‘value’:value being passed in the return dict.

TypeError: sequence item 10: expected string or Unicode, Undefined found

When using deform, return {‘form’:form.render()} rather than return {‘form’:form}.

Summary

This is the second application I’ve converted from Pylons 1.0 to Pyramid and most of the issues have been syntax issues. Hopefully the error summary above will save someone some time.

AttributeError: ‘function’ object has no attribute ‘replace’

Wednesday, September 30th, 2009

While doing some coding to move a Turbogears2 application over to Pylons, I ran into an issue with Validation and ToscaWidgets.

AttributeError: ‘function’ object has no attribute ‘replace’

Validation in Pylons listed:

@validate(form=movie_form, error_handler=index)

for the decorator syntax, but, error_handler should be a name, not a function. The correct decorator should be:

@validate(form=movie_form, error_handler=’index’)

Rapid Application Development using Turbogears and Django

Saturday, August 8th, 2009

For the last 14 months we’ve been developing an application to replace 90000 lines of PHP code. Rewriting the application from scratch to support I18N and many of the enhancements it needed was deemed to be a better long term solution.

When that project was first started, I spent a month with Django and a month with Turbogears writing the same test application so that I could compare the development cycle. Both have matured and I needed to do a rapid turnaround on another project. I decided to give Django another look since it had hit version 1.0 and had added quite a few features that were missing in my preliminary evaluation. What follows is a discussion of the major points from both of the frameworks.

Turbogears has excellent form handling. Except for the Forms Wizard in Django, working with forms is much easier in Turbogears. Validation of the form and the resulting database update methods are much cleaner in Turbogears. Django, while slightly more complex in handling form input, does handle things with a single function which might enhance readability in a large project.

Database handling through SQL Alchemy in Turbogears is much better than the database methods in Django. Yes, you can use SQL Alchemy in Django now, but, their default ORM has plenty of room for improvement.

Turbogears is true MVC. Their terminology and methods are true to the paradigm set forth by Smalltalk. Django is MVC, but they call it MTV, Model, Template, View. The differences are slight and the developers of both have made good decisions. Adapting to either project’s methods is quick and not a hindrance.

Django’s definitely wins with Authentication/Authorization. Methods to handle registration, user creation, login and forgotten passwords are built in and wrapped with very nice templates that can be overridden. For Turbogears, repoze.who and repoze.what have been pulled from Plone and put into place. While Turbogears works with repoze, the decisions made and the lack of full support behind it make it difficult to implement.

Django feels faster. Comparing 14 months of development in Turbogears on an application to an application written in Django this week, the template engine, database access and pageload time seemed faster. Django is a lighter weight framework and you are closer to the data. Turbogears puts a little more insulation in which makes some coding easier at the expense of performance.

Maintainability of code would have to go to Turbogears. IBM once stated that the maximum number of bugfree lines of code that could be written was 23. With Turbogears, much of the heavy lifting is handled by widgets and decorators and your model resulting in a smaller codebase. Django requires more code to do the same task unless you utilize some of the snippets. Turbogears makes certain assumptions and has wrapped many of the libraries that make development easy in the default installation. Django’s default installation lacks those decisions, but, you are not prevented from establishing your own middleware. If you were developing a number of Django projects, you would pick and choose snippets that would replicate the decisions that Turbogears has already made.

URL Mapping is much easier to implement with Django. While routes support in Turbogears is present, Django’s regexp mapping is much easier to manipulate.

Community, hands down, Django wins. With a much larger installed base, bugs are found and fixed much more quickly. While Turbogears was founded on loftier principles, execution and follow through are lacking. Development is done when it is needed by a client project in the core group of developers. There is a definite air of condescension when the project developers handle questions from potential developers. With Django, there are people of all experience levels willing to help on groups.google, IRC, and thorough documentation that far exceeds most of the open source documentation out there.

Documentation, again, Django. Well organized, well thought out and well documented examples on Django’s part show dedication to having a framework that is well understood and well used. Turbogears recently moved to Sphinx, but, documentation generated from poorly documented code still means poor documentation. The tutorials and examples have been improving, but, they have a long way to go.

Genshi and Mako are supported fairly well with Turbogears and are both very good engines. Jinja is also supported which is a bit faster than Genshi and is powerful and very easy to work with. Django’s template language is also very flexible, powerful and easy to work with. Django had some definite advantages with a simpler language, but, neither Django or Turbogears stood out as a clear winner.

If you were looking to write an extremely heavy database or form involved site, I think Turbogears would be a good solution. If you choose Turbogears, be prepared to delve into the code when you are faced with problems. Bugs are not dealt with very promptly even though upgrades are pushed out. Be prepared to patch upgraded packages or hold certain packages once you move into production.

On the other hand, if you are writing a less complicated application, Django would be a good choice.

All told, the application developed this week in Django took about 12 hours and had I been working with Django for the last 14 months, I would estimate the project to have taken roughly 8 hours. Developed in Turbogears, I probably could have written it in 6 hours. PHP, to mimic all of the functionality would have taken 14-16 hours even using one of the numerous frameworks.

There is a definite advantage to using a framework and Python frameworks do appear to offer Rapid Application Development even over most of the PHP frameworks. For me, new development will probably be done in Django unless there is a strong case to use Turbogears.

* Turbogears 2.0
* Django
* Django Snippets

User Interface Design

Wednesday, June 24th, 2009

Programmers are not designers. Technical people should not design User Interfaces.

* 810 source files
* 90658 lines of code
* 10213 lines of html

For an internal project tasked to a series of programmers throughout the years without enough oversight, it is a mass of undocumented code with multiple programming styles. PHP allowed lazy programming, Smarty didn’t have some of the finesse required, so, the User Interface suffered. Functional but confusing to anyone that hadn’t worked intimately with the interface or been walked through it.

The truest statement is that it is easier for me to do things through the MySQL command line than through the application. While this does have a tendency to introduce possible typos, it has altered SQL practices here.

update table set value=123 where othervalue=246;

could have an accidental typo of

update table set value=123 where othervalue-=246;

which would have completely unintended consequences. One typo altered the DNS entries for 48000 records. Shortly after that typo, ingrained in company policy was that I never wanted to ever see a query like that executed in the command line regardless of how simple the command.

Even within code, the above command would be entered as:

update table set value=123 where othervalue in (246);

This prevented a number of potential typos. Even limit clauses with deletions were enforced to make sure things didn’t go too haywire in an update.

With Python, indenting is mandatory which results in multiple programmer’s code looking similar and easier to troubleshoot. Utilizing SQLAlchemy which enforces bind variables when talking with the database engine, we’ve eliminated the potential for a typo updating too many records. Even cascade deletes are enforced in SQLAlchemy even when running on top of MyISAM. With MVC, our data model is much better defined and we’re not tied down to remembering the relationship between two tables and possible dependencies. Conversion from the existing MySQL database to a DeclarativeBase model hasn’t been without issues, but, a simple python program allowed the generation of a simple model that took care of most of the issues. Hand tweaking the database model while developing the application has allowed for quite a bit of insight into issues that had been worked around rather than making adjustments to the database.

Fundamental design issues in the database structure were worked around with code rather than fixed. Data that should have been retained was not, relationships between tables was defined in code rather than in the database leading to a painful conversion.

When it was decided to rewrite the application in Python using TurboGears, I wasn’t that familiar with the codebase nor the user interface. Initially it was envisioned that the templates would be copied and the backend engine would be written to power those templates. After a few hours running through the application, and attempting the conversion on a number of templates, I realized the application was functional but it was extremely difficult to use in its current state. So much for having a programmer design an interface.

Some functionality from the existing system was needed so I peered into the codebase and was unprepared for that surprise. At this point it became evident that a non-programmer had designed the interface. While Smarty was a decent template language, it was not a formtool, so, methods were designed to give a consistent user experience when dealing with error handling. A single php file was responsible for display, form submission and validation and writing to the database for each ‘page’ in the application. The code inside should have been straightforward.

* Set up default CSS classes for each form field for an ‘ok’ result
* Validate any passed values and set the CSS class as ‘error’ for any value that fails validation
* Insert/Update the record if the validation passes
* Display the page

Some validation takes place numerous times throughout the application, and, for some reason one of the ‘coders’ decided that copy and paste of another function that used that same validation code was better than writing a function to do the validation. Of course when that validation method needed to be changed, it needed to be changed in eight places.

So, what should have been somewhat simple has changed considerably:

* Evaluate each page
* Redesign each page to make the process understandable
* Adjust terminology to make it understandable to the application’s users
* modify the database model
* rewrite the form and validation

A process that should have been simple has turned into quite a bit more work than anticipated. Basically, development boils down to looking at the page, figuring out what it should be, pushing the buttons to see what they do and rewriting from scratch.

TurboGears has added a considerable amount of efficiency to the process. One page that dealt with editing a page of information was reduced from 117 lines of code to 12 lines of code. Since TurboGears uses ToscaWidgets and Formencode, validation and form presentation is removed from the code resulting in a controller that contains the code that modifies the tables in the database with validated input. Since Formencode already has 95% of the validators that are needed for this project, we can rest assured that someone else has done the work to make sure that field will be properly validated. Other validation methods can be maintained and self-tested locally, but, defined in such a manner that they are reused throughout the application rather than being cut and pasted into each model that is validating data. In addition, bugs should be much less frequent as a result of a much-reduced codebase.

Due to the MVC framework and the libraries selected by the developers at TurboGears, I wouldn’t be surprised if the new codebase is 10%-15% the size of the existing application with greater functionality. The code should be more maintainable as python enforces some structure which will increase readability.

While I am not a designer, even using ToscaWidgets and makeform, the interface is much more consistent. Picking the right words, adding the appropriate help text to the fields and making sure things work as expected has resulted in a much cleaner, understandable interface.

While there are some aspects of ToscaWidgets that are a little too structured for some pages, our current strategy is to develop the pages using ToscaWidgets or makeform to make things as clear as possible making notes to overload the Widget class for our special forms at a later date.

While it hasn’t been a seamless transition, it did provide a good opportunity to rework the site and see a number of the problems that the application has had for a long time.

TurboGears, Tableform and a callable option to a Widget

Wednesday, June 10th, 2009

While doing some TurboGears development I ran into an issue where I needed to generate a select field’s options from the database that was dependent on authentication. Since defining the query in the model results in a cached result when the class is instantiated, the query couldn’t be defined there. There are multiple mentions of using a callable to deal with this situation, but, no code example.

From this posting in Google Groups for TurboGears, we were able to figure out the code that made this work.

template:

<div xmlns="http://www.w3.org/1999/xhtml"
      xmlns:py="http://genshi.edgewall.org/"
      xmlns:xi="http://www.w3.org/2001/XInclude" 
      py:strip="">

${tmpl_context.form(value=value)}

</div>

controller:

    @expose('cp.templates.template')
    def form(self):
        c.form = TestForm()
        c.availips = [[3,3],[2,2]]
        return dict(template='form',title='Test Form',value=None)

model:

from pylons import c

def get_ips():
    return c.availips

class TestForm(TableForm):
    action = '/test/testadd'
    submit_text = 'Add test'

    class fields(WidgetsList):
        User = TextField(label_text='FTP Username', size=40, validator=NotEmpty())
        Password = PasswordField(label_text='FTP Password', size=40, validator=NotEmpty())
        ip = SingleSelectField(label_text="Assign to IP",options=get_ips)

Entries (RSS) and Comments (RSS).
Cluster host: li