Within 15 minutes of installing Tornado and using their simple hello world example, I imported SQLAlchemy and ended up boosting this to 280 requests per second. As I really didn’t need any of the features from the ORM, I decided to use tornado.database which isn’t much more than a bare wrapper to python-mysql. Even with a single worker process, I was able to get 870 requests per second. 56 million requests per day, without any tuning?

I’m reasonably impressed. Once I put it on production hardware, I’m thinking I’ll easily be able to count on double those numbers if not more.

Next weekend, Traffic Server.

First, our platform needs to be designed. For this set of projects we’re going to use Nginx with uwsgi. Since we’re serving static content, we’re going to set up our directories on Nginx to allow that content to be served outside our Pylons virtual environment. Tony Landis was one of the first to provide an implementation guide for uwsgi with Nginx for Pylons which provided some of the information needed to get things working.

Our theoretical layout looks like the following:

/webroot
  |--- /static
  |--- /fb
/virtualenv
  |--- /fbappone
  |--- /fbapptwo

Later we’ll add a CDN that does origin pulls from /webroot/static. This application would have worked wonderfully with Varnish and ESI if the ESI could be compressed, but, setting up Nginx -> Varnish -> Nginx -> uwsgi seemed somewhat inefficient just to add compression. The Facebook application we’ve developed is an IFrame canvas which took roughly fifteen hours to debug after the original concept was decided. The majority of that time was spent dealing with the IFrame canvas issues. FBML was much easier to get working properly.

What we end up with is a url structure like:

http://basedomain.com/

     /static/ (xd_receiver.html, jquery support modules, CSS files)
     /fb/ (Generic facebook files, support, tos, help)
     /(fbapp)/application_one/
     /(fbapp)/application_two/

As a result of this structure, we don’t need to manipulate config/routing.py as the default’s set by Pylons map things the way we want. In the /static/ directory, we can put our CSS, js and static media files. Remember to minify the CSS and js files and combine them if possible.

Our nginx config looks like:

server {
    listen   1.2.3.4:80;
    server_name  xxxxxx.com;
    access_log /var/log/nginx/xxxxxx.com-access.log;

    location ~* (css|js|png|jpe?g|gif|ico|swf|flv)$ {
        expires max;
    }

    gzip on;
    gzip_min_length 500;
    gzip_types text/plain application/xml text/html text/javascript;
    gzip_disable "MSIE [1-6]\.";

    location ^~ /static/ {
    	alias   /var/www/xxxxxx.com/static/;
    }
    location ^~ /fb/ {
    	alias   /var/www/xxxxxx.com/fb/;
    }
    location / {
        uwsgi_pass  unix:/tmp/uwsgi.sock;
        include     uwsgi_params;
    }
}

We could modify the nginx config to pull / from the static page, but, we’re actually capturing that with a root controller that knows what applications reside below it as a directory of sorts.

We used Debian which doesn’t support uwsgi yet. A brief set of instructions follows which should work on any Debian based distribution as well:

apt-get install libxml2-dev dpkg-dev debhelper
cd /usr/src
apt-get source nginx
wget http://projects.unbit.it/downloads/uwsgi-0.9.4.4.tar.gz
tar xzf uwsgi-0.9.4.4.tar.gz
cd nginx
vi debian/rules
  add:  --add-module=/usr/src/uwsgi-0.9.4.4/nginx/ \
dpkg-buildpackage
dpkg -i ../nginx_0.7.65-5_i386.deb
mkdir /usr/local/nginx/
cp /usr/src/uwsgi-0.9.4.4/nginx/uwsgi_params /etc/nginx

/etc/nginx/uwsgi_params, add:

uwsgi_param  SCRIPT_NAME        /;

Note: I had problems with 0.9.5.1 and paster enabled wsgi applications which caused issues with Pylons.

Our uwsgi command line for development:

/usr/src/uwsgi-0.9.4.4/uwsgi -s /tmp/uwsgi.sock -C -iH /var/www/facebook/ --paste config:/var/www/facebook/fpapp/development.ini

One of the things that made Facebook integration difficult was somewhat incomplete documentation or even incorrect documentation on Facebook’s site. While the Graph API is new, it is quite a bit more powerful. While they do have official support, I think I’ll use velruse for OAuth integration next time and use the Python-SDK for the Graph API integration. See my previous post on using Pylons for a Facebook Application for a little more detailed information on how to get the application working.

And another site offering a tutorial of sorts dealing with ORMs Why should you use an ORM.

While both have their points, both have missed a fundamental benefit that an ORM hands you.

Most of my development is in Pylons. Django’s ORM and template language can do the same thing. A programmer that has used PHP/Smarty to develop large scale systems will likely resist ORMs. After working with a team to develop 90k+ lines of PHP/Smarty over a six year period, making the shift required a paradigm shift.

Let’s consider the following structure. We have a cp_ticket table and a cp_ticket_detail table. A Ticket can have multiple detail records. The output we wish to have is:

ticket id, ticket header information
         ticket detail line
         ticket detail line #2
ticket id, ticket header information
         ticket detail line
         ticket detail line #2
         ticket detail line #3
ticket id, ticket header information
         ticket detail line
         ticket detail line #2

Our model:

class cp_ticket(DeclarativeBase):
    __tablename__ = 'cp_ticket'

    ticket_id = Column(mysql.MSBigInteger(20, unsigned = True), primary_key=True, autoincrement = True)
    priority = Column(mysql.MSEnum('1','2','3','4','5'), default = '3')

    ticket_detail = relation('cp_ticket_detail', order_by='cp_ticket_detail.ticket_detail_id')

class cp_ticket_detail(DeclarativeBase):
    __tablename__ = 'cp_ticket_detail'

    ticket_id = Column(mysql.MSBigInteger(20, unsigned = True), ForeignKey('cp_ticket.ticket_id'), default = '0')
    ticket_detail_id = Column(mysql.MSBigInteger(20, unsigned = True), primary_key=True, autoincrement = True)
    stamp = Column(mysql.MSTimeStamp, PassiveDefault('CURRENT_TIMESTAMP'))
    detail = Column(mysql.MSLongText, default = '')

Our query to pass to our template:

        tickets = meta.Session.query(cp_ticket).filter(cp_ticket.client_id==1).all()

Compared with the query as you would write it without an ORM:

select * from cp_ticket,cp_ticket_detail where client_id=1 and cp_ticket.ticket_id=cp_ticket_detail.ticket_id;

Both are doing the same fundamental thing, but, the ORM maps the results almost identical to the way we want to display the data. This makes template design easy.

Using Mako, we use the following code to display the results:

<table border="1">
 <tr><th>Ticket ID</th><th>Status</th><th>Detail</th></tr>
%for ticket in tmpl_context.tickets:
  <tr>
    <td><strong>${ticket.ticket_id}</strong></td>
    <td><strong>${ticket.priority}</strong></td>
  </tr>
  %for detail in ticket.ticket_detail:
  <tr>
    <td></td>
    <td>${detail.stamp}</td>
    <td>${detail.detail}</td>
  </tr>
  % endfor
% endfor
</table>

To do the same thing without using an ORM, you need to revert to a control break structure similar to the following:

current_ticket=0
for ticket in tickets:
  if (current_ticket != ticket.ticket_id):
    #new row, print the header
    print "<tr><td>first piece</td></tr>"
    current_ticket = ticket.ticket_id
  # print our detail row
  print "<tr><td></td><td>stamp and detail</td></tr>"

Control Break structures require you to be able to set a variable within your template language. Some template languages don’t allow that. If your template language (in any language) can’t do variable assignments in the template, guess where your html generation logic needs to go?

With an ORM, the template contains your display logic. Your webmaster/design team can modify the template without having to modify html contained within your code. The loops are simple to understand and designers usually have little problem avoiding the lines that start with %.

Sure, you could wrap much of this logic in your template to do the control-break structure, but, as you get more complex data, deciding how to display the data requires a define or some other functionality.

An ORM adds some insulation to the process, but, the result is a much easier page structure when displaying related data. Granted there are some performance hits and SQLAlchemy appears to create some queries that are not optimal, unless there is a tremendous performance hit, I think the benefits of the ORM for developing a web application are tremendous.

Once you move into an environment where you are dealing with multiple developers, having a defined schema with comments is much easier than using reflection to figure out what the meaning of a status field as enum(‘U’,'A’,'P’,'C’,'R’,'S’).

However, as the original poster mentions, you can do raw SQL within SQLAlchemy and do all of your work with reflection as he has done with his ORM^H^H^H, abstraction. If he’s still using SQLAlchemy, he can selectively decide when to use it and when to avoid it.

* Install mysql’s x86_64 version from the .dmg file (I had a problem doing this when booted into the 64bit kernel, a 32bit kernel macbook had no problem) With the 64bit kernel, the system reported ‘no mountable file systems’ when trying to mount the .dmg file. A reboot into 32bit mode allowed the .dmg to be mounted and installed.
* Change into your virtual environment if desired, source bin/activate
* fetch MySQL-python-1.2.3c1

tar xzf MySQL-python-1.2.3c1.tar.gz
cd MySQL-python-1.2.3c1
ARCHFLAGS='-arch x86_64' python setup.py build
ARCHFLAGS='-arch x86_64' python setup.py install

If everything works, you should see:

$ python
Python 2.6.1 (r261:67515, Jul  7 2009, 23:51:51)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
>>>

Some of the possible things you’ll encounter:

After python setup.py build:

ld: warning: in build/temp.macosx-10.6-universal-2.6/_mysql.o, file is not of required architecture
ld: warning: in /usr/local/mysql/lib/libmysqlclient.dylib, file is not of required architecture
ld: warning: in /usr/local/mysql/lib/libmygcc.a, file is not of required architecture
ld: warning: in build/temp.macosx-10.6-universal-2.6/_mysql.o, file is not of required architecture
ld: warning: in /usr/local/mysql/lib/libmysqlclient.dylib, file is not of required architecture
ld: warning: in /usr/local/mysql/lib/libmygcc.a, file is not of required architecture

This means that you have the i386 version of mysql installed. Or, if you have the x86_64 version, you have didn’t include the proper ARCHFLAGS command.

ImportError: dynamic module does not define init function (init_mysql)

this means that the easy_install or the build/install process was run which tried to build a ppc/i386/x86_64 combined build.

If you see messages like:

$ python
Python 2.6.1 (r261:67515, Jul  7 2009, 23:51:51)
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
/Users/xxxxx/Python/django/lib/python2.6/site-packages/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg/_mysql.py:3: UserWarning: Module _mysql was already imported from /Users/xxxxx/Python/django/lib/python2.6/site-packages/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg/_mysql.pyc, but /Users/xxxxx/Python/django/MySQL-python-1.2.3c1 is being added to sys.path
>>>

Then you are still in the build directory. cd .. and try again.

While the app engine platform is still running, it appears to be an issue with this particular application’s committed resources. The App Gallery has exceeded it’s memory quota.

* 810 source files
* 90658 lines of code
* 10213 lines of html

For an internal project tasked to a series of programmers throughout the years without enough oversight, it is a mass of undocumented code with multiple programming styles. PHP allowed lazy programming, Smarty didn’t have some of the finesse required, so, the User Interface suffered. Functional but confusing to anyone that hadn’t worked intimately with the interface or been walked through it.

The truest statement is that it is easier for me to do things through the MySQL command line than through the application. While this does have a tendency to introduce possible typos, it has altered SQL practices here.

update table set value=123 where othervalue=246;

could have an accidental typo of

update table set value=123 where othervalue-=246;

which would have completely unintended consequences. One typo altered the DNS entries for 48000 records. Shortly after that typo, ingrained in company policy was that I never wanted to ever see a query like that executed in the command line regardless of how simple the command.

Even within code, the above command would be entered as:

update table set value=123 where othervalue in (246);

This prevented a number of potential typos. Even limit clauses with deletions were enforced to make sure things didn’t go too haywire in an update.

With Python, indenting is mandatory which results in multiple programmer’s code looking similar and easier to troubleshoot. Utilizing SQLAlchemy which enforces bind variables when talking with the database engine, we’ve eliminated the potential for a typo updating too many records. Even cascade deletes are enforced in SQLAlchemy even when running on top of MyISAM. With MVC, our data model is much better defined and we’re not tied down to remembering the relationship between two tables and possible dependencies. Conversion from the existing MySQL database to a DeclarativeBase model hasn’t been without issues, but, a simple python program allowed the generation of a simple model that took care of most of the issues. Hand tweaking the database model while developing the application has allowed for quite a bit of insight into issues that had been worked around rather than making adjustments to the database.

Fundamental design issues in the database structure were worked around with code rather than fixed. Data that should have been retained was not, relationships between tables was defined in code rather than in the database leading to a painful conversion.

When it was decided to rewrite the application in Python using TurboGears, I wasn’t that familiar with the codebase nor the user interface. Initially it was envisioned that the templates would be copied and the backend engine would be written to power those templates. After a few hours running through the application, and attempting the conversion on a number of templates, I realized the application was functional but it was extremely difficult to use in its current state. So much for having a programmer design an interface.

Some functionality from the existing system was needed so I peered into the codebase and was unprepared for that surprise. At this point it became evident that a non-programmer had designed the interface. While Smarty was a decent template language, it was not a formtool, so, methods were designed to give a consistent user experience when dealing with error handling. A single php file was responsible for display, form submission and validation and writing to the database for each ‘page’ in the application. The code inside should have been straightforward.

* Set up default CSS classes for each form field for an ‘ok’ result
* Validate any passed values and set the CSS class as ‘error’ for any value that fails validation
* Insert/Update the record if the validation passes
* Display the page

Some validation takes place numerous times throughout the application, and, for some reason one of the ‘coders’ decided that copy and paste of another function that used that same validation code was better than writing a function to do the validation. Of course when that validation method needed to be changed, it needed to be changed in eight places.

So, what should have been somewhat simple has changed considerably:

* Evaluate each page
* Redesign each page to make the process understandable
* Adjust terminology to make it understandable to the application’s users
* modify the database model
* rewrite the form and validation

A process that should have been simple has turned into quite a bit more work than anticipated. Basically, development boils down to looking at the page, figuring out what it should be, pushing the buttons to see what they do and rewriting from scratch.

TurboGears has added a considerable amount of efficiency to the process. One page that dealt with editing a page of information was reduced from 117 lines of code to 12 lines of code. Since TurboGears uses ToscaWidgets and Formencode, validation and form presentation is removed from the code resulting in a controller that contains the code that modifies the tables in the database with validated input. Since Formencode already has 95% of the validators that are needed for this project, we can rest assured that someone else has done the work to make sure that field will be properly validated. Other validation methods can be maintained and self-tested locally, but, defined in such a manner that they are reused throughout the application rather than being cut and pasted into each model that is validating data. In addition, bugs should be much less frequent as a result of a much-reduced codebase.

Due to the MVC framework and the libraries selected by the developers at TurboGears, I wouldn’t be surprised if the new codebase is 10%-15% the size of the existing application with greater functionality. The code should be more maintainable as python enforces some structure which will increase readability.

While I am not a designer, even using ToscaWidgets and makeform, the interface is much more consistent. Picking the right words, adding the appropriate help text to the fields and making sure things work as expected has resulted in a much cleaner, understandable interface.

While there are some aspects of ToscaWidgets that are a little too structured for some pages, our current strategy is to develop the pages using ToscaWidgets or makeform to make things as clear as possible making notes to overload the Widget class for our special forms at a later date.

While it hasn’t been a seamless transition, it did provide a good opportunity to rework the site and see a number of the problems that the application has had for a long time.

model:

class AchievementCategory(DeclarativeBase):
	__tablename__ = 'achievement_categories'

	id = Column(mysql.MSBigInteger(20, unsigned = True), primary_key = True)
	parent_id = Column(mysql.MSBigInteger(20, unsigned = True), default = 0)
	name = Column(Unicode(80))

code:

def get_cats(n = 0, c_list = [], level = 0):
	sql = DBSession.query(AchievementCategory).filter_by(parent_id = n).order_by(AchievementCategory.name).all()
	for e in sql:
		c_list.append([e.id, level * " " + e.name, level])
		get_cats(e.id, c_list, level+1)
	return c_list

print get_cats()

From this posting in Google Groups for TurboGears, we were able to figure out the code that made this work.

template:

<div xmlns="http://www.w3.org/1999/xhtml"
      xmlns:py="http://genshi.edgewall.org/"
      xmlns:xi="http://www.w3.org/2001/XInclude"
      py:strip="">

${tmpl_context.form(value=value)}

</div>

controller:

    @expose('cp.templates.template')
    def form(self):
        c.form = TestForm()
        c.availips = [[3,3],[2,2]]
        return dict(template='form',title='Test Form',value=None)

model:

from pylons import c

def get_ips():
    return c.availips

class TestForm(TableForm):
    action = '/test/testadd'
    submit_text = 'Add test'

    class fields(WidgetsList):
        User = TextField(label_text='FTP Username', size=40, validator=NotEmpty())
        Password = PasswordField(label_text='FTP Password', size=40, validator=NotEmpty())
        ip = SingleSelectField(label_text="Assign to IP",options=get_ips)

#!/usr/bin/python

import os,re
from string import split

dirs = os.listdir('/etc/awstats')

domainlist = {}

for dir in dirs:
  if (re.search('\.conf$',dir)):
    dom = re.sub('^awstats\.', '', dir)
    dom = re.sub('\.conf$', '', dom)
    domainlist[dom] = 1

loglist = open('/var/log/apache2/combined-access.log.1','r')
for line in loglist:
  (domain,logline) = line.split(None, 1)
  if (domain in domainlist):
    if (domainlist[domain] == 1):
      domainlist[domain] = open('/var/log/apache2/' + domain + '-access.log.1', 'w')
    domainlist[domain].write(logline)

While the code isn’t particularly earthshattering, it cut down log processing to roughly 20 minutes per day rather than the previous 16-30 hours per day.

Perl pack/unpack definitions

PING_FORMAT => ‘(a4n2N2N/a*)@245′;
TASK_FORMAT => ‘a4NIN/a*a*’;
RETR_FORMAT => ‘a4N/a*N’;
ENCPAYLOAD_FORMAT => ‘Na*’;

PHP pack/unpack definitions

define(‘TASK_FORMAT’, ‘a4NINa*a*’);
define(“ENCPAYLOAD_FORMAT”,’Na*’);

For a communications package written in perl that communicates with 32 bit and 64 bit machines that may not share the same endian structure.  The problem I’ve run into now is that Python does not support the Perl method, and, I don’t know why they didn’t at least offer some compatibility.  pack and unpack give enormous power to communication systems between machines and their support of the perl methods allowed for reasonable interoperability between the two platforms.

Python on the other hand opted to not support some of the features, which was one issue, but, their requirement is that you cannot send variable length packets.

In Python, we’re able to replicate N, network endian Long by using !L:

>>> import struct
>>> print struct.unpack(‘!L’,'\0\0\1\0′);
(256,)

However, there is no method to support a variable length payload behind that value.  We’re able to set a fixed length like 5s, but, this means that we’ve got to know the length of the payload being sent.

>>> print struct.unpack(‘!L5s’,'\0\0\1\0abcde’);
(256, ‘abcde’)

If we overstate the size of the field, Python is more than happy to tell us that the payload length doesn’t match the length of the data.

>>> print struct.unpack(‘!L8s’,'\0\0\1\0abcde’);
Traceback (most recent call last):
File “<stdin>”, line 1, in <module>
File “/usr/lib/python2.5/struct.py”, line 87, in unpack
return o.unpack(s)
struct.error: unpack requires a string argument of length 12

The cheeseshop/pypi seems to show no suitable alternative which brings up a quandry.  For this particular solution, I’ll write a wrapper function to do the heavy lifting on the two unpack strings I need to deal with and then I’ll debate pulling the perl unpack/pack routines out of the perl source and wrapping it into an .egg, possibly for distribution.