Random Musings of an Insane Mind

While I spent quite a bit of time deciphering the Graph API documentation and the OAuth guides that Facebook puts forth and submitted three documentation fixes for examples that call non-existent parameters and consequently don’t work, I came to the realization that my original layout really only works if you use a single Pylons instance per Facebook application. Since we’re focused on Don’t Repeat Yourself (DRY) Principles, some thought needs to go into things.

First, our platform needs to be designed. For this set of projects we’re going to use Nginx with uwsgi. Since we’re serving static content, we’re going to set up our directories on Nginx to allow that content to be served outside our Pylons virtual environment. Tony Landis was one of the first to provide an implementation guide for uwsgi with Nginx for Pylons which provided some of the information needed to get things working.

Our theoretical layout looks like the following:

/webroot
  |--- /static
  |--- /fb
/virtualenv
  |--- /fbappone
  |--- /fbapptwo

Later we’ll add a CDN that does origin pulls from /webroot/static. This application would have worked wonderfully with Varnish and ESI if the ESI could be compressed, but, setting up Nginx -> Varnish -> Nginx -> uwsgi seemed somewhat inefficient just to add compression. The Facebook application we’ve developed is an IFrame canvas which took roughly fifteen hours to debug after the original concept was decided. The majority of that time was spent dealing with the IFrame canvas issues. FBML was much easier to get working properly.

What we end up with is a url structure like:

http://basedomain.com/
     /static/ (xd_receiver.html, jquery support modules, CSS files)
     /fb/ (Generic facebook files, support, tos, help)
     /(fbapp)/application_one/
     /(fbapp)/application_two/

As a result of this structure, we don’t need to manipulate config/routing.py as the default’s set by Pylons map things the way we want. In the /static/ directory, we can put our CSS, js and static media files. Remember to minify the CSS and js files and combine them if possible.

Our nginx config looks like:

server {
    listen   1.2.3.4:80;
    server_name  xxxxxx.com;
    access_log /var/log/nginx/xxxxxx.com-access.log;

    location ~* (css|js|png|jpe?g|gif|ico|swf|flv)$ {
        expires max;
    }

    gzip on;
    gzip_min_length 500;
    gzip_types text/plain application/xml text/html text/javascript;
    gzip_disable "MSIE [1-6]\.";

    location ^~ /static/ {
    	alias   /var/www/xxxxxx.com/static/;
    }
    location ^~ /fb/ {
    	alias   /var/www/xxxxxx.com/fb/;
    }
    location / {
        uwsgi_pass  unix:/tmp/uwsgi.sock;
        include     uwsgi_params;
    }
}

We could modify the nginx config to pull / from the static page, but, we’re actually capturing that with a root controller that knows what applications reside below it as a directory of sorts.

We used Debian which doesn’t support uwsgi yet. A brief set of instructions follows which should work on any Debian based distribution as well:

apt-get install libxml2-dev dpkg-dev debhelper
cd /usr/src
apt-get source nginx
wget http://projects.unbit.it/downloads/uwsgi-0.9.4.4.tar.gz
tar xzf uwsgi-0.9.4.4.tar.gz
cd nginx
vi debian/rules
  add:  --add-module=/usr/src/uwsgi-0.9.4.4/nginx/ \
dpkg-buildpackage
dpkg -i ../nginx_0.7.65-5_i386.deb
mkdir /usr/local/nginx/
cp /usr/src/uwsgi-0.9.4.4/nginx/uwsgi_params /etc/nginx

/etc/nginx/uwsgi_params, add:

uwsgi_param  SCRIPT_NAME        /;

Note: I had problems with 0.9.5.1 and paster enabled wsgi applications which caused issues with Pylons.

Our uwsgi command line for development:

/usr/src/uwsgi-0.9.4.4/uwsgi -s /tmp/uwsgi.sock -C -iH /var/www/facebook/ --paste config:/var/www/facebook/fpapp/development.ini

One of the things that made Facebook integration difficult was somewhat incomplete documentation or even incorrect documentation on Facebook’s site. While the Graph API is new, it is quite a bit more powerful. While they do have official support, I think I’ll use velruse for OAuth integration next time and use the Python-SDK for the Graph API integration. See my previous post on using Pylons for a Facebook Application for a little more detailed information on how to get the application working.

While working on a maintenance script, I’ve found myself using bash a bit more than using perl for many tasks. While I don’t mind writing in perl or python, sometimes it just seems like overkill.

I needed to read one line of a /proc/loadavg for a monitoring system and was using some code that had been written by one of our programmers in 2002 or so. The old code was written using three scripts, two of which were called from cron. The first script was written in perl, actually included /proc/loadavg as chomp($value=`cat /proc/loadvg`);

Ugh!

Then some math was done, and a decision made of what values to write for nagios’s local process to report back. Suffice it to say that the math done then involved executing a shell script from time to time based on some condition checks and a separate cron that cleaned up the status for nagios.

A literal translation of the script from perl to bash ended up with a snippet of code that looked like this:

#!/bin/bash

  LOAD=`cat /proc/loadavg`
  LOADAVG=${LOAD%%.*}

echo $LOADAVG

We’ll spare the details regarding how badly the existing script converted LOADAVG to an int in perl. While this method works, it does cause a process to be forked needlessly to read /proc/loadavg.

A quick rewrite and we end up with:

#!/bin/bash

  read LOAD < /proc/loadavg
  LOADAVG=${LOAD%%.*}

echo $LOADAVG

While this doesn't seem overly bad for a process executed once a minute, it does cause a larger problem when someone copies that snippet of code and uses it elsewhere.

If we modify the code slightly to:

#!/bin/bash

for i in {1..10000}
do
  LOAD=`cat /proc/loadavg`
  LOADAVG=${LOAD%%.*}
done

echo $LOADAVG

and

#!/bin/bash

for i in {1..10000}
do
  read LOAD < /proc/loadavg
  LOADAVG=${LOAD%%.*}
done

echo $LOADAVG

and run the following benchmark:

root@tsavo:/home/cd34# time ./s.sh 
0

real	0m13.641s
user	0m0.636s
sys	0m2.304s
root@tsavo:/home/cd34# time ./t.sh 
0

real	0m0.485s
user	0m0.332s
sys	0m0.148s

we can see that there is a substantial impact to doing things.

Taking the integer portion of a floating point number also yields a number of terrible solutions involving sed, awk and cut — all forking a separate process or two.

Here’s a programmer that is saying goodbye to ORMs at Hatful of Hollow.

And another site offering a tutorial of sorts dealing with ORMs Why should you use an ORM.

While both have their points, both have missed a fundamental benefit that an ORM hands you.

Most of my development is in Pylons. Django’s ORM and template language can do the same thing. A programmer that has used PHP/Smarty to develop large scale systems will likely resist ORMs. After working with a team to develop 90k+ lines of PHP/Smarty over a six year period, making the shift required a paradigm shift.

Let’s consider the following structure. We have a cp_ticket table and a cp_ticket_detail table. A Ticket can have multiple detail records. The output we wish to have is:

ticket id, ticket header information
         ticket detail line
         ticket detail line #2
ticket id, ticket header information
         ticket detail line
         ticket detail line #2
         ticket detail line #3
ticket id, ticket header information
         ticket detail line
         ticket detail line #2

Our model:

class cp_ticket(DeclarativeBase):
    __tablename__ = 'cp_ticket'

    ticket_id = Column(mysql.MSBigInteger(20, unsigned = True), primary_key=True, autoincrement = True)
    priority = Column(mysql.MSEnum('1','2','3','4','5'), default = '3')

    ticket_detail = relation('cp_ticket_detail', order_by='cp_ticket_detail.ticket_detail_id')

class cp_ticket_detail(DeclarativeBase):
    __tablename__ = 'cp_ticket_detail'

    ticket_id = Column(mysql.MSBigInteger(20, unsigned = True), ForeignKey('cp_ticket.ticket_id'), default = '0')
    ticket_detail_id = Column(mysql.MSBigInteger(20, unsigned = True), primary_key=True, autoincrement = True)
    stamp = Column(mysql.MSTimeStamp, PassiveDefault('CURRENT_TIMESTAMP'))
    detail = Column(mysql.MSLongText, default = '')

Our query to pass to our template:

        tickets = meta.Session.query(cp_ticket).filter(cp_ticket.client_id==1).all()

Compared with the query as you would write it without an ORM:

select * from cp_ticket,cp_ticket_detail where client_id=1 and cp_ticket.ticket_id=cp_ticket_detail.ticket_id;

Both are doing the same fundamental thing, but, the ORM maps the results almost identical to the way we want to display the data. This makes template design easy.

Using Mako, we use the following code to display the results:

<table border="1">
 <tr><th>Ticket ID</th><th>Status</th><th>Detail</th></tr>
%for ticket in tmpl_context.tickets:
  <tr>
    <td><strong>${ticket.ticket_id}</strong></td>
    <td><strong>${ticket.priority}</strong></td>
  </tr>
  %for detail in ticket.ticket_detail:
  <tr>
    <td></td>
    <td>${detail.stamp}</td>
    <td>${detail.detail}</td>
  </tr>
  % endfor
% endfor
</table>

To do the same thing without using an ORM, you need to revert to a control break structure similar to the following:

current_ticket=0
for ticket in tickets:
  if (current_ticket != ticket.ticket_id):
    #new row, print the header
    print "<tr><td>first piece</td></tr>"
    current_ticket = ticket.ticket_id
  # print our detail row
  print "<tr><td></td><td>stamp and detail</td></tr>"

Control Break structures require you to be able to set a variable within your template language. Some template languages don’t allow that. If your template language (in any language) can’t do variable assignments in the template, guess where your html generation logic needs to go?

With an ORM, the template contains your display logic. Your webmaster/design team can modify the template without having to modify html contained within your code. The loops are simple to understand and designers usually have little problem avoiding the lines that start with %.

Sure, you could wrap much of this logic in your template to do the control-break structure, but, as you get more complex data, deciding how to display the data requires a define or some other functionality.

An ORM adds some insulation to the process, but, the result is a much easier page structure when displaying related data. Granted there are some performance hits and SQLAlchemy appears to create some queries that are not optimal, unless there is a tremendous performance hit, I think the benefits of the ORM for developing a web application are tremendous.

Once you move into an environment where you are dealing with multiple developers, having a defined schema with comments is much easier than using reflection to figure out what the meaning of a status field as enum(‘U’,’A’,’P’,’C’,’R’,’S’).

However, as the original poster mentions, you can do raw SQL within SQLAlchemy and do all of your work with reflection as he has done with his ORM^H^H^H, abstraction. If he’s still using SQLAlchemy, he can selectively decide when to use it and when to avoid it.

After the upgrade to Snow Leopard, mysql-python cannot be installed through easy_install.

* Install mysql’s x86_64 version from the .dmg file (I had a problem doing this when booted into the 64bit kernel, a 32bit kernel macbook had no problem) With the 64bit kernel, the system reported ‘no mountable file systems’ when trying to mount the .dmg file. A reboot into 32bit mode allowed the .dmg to be mounted and installed.
* Change into your virtual environment if desired, source bin/activate
* fetch MySQL-python-1.2.3c1

tar xzf MySQL-python-1.2.3c1.tar.gz
cd MySQL-python-1.2.3c1
ARCHFLAGS='-arch x86_64' python setup.py build
ARCHFLAGS='-arch x86_64' python setup.py install

If everything works, you should see:

$ python
Python 2.6.1 (r261:67515, Jul  7 2009, 23:51:51) 
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
>>>

Some of the possible things you’ll encounter:

After python setup.py build:

ld: warning: in build/temp.macosx-10.6-universal-2.6/_mysql.o, file is not of required architecture
ld: warning: in /usr/local/mysql/lib/libmysqlclient.dylib, file is not of required architecture
ld: warning: in /usr/local/mysql/lib/libmygcc.a, file is not of required architecture
ld: warning: in build/temp.macosx-10.6-universal-2.6/_mysql.o, file is not of required architecture
ld: warning: in /usr/local/mysql/lib/libmysqlclient.dylib, file is not of required architecture
ld: warning: in /usr/local/mysql/lib/libmygcc.a, file is not of required architecture

This means that you have the i386 version of mysql installed. Or, if you have the x86_64 version, you have didn’t include the proper ARCHFLAGS command.

ImportError: dynamic module does not define init function (init_mysql)

this means that the easy_install or the build/install process was run which tried to build a ppc/i386/x86_64 combined build.

If you see messages like:

$ python
Python 2.6.1 (r261:67515, Jul  7 2009, 23:51:51) 
[GCC 4.2.1 (Apple Inc. build 5646)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import MySQLdb
/Users/xxxxx/Python/django/lib/python2.6/site-packages/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg/_mysql.py:3: UserWarning: Module _mysql was already imported from /Users/xxxxx/Python/django/lib/python2.6/site-packages/MySQL_python-1.2.3c1-py2.6-macosx-10.6-universal.egg/_mysql.pyc, but /Users/xxxxx/Python/django/MySQL-python-1.2.3c1 is being added to sys.path
>>> 

Then you are still in the build directory. cd .. and try again.

While Google’s App Engine is a well planned service and it does work incredibly well for what it does, sometimes things break due to resource limits, etc.

While the app engine platform is still running, it appears to be an issue with this particular application’s committed resources. The App Gallery has exceeded it’s memory quota.