Random Musings of an Insane Mind

Of course, a new upgrade to Python on my devbox and the virtualenv breaks. And, merely reinstalling the virtualenv over top doesn’t work.

This is where you need to make sure your python project has an adequate setup.py to install the dependencies or you’re going to end up reinstalling by hand.

First, deactivate and make sure you’re not in the virtualenv. Get into the virtualenv’s top level directory and:


cd /virtualenvroot
rm -rf bin/ include/ lib/ local/
virtualenv /virtualenvroot
source bin/activate


Then, rerun setup.py with either install or develop and you should be set.

Almost every machine we run has geoip loaded and clients are able to access the country code of a surfer without having to write too much code. One client decided that they wanted to redirect users from certain states to another page to deal with sales tax and shipping issues but had a mixture of perl scripts and php scripts running their shopping cart.

Having done something very similar using mod_rewrite and rewritemap with a text file, the simple solution appeared to be a short Python script to interface with the GeoIPLite data and use mod_rewrite.

In our VirtualHost config, we put:

RewriteMap statecode prg:/var/www/rewritemap/tools/rewritemap.py

To set up our Python environment, we did:

virtualenv /var/www/rewritemap
cd /var/www/rewritemap
source bin/activate
git clone https://github.com/maxmind/geoip-api-python.git
cd geoip-api-python
python setup.py install
cd ..
mkdir tools
cd tools
wget -N http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
gunzip GeoLiteCity.dat.gz

rewritemap.py

#!/var/www/rewritemap/bin/python

import sys

import GeoIP
gi = GeoIP.open("/var/www/rewritemap/tools/GeoLiteCity.dat", \
         GeoIP.GEOIP_STANDARD)

STATE = 'NULL'
gir = gi.record_by_name(sys.stdin.readline())
if gir != None:
  if gir['country_code'] == 'US':
    STATE = gir['region']

print STATE

Then, in our .htaccess we put:

RewriteEngine on
RewriteCond ${statecode:%{REMOTE_ADDR}|NONE} ^(PA|DC|CO)$
RewriteRule .* /specialpage/ [R=302,L]

In Feb 2009 I started converting a PHP application over to Python and a framework. Today, I have finished all of the functional code and am at the point where I need to write a number of interfaces to the message daemon (currently a Perl script I wrote after ripping apart a C/libace daemon we had written).

The code did work in prior frameworks, I’ve moved to Pyramid, but, now I’m having to figure out why I ever used __init__ with 13 arguments. Of course everything is a wrapper around a wrapper around a single call and nothing is documented other than some sparse comments. Encrypted RPC payloads are sent to the daemon – oops, I also changed the host and key I’m testing from.

Yes, I actually am using RPC, in production, the right way.

Total Physical Source Lines of Code (SLOC) = 5,154

The penultimate 3% has added almost 200 lines of code. I suspect the last 2% adding the interfaces will add another 100 or so lines. Had I written better internal documentation, getting pulled away from the project for weeks or months at a time would have resulted in less ramp-up time when sitting back down to code. There were a few times where it would take me a few hours just to get up to speed with something I had written 18 months ago because I didn’t know what my original intentions were, or, what problem I was fixing.

Original PHP/Smarty project:

Total Physical Source Lines of Code (SLOC) = 45,040

In 2009, when I started this project, test code written resulted in roughly a 10:1 reduction in the codebase. It isn’t a truly fair comparison — the new code does more, has much better validation checks, and adds a number of features.

It’s been a long road and I’ve faced a number of challenges along the way. After Coderetreat, I’ve attempted to write testing as I’m writing code. That is a habit that I’ll have to reinforce. I don’t know that I’ll actually do Test Driven Development, but, I can see more test code being written during development, rather than sitting down with the project after it is done and writing test code. Additionally, I’m going to use Sphinx even for internal documentation.

People might question why I went with Turbogears, Pylons, and ended up with Pyramid, but, at the time I evaluated a number of frameworks, Django‘s ORM wasn’t powerful enough for some of the things I needed to do and I knew I needed to use SQLAlchemy. While Django and SQLAlchemy could be used at the time, I felt TurboGears was a closer match. As it turns out, Pyramid is just about perfect for me. Light enough that it doesn’t get in the way, heavy enough that it contains the hooks that I need to get things done.

If I wrote a framework, and I have considered it, Pyramid is fairly close to what I would end up with.

Lesson learned… document.

Today is going to be a very frustrating day wiring up stuff to classmethods that have very little documentation and buried __init__ blocks. Yes, I’ll be documenting things today.

Since Google+’s release, I’ve wanted access to an API. I’m told soon. I couldn’t wait.

#!/usr/bin/env python

import mechanize

cj = mechanize.LWPCookieJar()
cj.load("cookies.txt")

br = mechanize.Browser()
br.set_cookiejar(cj)
br.set_handle_redirect(True)
br.set_handle_referer(True)
br.set_handle_robots(False)
br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)
br.addheaders = [('User-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.810.0 Safari/535.1 cd34/0.9b')]
br.open('https://www.google.com/accounts/ServiceLogin?service=oz&passive=1209600&continue=https://plus.google.com/up/start/')

br.select_form(nr=0)

br.form.find_control("Email").readonly = False
br.form['Email'] = 'email@address.com'
br.form['Passwd'] = 'supersecretpasswordhere'

br.submit()

for l in br.links():
    print l

cj.save("cookies.txt")

While working on a project, I ran across a number of emails that didn’t properly validate using Colander. Digging into Colander’s code, the regexp used was rather basic. Chris McDonough confirmed this and said he would welcome a newer regexp.

However, FormEncode’s email validation also allows one to optionally check the DNS to see if there is a valid A or MX record – validation that might be handy. Additionally, we need to do credit card verification which is not currently included in Colander. The quick solution is to use FormEncode’s validation routines through Colander.

Our Form Schema:

def email_validate(address):
    try:
        valid = formencode.validators.Email().to_python(address)
        return True
    except Exception as message:
        return unicode(message)
        
class EmailTestSchema(colander.Schema):
    email = colander.SchemaNode(colander.String(),
        widget = deform.widget.TextInputWidget(size=60),
        validator = colander.Function(email_validate),
    )

Our view:

    @action(renderer='email_test.mako')
    def test(self):
        schema = EmailTestSchema()
        form = deform.Form(schema, buttons=('Add Email',))
        if self.request.POST:
            try:
                appstruct = form.validate(self.request.POST.items())
            except deform.ValidationFailure, e:
                return {'form':e.render()}

        return {'form':form.render()}

Now, our email validation uses FormEncode which contains some fairly detailed error messages which are passed through Colander.

After more coding, I’ll post another solution that might answer some other issues I ran into with email validation.