Random Musings of an Insane Mind

Recently I’ve been refactoring a lot of code and I’m seeing a few trends that I find easier to read.

With imports, I prefer putting each import on its own line. It takes up more screenspace, but, when looking at a commit diff, I can see what was added or removed much more easily than a string changed in the middle (though, more of the diff tools are showing inline differences).

However, what I started to do recently was do imports like:

from module import blah, \
                   blah2, \
                   blah3

I keep them in alphabetical order, but, I find that reading through that removes the ‘wall of text’ effect.

Original imports:

from module import blah
from module import blah2
from module import blah3

I find that the new method I’m using allows me to more easily see that two imports came from the same module.

Another possibility as mentioned by Chris McDonough is:

from module import (blah,
                    blah2,
                    blah3)

Have a client machine that is a little loaded that has a ton of modified files. Normally we just restore off the last backup or the previous generation backup, but, over 120k files since June 2011 have been exploited. Since the machine is doing quite a bit of work, we need to throttle our replacements so that we don’t kill the server.

#!/usr/bin/python
"""

Quick search and replace to replace an exploit on a client's site while
trying to keep the load disruption on the machine to a minimum.

Replace the variable exploit with the code to be replaced. By default, 
this script starts at the current directory. max_load controls our five
second sleep until the load drops.

"""

import glob
import os
import re
import time

path = '.'
max_load = 10

exploit = """
<script>var i,y,x="3cblahblahblah3e";y='';for(i=0;i
""".strip()

file_exclude = re.compile('\.(gif|jpe?g|swf|css|js|flv|wmv|mp3|mp4|pdf|ico|png|zip)$', \
                          re.IGNORECASE)

def check_load():
    load_avg = int(os.getloadavg()[0])
    while load_avg > max_load:
        time.sleep(30)
        load_avg = int(os.getloadavg()[0])

def getdir(path):
    check_load()
    for file in os.listdir(path):
        file_path = os.path.join(path,file)
        if os.path.isdir(file_path):
            getdir(file_path)
        else:
            if not file_exclude.search(file_path):
                process_file(file_path)

def process_file(file_path):
    file = open(file_path, 'r+')
    contents = file.read()
    if exploit in contents:
        print 'fixing:', file_path
        contents = contents.replace(exploit, '')
        file.truncate(0)
        file.seek(0, os.SEEK_SET )
        file.write(contents)
    file.close()

getdir(path)

Thankfully, since this server is run as www-data rather than SetUID, the damage wasn’t as bad as it could have been.

After quite a bit of work we’ve finally gotten Pyramid Apex to a point where I can deploy it on two production apps to make sure things are working as I expect they should.

If you’re developing a Pyramid Application and are using Authentication/Authorization, I18N/L10N, Flash Messages and a Form Library, take a look at Pyramid Apex, a library Matthew Housden and I wrote to make it easier to quickly develop Pyramid applications.

It supports OpenID, Local authentication storage using bcrypt and a number of other basic features.

Since Google+’s release, I’ve wanted access to an API. I’m told soon. I couldn’t wait.

#!/usr/bin/env python

import mechanize

cj = mechanize.LWPCookieJar()
cj.load("cookies.txt")

br = mechanize.Browser()
br.set_cookiejar(cj)
br.set_handle_redirect(True)
br.set_handle_referer(True)
br.set_handle_robots(False)
br.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)
br.addheaders = [('User-agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/535.1 (KHTML, like Gecko) Chrome/14.0.810.0 Safari/535.1 cd34/0.9b')]
br.open('https://www.google.com/accounts/ServiceLogin?service=oz&passive=1209600&continue=https://plus.google.com/up/start/')

br.select_form(nr=0)

br.form.find_control("Email").readonly = False
br.form['Email'] = 'email@address.com'
br.form['Passwd'] = 'supersecretpasswordhere'

br.submit()

for l in br.links():
    print l

cj.save("cookies.txt")

After working on a Pylons project for a week or so, there was a minor part of it that I felt didn’t need the complexity of a framework. Some quick benchmarking of the most minimal Pylons/SQLAlchemy project I could muster came in around 200 requests per second which put me at roughly 12 million requests per day based on the typical curve.

Within 15 minutes of installing Tornado and using their simple hello world example, I imported SQLAlchemy and ended up boosting this to 280 requests per second. As I really didn’t need any of the features from the ORM, I decided to use tornado.database which isn’t much more than a bare wrapper to python-mysql. Even with a single worker process, I was able to get 870 requests per second. 56 million requests per day, without any tuning?

I’m reasonably impressed. Once I put it on production hardware, I’m thinking I’ll easily be able to count on double those numbers if not more.

Next weekend, Traffic Server.