Random Musings of an Insane Mind

    import os, sys, imp, types, tempfile, optparse
  File "/usr/lib/python2.7/tempfile.py", line 34, in 
    from random import Random as _Random
  File "/usr/lib/python2.7/random.py", line 47, in 
    from os import urandom as _urandom
ImportError: cannot import name urandom

You’re using a virtualenv, you’re probably on Debian or Ubuntu, and you upgraded python through apt-get.

To fix it:

virtualenv /path/to/your/virtualenvironment

With one of the most recent projects which clocks in at 6331 lines of code, it becomes difficult to determine whether the production and development versions are running the same version after a number of commits.

The first post I came across, Canonical Version Numbers with Git seemed like a good starting point.

The first thing you need to do is set a tag on your repository:

git tag Prod-1

and we can test to make sure our tag was set:

git describe --tags --long

Now, we create /usr/local/bin/git-revision.sh (as mentioned in one of the comments with a minor change):

#!/bin/bash
revisioncount=`git log --oneline | wc -l`
projectversion=`git describe --tags --long`
cleanversion=${projectversion%%-*}

echo "$projectversion-$revisioncount"
#echo "$cleanversion.$revisioncount"

You can use either the projectversion or the cleanversion depending on the format you prefer.

The projectversion will contain the sha snippet along with the version and looks like: 1.0-1-g8f63877-291.

The cleanversion removes the sha key leaving 1.0-291.

Now that we’ve got a version number that changes on each commit, we can use this somewhere. This version is updated AFTER the last commit, so, you’ll want to import it into your project somewhere, or, in your commit message.

Before you commit, as one of the commenters in the previous post said, execute:

git-getrevision.sh > version

For our footer.mako file, we have the following:

<footer>
${request.user.company} ${request.user.phone} ${request.user.email}
Version: 1.2.3
</footer>

We now create post-commit in $GIT/hooks:

#!/bin/bash

VERSION=`/usr/local/bin/git-revision.sh`

FOOTER='/var/www/path/to/templates/footer.mako'

CONTENTS=`cat $FOOTER`

UP_TO_VERSION="${CONTENTS%%Version:*} Version:"
CLOSE_FOOTER="</footer>${CONTENTS##*</footer>}"

echo "$UP_TO_VERSION $VERSION $CLOSE_FOOTER" > $FOOTER

You can do whatever you need with the post-commit script. I just chose to use bash to rewrite the version number present in the footer of the site. You can just output the version number to a header file that is included if that works with your system.

While working on a new project I have to come up with a database schema and variable names for use throughout the project. While some names appear humanized, there is some logic to choosing names that are more descriptive.

Consider the difference between start_date and end_date as variables. While you read the code, when you later look at the database schema, you may not make the association that they are related. date_start and date_end are probably better names.

Likewise when you’re dealing with storing the files submitted by users, thumbnail, high_res, low_res probably isn’t as descriptive as file_thumbnail, file_highres, file_lowres.

Later, when you decide to add dimensions, you can then have file_thumbnail, file_thumbnail_width and file_thumbnail_height and your database schema will be more readable.

Also, you can do the same with table names. trouble_ticket, trouble_ticket_detail and trouble_ticket_attachment will make it easier to associate than trouble_ticket, attachment_ticket, ticket_detail.

A little planning can make things much easier down the road.

I’ve written software for a number of years and I’ve used a lot of different version control systems. From the old VMS ;12 days to today, where I primarily use git.

For the last nine years, I’ve used SVN with its quirky Apache DAV setup and the stupid uid/gid issues of running svn on a development server where that Apache was also used for testing. Ok, so, that was a poor architecture choice on my part.

With Pyramid, I started to run into small issues that I knew I could fix and my early tickets consisted of

diff -Naur

output pasted into the ticket, or, telling the team what to fix. While dealing with Pyramid, I found a bug, broke down and decided to submit bug fixes the right way.

I forked it, I cloned it, I made my changes, I did my git add . and git commit, followed by my git push, then, from the web interface, created a Pull request. I do intend to figure out how to do the fork and pull without having to resort to the web interface. I don’t remember how long it took for the fix to be imported, but, it wasn’t long. It was a very minor change to make some templates XHTML compliant, but, the project lead merely had to merge my fix (if they agreed) and it was done. They didn’t have to remake the changes on their copy of the source.

Git isn’t that hard

With that newfound appreciation, I submitted a fix to Pyramid OpenID which took roughly a month to get incorporated. It was a small fix, but again, very little effort required to merge the changes in.

I’ve used GitHub, Bitbucket, code.google.com (for SVN and Git) and recently set up Gitosis with Gitweb for some private repositories. After a few months of working with git, I exported all of my local SVN repositories and imported them into git. Over the next few days, I would find a few stray projects that were quick weekend hacks that I never used VCS for, and decided to import them as well.

While I have had a mystery issue with a commit that appears from out of nowhere on GitHub that no one seems to be able to fix, overall, my experience with git has been fairly positive. Once I merge the other branch, the mystery commit should disappear, but, it is annoying having to specify the branch on every push request so that git push doesn’t submit changes to master and my branch. If I forget, I have a git command I put in a bash script that allows me to revert back to the version before that change.

Why use Version Control Software as the sole developer?

First and foremost, it is a simple, almost realtime backup (to your last commit) of your codebase. You can go back in time and look at the changes to see what changed. You can commit chunks of code as ‘save points’ so you can look back and see what has gone on. GitHub seems to be the preferred repo for Open Source projects, though, if you have five or fewer team members and need a private repo, BitBucket might fill that need since they charge per team member rather than by number of projects. With fewer than five team members, BitBucket’s private repository hosting is free. Or, if you feel like setting up Gitosis+Gitweb, you can host your own private repo on a machine where you have a shell account.

Deploying code from git is easy as well. git pull, restart apache, done. It isn’t difficult to set up multiple branches so that you have a production, staging and development branch. This allows you to fix bugs on staging and production, then push to production while having longer term additions handled on your development branch.

What about multiple users?

This is where git, or any version control software, really becomes powerful. Multiple people can work on the same codebase, changes can be merged, branches and tags can be used to do tests without affecting production code and then later merged.

How do I set up Gitosis?

I used the guide from here and had Gitosis running after 15 minutes. I tried Gitolite prior to this, but, preferred Gitosis. After a few days, I decided it was time to set up Gitweb which was fairly straightforward. If you get a 404 when viewing your gitweb root, make sure there is no trailing / on your $projectroot.

What benefit is there?

If you’re doing any development, use version control. It doesn’t matter which one, just use one. If you have multiple people on your team, absolutely use version control. It ends the ‘what did you change?’ phase of software development when something breaks. With git or any other VCS, you gain accountability. You can see who made which changes and track the evolution of a problem. Maybe you want to test a new feature and keep it separate from production – use branches or tags. Once that branch is declared complete, you can merge it with production. Even if there are modifications made to the master, you can merge those in along the way so that you’re not maintaining two codebases that require a large merge later on. Conflict resolution is a little cumbersome, but, it is much easier to keep a development branch in sync with staging/production bugfixes than it is to do a huge merge at the end of a large project. Save yourself some time when working on a new branch and merge master in frequently.

What do I use?

Mostly public projects? GitHub, BitBucket, Code.google.com, SourceForge (really, they are coming back and their offerings do include git)

A few small private projects and some public projects? BitBucket is free if you have fewer than five team members. GitHub seems somewhat costly for a small organization to have private repos. Gitosis can be run on a single account on a small VPS.

Mostly private projects? Gitosis. It took 15 minutes to get it set up and working and import the first project. It took a few days before I installed GitWeb which isn’t needed, but, is a handy tool at times to look through the commit logs.

Do I have to use git?

No, you can use svn, mercurial or git or any other version control software. You’ll find that the open source world seems to have embraced git and GitHub appears to be the most popular hosting for open source projects.

If you’re managing an open source project, the number of people familiar with git is increasing every day.

What am I missing?

The only thing I miss is the updated release or version number. I’ve not found a way to update a listed ‘release’ number that can be incorporated in file headers. That way, if I look at a production system, I can easily see what codebase it is using. I’d like to put that version number in a template. With SVN, I could set an svn-property and use $Revision$. While I’ve been manually updating the build id in a template, it would be nice to have that hook without a git add/git commit/update/git add/git commit.

For a few projects we’re using Sphinx for documentation, and, having those docs autobuild and push to the document hosting would be nice. I believe this can be done with git hooks, but, I haven’t really investigated it too much.

Version control of files that should be hidden. .gitignore doesn’t track files, but, if I want to have files tracked, but not published, I haven’t found a way to do that. I have a document push script that I don’t want to be public, but, I would like it if I could do a git pull and not have to find an old repo with a copy of that script each time. I know you can set up masking on scripts which would allow me to hide the hostname or other private parts of files – allowing me to post actual production.ini or development.ini to the repo. I find that important because documentation is usually clipped from a production file, but, when new changes are made, sometimes modifications to those files are forgotten and one needs to dig around to see what changed.

All in all, git works very well. Any version control software is a benefit. Use it.

In Feb 2009 I started converting a PHP application over to Python and a framework. Today, I have finished all of the functional code and am at the point where I need to write a number of interfaces to the message daemon (currently a Perl script I wrote after ripping apart a C/libace daemon we had written).

The code did work in prior frameworks, I’ve moved to Pyramid, but, now I’m having to figure out why I ever used __init__ with 13 arguments. Of course everything is a wrapper around a wrapper around a single call and nothing is documented other than some sparse comments. Encrypted RPC payloads are sent to the daemon – oops, I also changed the host and key I’m testing from.

Yes, I actually am using RPC, in production, the right way.

Total Physical Source Lines of Code (SLOC) = 5,154

The penultimate 3% has added almost 200 lines of code. I suspect the last 2% adding the interfaces will add another 100 or so lines. Had I written better internal documentation, getting pulled away from the project for weeks or months at a time would have resulted in less ramp-up time when sitting back down to code. There were a few times where it would take me a few hours just to get up to speed with something I had written 18 months ago because I didn’t know what my original intentions were, or, what problem I was fixing.

Original PHP/Smarty project:

Total Physical Source Lines of Code (SLOC) = 45,040

In 2009, when I started this project, test code written resulted in roughly a 10:1 reduction in the codebase. It isn’t a truly fair comparison — the new code does more, has much better validation checks, and adds a number of features.

It’s been a long road and I’ve faced a number of challenges along the way. After Coderetreat, I’ve attempted to write testing as I’m writing code. That is a habit that I’ll have to reinforce. I don’t know that I’ll actually do Test Driven Development, but, I can see more test code being written during development, rather than sitting down with the project after it is done and writing test code. Additionally, I’m going to use Sphinx even for internal documentation.

People might question why I went with Turbogears, Pylons, and ended up with Pyramid, but, at the time I evaluated a number of frameworks, Django‘s ORM wasn’t powerful enough for some of the things I needed to do and I knew I needed to use SQLAlchemy. While Django and SQLAlchemy could be used at the time, I felt TurboGears was a closer match. As it turns out, Pyramid is just about perfect for me. Light enough that it doesn’t get in the way, heavy enough that it contains the hooks that I need to get things done.

If I wrote a framework, and I have considered it, Pyramid is fairly close to what I would end up with.

Lesson learned… document.

Today is going to be a very frustrating day wiring up stuff to classmethods that have very little documentation and buried __init__ blocks. Yes, I’ll be documenting things today.