Random Musings of an Insane Mind

My server backup script broke a while back – probably around Dec 14th, 2014 with a python update that debian pushed which broke my virtualenv. This isn’t the first time debian has broken virtualenvs and my last post was about this. In addition the backup script filled up the backup drive without triggering an exception which is odd, because the source didn’t exceed the size of the destination drive even prior to things breaking. The script just does a simple rsync so it wouldn’t have duplicated symlinks.

It finished its backup last night in about 30 minutes (usually takes 5-7 minutes) and now has 6gb free which matches my server.

I recognized my backups were failing because debian pushed another change where varnish overwrote the startup script without asking – and I had to reconstruct that file from the machine configuration.

At times, I wonder why OS upgrades push breaking changes without any mention. The conversion to systemd was also

Of course, a new upgrade to Python on my devbox and the virtualenv breaks. And, merely reinstalling the virtualenv over top doesn’t work.

This is where you need to make sure your python project has an adequate setup.py to install the dependencies or you’re going to end up reinstalling by hand.

First, deactivate and make sure you’re not in the virtualenv. Get into the virtualenv’s top level directory and:


cd /virtualenvroot
rm -rf bin/ include/ lib/ local/
virtualenv /virtualenvroot
source bin/activate


Then, rerun setup.py with either install or develop and you should be set.

Another update to Python 2.7 and the error:

ImportError: cannot import name MAXREPEAT

occurs when starting an app in the virutalenv.

To fix this, just run:

virtualenv /path/to/existingvirtualenv

    import os, sys, imp, types, tempfile, optparse
  File "/usr/lib/python2.7/tempfile.py", line 34, in 
    from random import Random as _Random
  File "/usr/lib/python2.7/random.py", line 47, in 
    from os import urandom as _urandom
ImportError: cannot import name urandom

You’re using a virtualenv, you’re probably on Debian or Ubuntu, and you upgraded python through apt-get.

To fix it:

virtualenv /path/to/your/virtualenvironment

In Feb 2009 I started converting a PHP application over to Python and a framework. Today, I have finished all of the functional code and am at the point where I need to write a number of interfaces to the message daemon (currently a Perl script I wrote after ripping apart a C/libace daemon we had written).

The code did work in prior frameworks, I’ve moved to Pyramid, but, now I’m having to figure out why I ever used __init__ with 13 arguments. Of course everything is a wrapper around a wrapper around a single call and nothing is documented other than some sparse comments. Encrypted RPC payloads are sent to the daemon – oops, I also changed the host and key I’m testing from.

Yes, I actually am using RPC, in production, the right way.

Total Physical Source Lines of Code (SLOC) = 5,154

The penultimate 3% has added almost 200 lines of code. I suspect the last 2% adding the interfaces will add another 100 or so lines. Had I written better internal documentation, getting pulled away from the project for weeks or months at a time would have resulted in less ramp-up time when sitting back down to code. There were a few times where it would take me a few hours just to get up to speed with something I had written 18 months ago because I didn’t know what my original intentions were, or, what problem I was fixing.

Original PHP/Smarty project:

Total Physical Source Lines of Code (SLOC) = 45,040

In 2009, when I started this project, test code written resulted in roughly a 10:1 reduction in the codebase. It isn’t a truly fair comparison — the new code does more, has much better validation checks, and adds a number of features.

It’s been a long road and I’ve faced a number of challenges along the way. After Coderetreat, I’ve attempted to write testing as I’m writing code. That is a habit that I’ll have to reinforce. I don’t know that I’ll actually do Test Driven Development, but, I can see more test code being written during development, rather than sitting down with the project after it is done and writing test code. Additionally, I’m going to use Sphinx even for internal documentation.

People might question why I went with Turbogears, Pylons, and ended up with Pyramid, but, at the time I evaluated a number of frameworks, Django‘s ORM wasn’t powerful enough for some of the things I needed to do and I knew I needed to use SQLAlchemy. While Django and SQLAlchemy could be used at the time, I felt TurboGears was a closer match. As it turns out, Pyramid is just about perfect for me. Light enough that it doesn’t get in the way, heavy enough that it contains the hooks that I need to get things done.

If I wrote a framework, and I have considered it, Pyramid is fairly close to what I would end up with.

Lesson learned… document.

Today is going to be a very frustrating day wiring up stuff to classmethods that have very little documentation and buried __init__ blocks. Yes, I’ll be documenting things today.