Apache2 – Using mod_rewrite and RewriteMap with a Python program to georedirect

August 2nd, 2013

Almost every machine we run has geoip loaded and clients are able to access the country code of a surfer without having to write too much code. One client decided that they wanted to redirect users from certain states to another page to deal with sales tax and shipping issues but had a mixture of perl scripts and php scripts running their shopping cart.

Having done something very similar using mod_rewrite and rewritemap with a text file, the simple solution appeared to be a short Python script to interface with the GeoIPLite data and use mod_rewrite.

In our VirtualHost config, we put:

RewriteMap statecode prg:/var/www/rewritemap/tools/rewritemap.py

To set up our Python environment, we did:

virtualenv /var/www/rewritemap
cd /var/www/rewritemap
source bin/activate
git clone https://github.com/maxmind/geoip-api-python.git
cd geoip-api-python
python setup.py install
cd ..
mkdir tools
cd tools
wget -N http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
gunzip GeoLiteCity.dat.gz

rewritemap.py

#!/var/www/rewritemap/bin/python

import sys

import GeoIP
gi = GeoIP.open("/var/www/rewritemap/tools/GeoLiteCity.dat", \
         GeoIP.GEOIP_STANDARD)

STATE = 'NULL'
gir = gi.record_by_name(sys.stdin.readline())
if gir != None:
  if gir['country_code'] == 'US':
    STATE = gir['region']

print STATE

Then, in our .htaccess we put:

RewriteEngine on
RewriteCond ${statecode:%{REMOTE_ADDR}|NONE} ^(PA|DC|CO)$
RewriteRule .* /specialpage/ [R=302,L]

Raid 1… or maybe not

July 19th, 2013

We received a client machine a while back. At some point they needed more disk space and sent us a pair of drives with instructions for mirroring data from the existing machine to the new drives.

When I looked at the drive I noticed something odd:

md4 : active raid1 sdb8[1](S) sda8[0]
      1942145856 blocks super 1.2 [1/1] [U]

While this was a Raid 1 set, notice that the indicator shows that sdb8 is a Spare (S) and that the Raid 1 set has 1 of 1 drive and isn’t broken. This usually occurs when someone is doing an in-place migration to a larger drive and creates a Raid 1 partition with a single device and forces it online with the intention of later adding the second drive. Had the primary drive failed, they would have experienced data loss.

To fix it:

mdadm --remove /dev/md4 /dev/sdb8
mdadm --grow /dev/md4 --raid-devices=2 --force
mdadm --add /dev/md4 /dev/sdb8

and we see the resulting md status:

md4 : active raid1 sda8[0]
      1942145856 blocks super 1.2 [2/1] [U_]

After the reconstruction we now have a properly configured Raid 1 set.

Varnish and Node.js

July 18th, 2013

While working with a client installation they wanted to run Varnish in front of their node.js powered site to eliminate having node serve the static assets. Socket.io uses HTTP/1.0 and cannot be cached. Minimally these few lines can be added to their respective functions and things will work. Obviously you’ll want to set expires on your static assets, strip cookies where possible, etc.

sub vcl_recv {
    if (req.url ~ "socket.io/") {
      return (pipe);
    }
}

sub vcl_pipe {
    if (req.http.upgrade) {
        set bereq.http.upgrade = req.http.upgrade;
    }
    return (pipe);
}

Tested with: Varnish 3.0.4, Node.js v0.10.13

From Dedicated Server to KVM Virtualized Instance

June 23rd, 2013

Recently we’ve had a few clients that have wanted to downsize from a dedicated server to a Private Virtual Server running a virtualized instance. In the past, that has been a very time consuming process. While it would probably be better to upgrade these clients to 64bit and take advantage of a fresh OS load, sometimes there are issues that preclude this. One potential problem is not having a recent enough kernel to work with KVM if you are using a kernel that was built specifically for your setup.

What follows is a general recipe for migrating these machines. Most of the tasks are ‘hurry up and wait’ and the actual work involved with the move is typically waiting for data to be moved. This guide starts from bare metal to migrating the first machine.

On the KVM box

  • install Linux
  • install kvm, virt-tools, and any base utilities (our minimal install includes libvirt-bin qemu-kvm virtinst rsync bridge-utils rsync cgroup-bin nvi)
  • Download the .iso for your initial build
  • Determine disk size for each instance
  • Install base image to COW file or LVM partition. From this, you’ll clone it to your new instances. Make sure the COW file is created with the same size as the resulting image, or, you’ll need to resize via LVM and your underlying filesystem.
  • pvcreate /dev/sda2 (or /dev/md1 if you use software rather than hardware raid)
  • vgcreate -s 16M vg0 /dev/sda2
  • lvcreate -L 80G -n c1 vg0
  • virt-clone –original base_image –name newvirtualmachine –file=/dev/vg0/c1

On the machine being moved

  • Install grub2 (if not already running grub2)
  • Edit /etc/default/grub and disable UUID
  • upgrade-grub
  • Install new kernel
  • Paths changing? /etc/fstab should be modified or needs to be done via vnc. If done via VNC, note that the machine may come up in singleuser mode as it will fail the fsck on devices that may not be present.
  • rsync (logs needed?)

Ready for the switch

On dedicated server

  • secondary network helpful, ipv6 on primary interface works
  • ifconfig primary interface to temporary IP, add default route
  • restart firewall (if pinned to primary ethernet)
  • log out, log back in using temporary IP
  • remove ipv6

On KVM Machine

  • virsh start newinstancename
  • connect via vnc
  • clear arp on your routers
  • dpkg-reconfigure grub-pc (sometimes, grub is not recognized on QEMU hard drive)
  • verify swap (double check /etc/fstab)

after grub is reinstalled, reboot just to ensure machine comes up with no issues

Some Tools

KVM Shell Scripts used when we migrated a number of machines.

Kernel Config Notes for KVM

If you are building your own kernels, here are some notes.

Make sure the following are installed in your KVM kernel

  • High Res Timers (required for most of the virtualization options)
  • CPU Task/Time accounting (if desired)
  • Bridge
  • VIRTIO_NET
  • CONFIG_VIRTIO_BLK
  • SCSI_VIRTIO
  • VIRTIO_CONSOLE
  • CONFIG_VIRTIO
  • VIRTIO_PCI
  • VIRTIO_BALLOON
  • VIRTIO_MMIO
  • VIRTIO_MMIO_CMDLINE_DEVICES

Make sure the following are installed in your guest kernel

  • sym53c8xx
  • virtio-pci
    PIIX_IDE

Remember that your guest kernel is running on the underlying hardware of your KVM machine. The guest kernel should have the CPU type set based on the KVM’s CPU type to take advantage of any hardware optimizations the CPU may have.

ImportError: cannot import name MAXREPEAT

May 24th, 2013

Another update to Python 2.7 and the error:

ImportError: cannot import name MAXREPEAT

occurs when starting an app in the virutalenv.

To fix this, just run:

virtualenv /path/to/existingvirtualenv

Entries (RSS) and Comments (RSS).
Cluster host: li