Making things a little more difficult to run exploits on compromised WordPress sites

September 30th, 2013

I was called in to fix a number of WordPress sites that had been hacked. Many were running older versions of WordPress and thankfully weren’t running SetUID, so, the damage was limited to exploit scripts running in some of the world writeable directories.

After cleaning up the sites, upgrading them to the latest version of WordPress and scanning for additional exploits, I added a number of rules to each of the Apache VirtualHost configs on his server.

<Directory /var/www/domain.com/wp-content/uploads/>
AllowOverride none
RemoveHandler .cgi .pl .py
<FilesMatch "\.(php|p?html?)$">
  SetHandler none
</FilesMatch>
</Directory>

<Directory /var/www/domain.com/wp-content/cache/>
AllowOverride none
RemoveHandler .cgi .pl .py
<FilesMatch "\.(php|p?html?)$">
  SetHandler none
</FilesMatch>
</Directory>

These rules need to be placed in the VirtualHost configuration and prevent PHP, cgi scripts, Perl and Python files from being executed in the two directories that WordPress is allowed to write to. To prevent other tampering, we disallow Overrides which prevents hackers from creating a directory and including their own .htaccess that would enable PHP or CGI to be parsed.

Since making these changes, we’ve seen a few files dropped into the uploads directory, but, none have been executable.

Amazon EC2 + EBS and rsync as a quick backup/mirror

August 28th, 2013

A few days ago a client came to me and asked how they could back up a lot of data on a nightly basis that only had a few changes. Many solutions with their current hosting providers were discussed, Amazon’s S3 and Glacier, but, none gave him the flexibility he was after. So, the logical conclusion here was Amazon EC2 + Elastic Block store (EBS).

We created a micro instance of Debian, logged in and added rsync, created a volume large enough to hold his backup with some room for growth and with a little command line magic he was able to mirror his data to EBS. Of course, the instance didn’t need to be running all the time so he would need to start it, run the rsync, shut it down.

After some thought I decided it would be easy enough to write a quick Python script using boto to start the instance, rsync the volume and stop the instance. If he needed access to the instance he could start it manually and log in when needed. Now, his backups could be run via cron on a regular basis.

I put the code on GitHub: https://github.com/cd34/spawncamping-octo-ninja

Most of the instance startup situations are handled and so far it seems robust.

My EC2 instance… oops

August 28th, 2013

I created an EC2 instance a while back to test a theory and had some time this evening to take a look at it again. I went to start the instance and:

    Xen Minimal OS!
  start_info: 0xac4000(VA)
    nr_pages: 0x26700
  shared_inf: 0x7de16000(MA)
     pt_base: 0xac7000(VA)
nr_pt_frames: 0x9
    mfn_list: 0x990000(VA)
   mod_start: 0x0(VA)
     mod_len: 0
       flags: 0x0
    cmd_line: root=/dev/sda1 ro 4
  stack:      0x94f860-0x96f860
MM: Init
      _text: 0x0(VA)
     _etext: 0x5ffbd(VA)
   _erodata: 0x78000(VA)
     _edata: 0x80ae0(VA)
stack start: 0x94f860(VA)
       _end: 0x98fe68(VA)
  start_pfn: ad3
    max_pfn: 26700
Mapping memory range 0xc00000 - 0x26700000
setting 0x0-0x78000 readonly
skipped 0x1000
MM: Initialise page allocator for c01000(c01000)-26700000(26700000)
MM: done
Demand map pfns at 26701000-2026701000.
Heap resides at 2026702000-4026702000.
Initialising timer interface
Initialising console ... done.
gnttab_table mapped at 0x26701000.
Initialising scheduler
Thread "Idle": pointer: 0x2026702010, stack: 0x26640000
Initialising xenbus
Thread "xenstore": pointer: 0x20267027c0, stack: 0x26650000
Dummy main: start_info=0x96f960
Thread "main": pointer: 0x2026702f70, stack: 0x26660000
"main" "root=/dev/sda1" "ro" "4" 
vbd 2049 is hd0
******************* BLKFRONT for device/vbd/2049 **********


backend at /local/domain/0/backend/vbd/3617/2049
Failed to read /local/domain/0/backend/vbd/3617/2049/feature-barrier.
Failed to read /local/domain/0/backend/vbd/3617/2049/feature-flush-cache.
16777216 sectors of 512 bytes
**************************

    [H
    [J  Booting '3.9-1-amd64'



root (hd0)

 Filesystem type is ext2fs, using whole disk

kernel /boot/vmlinuz-3.9-1-amd64 root=/dev/xvda1 ro 

initrd /boot/initrd.img-3.9-1-amd64



ERROR Invalid kernel: xc_dom_probe_bzimage_kernel: unknown compression format

xc_dom_bzimageloader.c:394: panic: xc_dom_probe_bzimage_kernel: unknown compression format
ERROR Invalid kernel: xc_dom_find_loader: no loader found

xc_dom_core.c:536: panic: xc_dom_find_loader: no loader found
xc_dom_parse_image returned -1



Error 9: Unknown boot failure



Press any key to continue...

This happens when you use a kernel compiled with .xz and the Xen Instance you’re using has the old Xen hypervisor which cannot support .xz.

What you would normally do to fix this is take another instance in the same availability zone, detach the EBS volume from the broken instance, attach the EBS volume to the other instance, make the changes to grub or put a new kernel on, detach the volume from the new instance, attach the volume to the old instance, and restart.

However, if you’re not using your own AMI, you might get the following message:

'vol-xxxxxxxx' with Marketplace codes may not be attached as a secondary device.

in which case I believe you’re stuck.

Apache2 – Using mod_rewrite and RewriteMap with a Python program to georedirect

August 2nd, 2013

Almost every machine we run has geoip loaded and clients are able to access the country code of a surfer without having to write too much code. One client decided that they wanted to redirect users from certain states to another page to deal with sales tax and shipping issues but had a mixture of perl scripts and php scripts running their shopping cart.

Having done something very similar using mod_rewrite and rewritemap with a text file, the simple solution appeared to be a short Python script to interface with the GeoIPLite data and use mod_rewrite.

In our VirtualHost config, we put:

RewriteMap statecode prg:/var/www/rewritemap/tools/rewritemap.py

To set up our Python environment, we did:

virtualenv /var/www/rewritemap
cd /var/www/rewritemap
source bin/activate
git clone https://github.com/maxmind/geoip-api-python.git
cd geoip-api-python
python setup.py install
cd ..
mkdir tools
cd tools
wget -N http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
gunzip GeoLiteCity.dat.gz

rewritemap.py

#!/var/www/rewritemap/bin/python

import sys

import GeoIP
gi = GeoIP.open("/var/www/rewritemap/tools/GeoLiteCity.dat", \
         GeoIP.GEOIP_STANDARD)

STATE = 'NULL'
gir = gi.record_by_name(sys.stdin.readline())
if gir != None:
  if gir['country_code'] == 'US':
    STATE = gir['region']

print STATE

Then, in our .htaccess we put:

RewriteEngine on
RewriteCond ${statecode:%{REMOTE_ADDR}|NONE} ^(PA|DC|CO)$
RewriteRule .* /specialpage/ [R=302,L]

Raid 1… or maybe not

July 19th, 2013

We received a client machine a while back. At some point they needed more disk space and sent us a pair of drives with instructions for mirroring data from the existing machine to the new drives.

When I looked at the drive I noticed something odd:

md4 : active raid1 sdb8[1](S) sda8[0]
      1942145856 blocks super 1.2 [1/1] [U]

While this was a Raid 1 set, notice that the indicator shows that sdb8 is a Spare (S) and that the Raid 1 set has 1 of 1 drive and isn’t broken. This usually occurs when someone is doing an in-place migration to a larger drive and creates a Raid 1 partition with a single device and forces it online with the intention of later adding the second drive. Had the primary drive failed, they would have experienced data loss.

To fix it:

mdadm --remove /dev/md4 /dev/sdb8
mdadm --grow /dev/md4 --raid-devices=2 --force
mdadm --add /dev/md4 /dev/sdb8

and we see the resulting md status:

md4 : active raid1 sda8[0]
      1942145856 blocks super 1.2 [2/1] [U_]

After the reconstruction we now have a properly configured Raid 1 set.