Archive for the ‘Web Security’ Category

Compression and Massive Logging to flatfiles for DDOS logging

Monday, January 2nd, 2012

While working with a DDOS attack that has gone on for over two years, we learn that varnishncsa is not the best logging platform out there. While Varnish does a superb job at protecting the site, the logging leaves a little to be desired. A kill/varnishncsa redirect script runs every night at midnight, logrotate compresses the files and we’re left with a big set of logfiles — logfiles that don’t represent the entire picture.

Because we’re firewalling attacker IPs, our logs only show the requests that make it through the firewall – which minimizes the data that we can collect. From a forensic analysis standpoint, that makes the collected data less valuable. As a result, we need to collect the data off a span port, but, even though it is a denial of service attack against the web, it is good to log all TCP/UDP/syn traffic on the machine to make sure we register everything.

In an ideal world, the machine should have three ethernet ports, or, you should do this monitoring from another machine, but, this is a component to the ISO I’m putting together that can be used as a front-end proxy-cache that logs the attacks. The concept is to create an ISO or USB stick installation that sets up Varnish, IPSet, this logger and the blocker that adds the rules to IPSet.

Tux, a kernel mode http accelerator, used to log to a compressed file and had a tux2w3c helper that would convert the logs to an ASCII readable format that could be processed by weblog software. Since we’re not logging the actual web request, but the TCP packet received, we have a lot more information that we can look at. Our analysis software can look for markers within that data to make decisions and send to IPSet to self-protect and self-heal through the use of expiration times on the rules.

Initially I believe the log format will look something like this:

<timestamp><attacker ip><countrycode><attacked IP><port><tcp payload>

A tool to output the logfile in an ASCII readable form will be written as well so that the data can later be analyzed. Each row will be bzip2 compressed so that the daemon can run endlessly. Logfile names will be portlog.incidentid.20120102 and won’t require rotation. I suspect it might be worthwhile to later allow the logfile to include the hour, resulting in 24 files per day.

SetUID versus www-data

Tuesday, October 25th, 2011

For years I’ve been an advocate of running apache as www-data rather than SetUID.

Quickly, an explanation of the differences:


The Apache (or alternate webserver) runs as a low privilege account, usually www-data or httpd or a similarly named user/group. When a request is served, the low privilege apache process needs to have access to read the file, which usually means that the files must be world readable, and, the directories world executable. As such, any rogue process on the machine that knows the filesystem structure could traverse the filesystem and read files, like, wp-config.php, etc. Preventing that traversal becomes difficult if one can read config files and know which domains are served from that machine. Using a predictable filesystem layout makes it easier. However, any file that is not world writeable cannot be modified by the web process. This is why an exploit running on a site is only able to write to particular directories – usually ones that are made world writeable to allow uploads.


In this case, the server takes the request, then, changes the UID to the user account that owns the files. Traversing the filesystem to find files like wp-config.php becomes difficult if the file and directory permissions are set correctly. The web process is the user, so, it is able to write to any file – just as it could with your FTP account. An exploit that is loaded now has access to modifying any file in your FTP account.

Why www-data or SetUID

While www-data has some shortcomings, SetUID is immensely more popular for two reasons: It avoids trouble tickets where people can’t understand why their application can’t upload a file to a directory, and, it protects the user’s files from being read by other people also running on that machine.

There is another mode that can be used – running the web server as www-data, but, running suPHP which spawns php processes as a particular user, while the webserver itself still runs in low privilege mode. Any PHP script still has the permission to write files on the filesystem, but, CGI/WSGI scripts would not have the ability to write to those files.

While I’m not a real fan of SetUID, one of the projects we’re working on will use it mostly to avoid the common questions regarding why this directory needs to be chmod 777 and it will cut down on the support tickets.

As a result, we need to plan for multiple generations of backups because rather than get trouble tickets regarding applications that can’t write files, we’ll be getting trouble tickets from users that have had their sites exploited and every file modified – rather than just the files in the directories that have been given permission.

I do have another theory on how to deal with this with groups. Basically, you would still run the webserver in a low privilege mode, but, it would switch to www-data.usergroup which would prevent traversal, and, the user could selectively allow group write on directories that needed it. Since usergroup would be given read access, a script running as a different user would not be able to traverse the directory since each user’s tree would be owned by their user:group.

I guess we would call this SetGID hosting.

vBulletin spam signup email addresses, combatting the problem

Thursday, March 3rd, 2011

A while back I had corresponded with Google’s spam team regarding a pattern I had discovered and sent it off to some people. It appears that they used some of that to clean up the search results removing this particular type of spam, but, the source of the problem still exists. Over the past 60 days, a particular client’s vBulletin site has received 2670 signups, over half using gmail addresses. A group of three people have independently looked at every signup to verify that these indeed do fit the spam pattern.

It appears an outsourcing company is hired to sign up, but, are given a list of email addresses that they can choose from. Signup and verification always take place from radically different IPs, so, we can assume that the people doing the actual signup have no idea that their verification email never goes out. This is confirmed by the fact that they use multiple periods in their gmail address to make the email address appear to be unique. Once we’ve determined that the email address has already been seen by modifying vBulletin to strip out the . and truncate at the +, it is instantly banned. We opted to allow the signups to be registered rather than saying that the email was in use.

A slight background to the issue. Google allows one to use a . or + in the email address which resolves to the same destination address. While I like that feature and have used it in the past, vBulletin appears to ignore this fact. So, the email address goes to the same destination as and Likewise, you can use the + in the email address to signify the source of the email. So, might signify that the email came from your twitter profile and comes from your facebook profile. Since they end up at the same place, google is a perfect way to have hundreds of email addresses that appear to be unique, but, are delivered to the same destination. This means your validation script can check fewer mailboxes, decode the validation email looking for the link, and can automatically click.

Initially our client installed Recaptcha which increases the chance that a human is probably filling out the form. Based on the number of resubmissions, I’m reasonably certain that a human is doing the data entry and they aren’t cracking Recaptcha.

I figured at one point these were created accounts, but, some of the names are so specific, one would have to assume that perhaps there are some compromised email accounts in here as well. If you glance through the list, you’ll see judicious use of the . and + to try to create unique email addresses.

The first thing we did was write a plugin that hooked into the signup process that cleaned up the email addresses. The second thing we did was look for a signup that took place in a country different from the verification click. Often times they did use proxy servers, so, using a few of the proxy dns blacklists, we were also able to make an educated guess that the signups were probably going to post spam. The first post at the board is moderated using Akismet for any that slip through, but, this method appears to be fairly good at hitting the right ones, and out of 2691 signups, it detected 2670 spam signups with 1 false positive. The false positive was a tough one – even looking at the signup data, the IP addresses used in both the signup request and validation took place in separate countries according to maxmind’s geoip database (the person signed up at work, drove home across a country border in Europe, and validated his email address from home). We also changed the registration form and put a second link above the first that said:

If you didn’t intend to sign up, click this link

For a few days, their spider was hitting the first link, banning the account for us. Often times there was a delay of a few days between an account that was validated and the first post.

If you look at the list, you can see where they have attempted to obfuscate the email address, and in some cases, are using the + to insert a counter. Based on the posts that were made, it suggests we might have more than one group actually spamming, all outsourcing the account creation to the same company.

Spammers are resourceful. It is a shame there isn’t a way to get these email addresses shut down to squelch some of the spam at the source.

Since starting this post, eight more signups came in, bringing the average to roughly 90 signups per day.

In short:

* Check a ‘cleaned’ email against the database, i.e. remove the . and truncate at the + for gmail/googlemail accounts
* Use Recaptcha
* Alter the signup form to include a link to decline the signup
* Look at the Signup IP and the validation IP
* Validate Signup IP isn’t coming from a proxy

Hackers bypass .htaccess security by using GETS rather than GET

Friday, December 10th, 2010

Last night I received an urgent message from a client. My machine has been hacked, someone got into the admin area, I need all of the details from this IP.

So, I grepped the logs, grabbed the appropriate entries and saw something odd. - - [09/Dec/2010:22:15:41 -0500] "GETS /admin/index.php HTTP/1.1" 200 3505 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv: Gecko/20101026 Firefox/3.6.12" - - [09/Dec/2010:22:17:09 -0500] "GETS /admin/usermanagement.php HTTP/1.1" 200 99320 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv: Gecko/20101026 Firefox/3.6.12" - - [09/Dec/2010:22:18:05 -0500] "GETS /admin/index.php HTTP/1.1" 200 3510 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv: Gecko/20101026 Firefox/3.6.12"

A modified snippet of the .htaccess file:

AuthUserFile .htpasswd
AuthName "Protected Area"
AuthType Basic

<Limit GET POST>
require valid-user

Of course, we know GETS isn’t valid, but, why is Apache handing out status 200s and content lengths that appear to be valid? We know the area was password protected behind .htaccess and with some quick keyboard work we’ve got a system that properly prompts for Basic Authentication with a properly formed HTTP/1.0 request. Removing the <Limit> restriction from the .htaccess protects the site, but, why are these other methods able to pass through? Replacing GETS with anything other than POST, PUT, DELETE, TRACK, TRACE, OPTIONS, HEAD results in Apache treating those requests as if GET had been typed.

Let’s set up a duplicate environment on another machine to figure out what Apache is doing.

tsavo:~ mcd$ telnet devel.mia 80
Trying x.x.x.x...
Connected to
Escape character is '^]'.
GET /htpasstest/ HTTP/1.0    

HTTP/1.1 401 Authorization Required
Date: Fri, 10 Dec 2010 21:29:58 GMT
Server: Apache
WWW-Authenticate: Basic realm="Protected Area"
Vary: Accept-Encoding
Content-Length: 401
Connection: close
Content-Type: text/html; charset=iso-8859-1

Let’s try what they did:

tsavo:~ mcd$ telnet devel.mia 80
Trying x.x.x.x...
Connected to
Escape character is '^]'.
GETS /htpasstest/ HTTP/1.0

HTTP/1.1 501 Method Not Implemented
Date: Fri, 10 Dec 2010 21:53:58 GMT
Server: Apache
Vary: Accept-Encoding
Content-Length: 227
Connection: close
Content-Type: text/html; charset=iso-8859-1

Odd, this is the behavior we expected, but, not what we are experiencing on the client’s machine. Digging a little further we look at the differences and begin to suspect the machine may have been compromised. The first thing that struck was mod_negotiation – probably not. mod_actions, maybe, but, no. DAV wasn’t loaded, but, Zend Optimizer was on the machine that appeared to have been exploited. Testing the above script on the client’s machine resulted in…. exactly the same behavior — method not supported. Testing the directory that was exploited results in the GETS request served as if it was a GET request.

So, now we’ve got a particular domain on the machine that is not behaving as the config files would suggest. A quick test on the original domain, and as expected, GETS responds with the data and bypasses the authorization clause in the .htaccess. Lets try one more test:

# telnet xxxxxx.mia 80
Trying x.x.x.x...
Connected to
Escape character is '^]'.
GETS /zend.php HTTP/1.1

HTTP/1.1 200 OK
Date: Fri, 10 Dec 2010 22:02:33 GMT
Server: Apache
Vary: Accept-Encoding
Content-Length: 602
Connection: close
Content-Type: text/html

Bingo. A Zend encoded file handed us an error 200 even though it contained an invalid request method.

The solution in this case was simple, remove the <Limit> clause from the .htaccess.

The question is, is Zend Optimizer actually doing the proper thing here. Watching Apache with gdb, Zend Optimizer does appear to hook the Apache request handler a bit higher, but why is it attempting to correct an invalid request?

One of the first rules in input validation is validate and reject on error. Never try to correct the data and accept it. If you try to correct it and make a mistake, you’re just as vulnerable and hackers will try to figure out those patterns and add extra escaping into their url request. In this case, only a few pages were able to be displayed as there were checks to make sure forms were POSTed. But, the Limit in .htaccess that should have protected the application, didn’t work as expected because the invalid methods weren’t specified.

As so many applications on the web generate .htpasswd files with the Limit clause, it makes me wonder how many Zend Encoded applications are vulnerable. Take a minute to check your systems.

When mime-type validation isn’t enough

Thursday, November 11th, 2010

Recently a client’s machine had been accessed through some holes in his application. We were given access to the source code and started to figure out how the hacker was able to get in and execute code to elevate his privileges, post financial transactions and reset accounts.

The first place we looked were places where images could be uploaded to the system as that is usually a very easy place to upload code. The file uploader checked for the presence of .gif/.jpg/.jpeg and checked the mime type, but, the check merely made sure that .jpg was contained within the filename, not that it was anchored to the right hand side. Looking through a number of directories where files could be written and be web accessible, we had a few possible locations to focus our efforts. Two sections of code were focused on and we came up with the following code:

00000000  ff d8 ff e0 00 10 4a 46  49 46 00 01 01 01 00 48  |......JFIF.....H|
00000010  00 48 00 00 ff db 00 43  00 01 01 01 01 01 01 01  |.H.....C........|
00000020  01 01 01 01 01 01 01 01  01 01 01 01 01 01 01 01  |................|
00000030  01 01 01 01 01 01 01 01  01 01 01 01 01 02 02 01  |................|
00000040  01 02 01 01 01 02 02 02  02 02 02 02 02 02 01 02  |................|
00000050  02 02 02 02 02 02 02 02  02 ff db 00 43 01 01 01  |............C...|
00000060  01 01 01 01 01 01 01 01  02 01 01 01 02 02 02 02  |................|
00000070  02 02 02 02 02 02 02 02  02 02 02 02 02 02 02 02  |................|
00000090  02 02 02 02 02 02 02 02  02 02 02 02 02 02 ff c2  |................|
000000a0  00 11 08 02 15 02 58 03  01 22 00 02 11 01 03 11  |......X.."......|
000000b0  01 ff c4 00 1e 00 00 00  06 03 01 01 00 00 00 00  |................|
000000c0  00 00 00 00 00 00 04 05  06 07 08 09 00 02 03 0a  |................|
000000d0  3c 3f 70 68 70 20 65 63  68 6f 20 22 74 65 73 74  |< ?php echo "test|
000000e0  22 3b 3f 3e 0a                                    |";?>.|

You can recreate the exploit:

head -n 1 somefile.jpg > file.jpg.php
echo '< ?php echo "hello";?>' >> file.jpg.php

The file was named file.jpg.php, uploaded through the application, the file was then written to the avatars directory and was web accessible. Since the file contained .jpg and had a proper jpeg header, it passed the two validation tests. The payload contained with the file shows ‘junk’ before the word test is printed.

A number of factors made this attack vector possible. A client could upload content that contained filenames that could be executed by .php/.cgi if they contained .gif/.jpg/.jpeg. In addition, the avatar directory (and one other) allowed execution of scripts. Using filesmatch or removing the mimetypes for anything but the static images allowed would have prevented the files from being executed.

In reality, the hole that was used was even easier to exploit as the application allowed preview of a work unit where the url wasn’t sanitized properly allowing XSS, however, this method could have been utilized.

Every time you deal with user supplied content, check, double-check and triple-check the server configuration, directory permissions, ability to traverse directories, etc. Ideally, making sure your server has minimal abilities in those directories is a step in the right direction.

Entries (RSS) and Comments (RSS).
Cluster host: li