Random Musings of an Insane Mind

Last night I received an urgent message from a client. My machine has been hacked, someone got into the admin area, I need all of the details from this IP.

So, I grepped the logs, grabbed the appropriate entries and saw something odd.

1.2.3.4 - - [09/Dec/2010:22:15:41 -0500] "GETS /admin/index.php HTTP/1.1" 200 3505 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12"
1.2.3.4 - - [09/Dec/2010:22:17:09 -0500] "GETS /admin/usermanagement.php HTTP/1.1" 200 99320 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12"
1.2.3.4 - - [09/Dec/2010:22:18:05 -0500] "GETS /admin/index.php HTTP/1.1" 200 3510 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12"

A modified snippet of the .htaccess file:

AuthUserFile .htpasswd
AuthName "Protected Area"
AuthType Basic

<Limit GET POST>
require valid-user
</Limit>

Of course, we know GETS isn’t valid, but, why is Apache handing out status 200s and content lengths that appear to be valid? We know the area was password protected behind .htaccess and with some quick keyboard work we’ve got a system that properly prompts for Basic Authentication with a properly formed HTTP/1.0 request. Removing the <Limit> restriction from the .htaccess protects the site, but, why are these other methods able to pass through? Replacing GETS with anything other than POST, PUT, DELETE, TRACK, TRACE, OPTIONS, HEAD results in Apache treating those requests as if GET had been typed.

Let’s set up a duplicate environment on another machine to figure out what Apache is doing.

tsavo:~ mcd$ telnet devel.mia 80
Trying x.x.x.x...
Connected to xxxxxxx.xxx.
Escape character is '^]'.
GET /htpasstest/ HTTP/1.0    

HTTP/1.1 401 Authorization Required
Date: Fri, 10 Dec 2010 21:29:58 GMT
Server: Apache
WWW-Authenticate: Basic realm="Protected Area"
Vary: Accept-Encoding
Content-Length: 401
Connection: close
Content-Type: text/html; charset=iso-8859-1

Let’s try what they did:

tsavo:~ mcd$ telnet devel.mia 80
Trying x.x.x.x...
Connected to xxxxxxx.xxx.
Escape character is '^]'.
GETS /htpasstest/ HTTP/1.0

HTTP/1.1 501 Method Not Implemented
Date: Fri, 10 Dec 2010 21:53:58 GMT
Server: Apache
Allow: GET,HEAD,POST,OPTIONS,TRACE
Vary: Accept-Encoding
Content-Length: 227
Connection: close
Content-Type: text/html; charset=iso-8859-1

Odd, this is the behavior we expected, but, not what we are experiencing on the client’s machine. Digging a little further we look at the differences and begin to suspect the machine may have been compromised. The first thing that struck was mod_negotiation – probably not. mod_actions, maybe, but, no. DAV wasn’t loaded, but, Zend Optimizer was on the machine that appeared to have been exploited. Testing the above script on the client’s machine resulted in…. exactly the same behavior — method not supported. Testing the directory that was exploited results in the GETS request served as if it was a GET request.

So, now we’ve got a particular domain on the machine that is not behaving as the config files would suggest. A quick test on the original domain, and as expected, GETS responds with the data and bypasses the authorization clause in the .htaccess. Lets try one more test:

# telnet xxxxxx.mia 80
Trying x.x.x.x...
Connected to xxxxxx.xxx.
Escape character is '^]'.
GETS /zend.php HTTP/1.1
Host: xxxxxx.xxx

HTTP/1.1 200 OK
Date: Fri, 10 Dec 2010 22:02:33 GMT
Server: Apache
Vary: Accept-Encoding
Content-Length: 602
Connection: close
Content-Type: text/html

Bingo. A Zend encoded file handed us an error 200 even though it contained an invalid request method.

The solution in this case was simple, remove the <Limit> clause from the .htaccess.

The question is, is Zend Optimizer actually doing the proper thing here. Watching Apache with gdb, Zend Optimizer does appear to hook the Apache request handler a bit higher, but why is it attempting to correct an invalid request?

One of the first rules in input validation is validate and reject on error. Never try to correct the data and accept it. If you try to correct it and make a mistake, you’re just as vulnerable and hackers will try to figure out those patterns and add extra escaping into their url request. In this case, only a few pages were able to be displayed as there were checks to make sure forms were POSTed. But, the Limit in .htaccess that should have protected the application, didn’t work as expected because the invalid methods weren’t specified.

As so many applications on the web generate .htpasswd files with the Limit clause, it makes me wonder how many Zend Encoded applications are vulnerable. Take a minute to check your systems.

While surfing Youtube a while back, I noticed that you could view the statistics for a given video. While most of the videos I view are quite boring and have low viewcounts, I thought that might be the trigger — Only popular videos have stats. However, while surfing Youtube today to see how they handled some statistics, I saw some patterns emerge that tossed that theory out the window. Videos with even a few hundred views had statistics.

Since we can assume that Google has kept track of every view and statistic possible since it was merged with their platform, even old videos have data back into late 2007 as evidenced by many different videos. Some videos mention 30 Nov 2007 as the earliest data collection date.

So, we face a quandary. We have videos from 2005 through today, stats from late 2007 through today and stats displayed on the video display page that have been rolled out since mid 2010. Old videos that don’t currently display stats obviously are gathering stats but must have a flag saying that the old data hasn’t been imported as it will only mention Honors for this Video. How do you approach the problem?

We know that the data is collected and applied in batches and it appears that every video has statistics from a particular date forward. Recent videos all have full statistics, even with a few hundred views, no comments, no favorites. The catalyst doesn’t appear to be when someone has an interaction with a video, merely viewing a video must signal the system to backfill statistics. There is probably some weight given to popular videos, though, those videos would have a lot more history. One must balance the time required to import a very popular video versus importing the history from hundreds of less popular videos. One of the benefits of bigtable – if architected properly – would be to process each video’s history in one shot, set the stats processed flag and do the next video. One might surmise that Google knew to collect the view data, but, may not have thought about how the data would be used.

How do you choose videos to be processed? When you process the live views, you might decide to put a video into a queue for backfill processing. But, on a very lightly viewed video, this might delay backfilling another video where statistics might be more interesting or provocative. We can assume that we have a fixed date in time where a video doesn’t require backfilling which makes our data backfill decision a little easier.

As the logs are being processed, we might keep a list of the video_id, creation date and number of daily views. That data would be inserted into a backfill queue for our backfill process. In the backfill process, we would look at the creation date, number of daily views and number of mentions in the backfill queue. To figure out a priority list of the items to process, we might look at the velocity of hits from one day to the next – triggering a job queue entry on a video that is suddenly getting popular. We might also influence decisions based on the views and the creation date delta off the fixed point in time where stat displays started. This would allow us to take a lightly viewed video that was created just before our fixed point and prioritize that in the backfill queue. Now we’ve got a dual priority system that would allow us to tackle two problems at the same time, and intersect in the middle. Each day, new entries are inserted into the queue, altering priority of existing and current entries which would allow the stats to be backfilled in a manner that would appear to be very proactive.

At some point, videos that haven’t been viewed that were created prior to the fixed point in time could be added to the cleanup queue. Since they weren’t viewed, generating the statistics for them isn’t as important. And, if a video has been viewed, it was already in the queue. Since the queue could dispatch the jobs to as many machines as Google wanted, stats could be added to Youtube videos based on the load of their distributed computer.

What do you think?

How would you backfill log data from an estimated 900 million videos, serving 2 billion video views a week.

My first attempt at migrating a Pylons project to Pyramid was accomplished without too much difficulty. That project was relatively small, however, this project hasn’t been put into production.

A brief history of this application:

We have a legacy PHP application which consists of 78k lines of code. Originally we started rewriting the application in Turbogears 2.0 and later moved over to Pylons with ToscaWidgets. We used ToscaWidgets because all of our forms had been written for TG2. The application never made it into production, but, was substantial enough that we felt it would be a good test to convert it to Pyramid since we’re focusing our development efforts on Pyramid.

Initially I started to write a script to do the migration from Pylons to Pyramid and from ToscaWidgets to Deform but abandoned that. Changing the controllers from Pylons to Pyramid was fairly easy.

Most of the pylons imports were commented out and replaced with:

import deform
import colander

from pyramid.response import Response
from pyramid.view import action
from pyramid.security import authenticated_userid
from pyramid.threadlocal import get_current_request
import webhelpers.paginate as paginate

References to tmpl_context. were altered, our __before__ action for authkit was removed and replaced with an __init__ since we were using handlers. Routes were modified and given unique names, templates were modified to remove references to ${h. and ${tmpl_context., and we began the process of stepping through the application. Initially I had written a script to convert the ToscaWidgets form models over to Deform schemas, but, after two hours, it became obvious it would take more time to do that than to manually recreate the forms. After working through much of the process, I’m debating whether this project should have been migrated over to FormAlchemy as almost every form is a duplicate of the SQL schema. Since it isn’t in production, we still have some time to make that decision.

What follows is a summary of the errors received and what caused the errors.

TypeError: ‘NoneType’ object is not iterable

I ran into this while importing some routes from an existing project. Since routes must be uniquely named in Pyramid, a route with the same name will replace a prior route. Some of the route names have gotten quite unwieldy.

TypeError: object.__new__() takes no parameters

Solution, add the __init__ block to your class in your handler.

class YourClass(object):
    def __init__(self, request):
        self.request = request

ValueError: Non-response object returned from view named (and no renderer): {‘template’: ‘billing_index’}

Turbogears 2.0 code:

    def index(self, **kw):
        return dict(template='billing_index')

modify to:

    @action(renderer='billing_index.mako', permission='client')
    def index(self, **kw):
        return {}

AttributeError: ‘Undefined’ object has no attribute ‘form’

Pyramid doesn’t pass the tmpl_context., c. or h. through to the template, so, the return values of each of the actions needs to pass the fields required. As a result, any template code that replies on these globals needs to be modified, and the corresponding action needs to return the values in the dictionary. You can use one of the pylons templates which will instantiate a subscriber method to replicate the Pylons globals if you want.

AttributeError: ‘Undefined’ object has no attribute ‘literal’

Existing forms are relying on the pylons h. global.

${h.literal(form(value=value))}

UnboundLocalError: local variable ‘clients’ referenced before assignment

This occurs when you remove tmpl_context. and are left with a variable assigned that matches the class. For example:

tmpl_context.clients = meta.Session.query(clients).filter_by(client_id==1).all()

When tmpl_context. is removed, the class clients is cast incorrectly.

AttributeError: ‘Undefined’ object has no attribute ‘pager’

Paginate itemset isn’t being passed to the template.

TypeError: ‘NoneType’ object is not iterable

Paginate is getting the result set, rather than the paginate set passed in the return dictionary.

NameError: global name ‘request’ is not defined

request.matchdict/request.params -> self.request.matchdict/self.request.params

In a handler, request. is referred to as self.request. If you’ve converted things over to a handler rather than writing the individual routes for each action, you’ll need to preface any request. with self.

NotImplementedError: no URL generator available

Webhelpers pagination doesn’t know how to generate URLs in Pyramid, but, you can use a callable to generate the URLs. This requires access to pyramid.threadlocal which is generally not recommended, but, does allow you to use paginate until a pyramid compatible paginate is written.

The generator you need looks like this:

from webhelpers.util import update_params
from pyramid.threadlocal import get_current_request

def get_page_url(**kw):
    return update_params(get_current_request().path_info, **kw)

In your paginate block, you need to add the following:

url=get_page_url,

        paginator = paginate.Page(
            features,
            page=int(self.request.params.get('page', 1)),
            items_per_page = 40,
            url=get_page_url,
        )

RuntimeError: Caught exception rendering template. TypeError: ‘int’ object is not iterable

deform select widget requires tuple for the dropdown creation. Toscawidgets would build the right hand side if it was passed a list of IDs.

TypeError: ‘Undefined’ object is unsubscriptable

Returning a dict, removing tmpl_context., a ${value[‘asdf’]} is missing ‘value’:value being passed in the return dict.

TypeError: sequence item 10: expected string or Unicode, Undefined found

When using deform, return {‘form’:form.render()} rather than return {‘form’:form}.

Summary

This is the second application I’ve converted from Pylons 1.0 to Pyramid and most of the issues have been syntax issues. Hopefully the error summary above will save someone some time.

Recently a client’s machine had been accessed through some holes in his application. We were given access to the source code and started to figure out how the hacker was able to get in and execute code to elevate his privileges, post financial transactions and reset accounts.

The first place we looked were places where images could be uploaded to the system as that is usually a very easy place to upload code. The file uploader checked for the presence of .gif/.jpg/.jpeg and checked the mime type, but, the check merely made sure that .jpg was contained within the filename, not that it was anchored to the right hand side. Looking through a number of directories where files could be written and be web accessible, we had a few possible locations to focus our efforts. Two sections of code were focused on and we came up with the following code:

00000000  ff d8 ff e0 00 10 4a 46  49 46 00 01 01 01 00 48  |......JFIF.....H|
00000010  00 48 00 00 ff db 00 43  00 01 01 01 01 01 01 01  |.H.....C........|
00000020  01 01 01 01 01 01 01 01  01 01 01 01 01 01 01 01  |................|
00000030  01 01 01 01 01 01 01 01  01 01 01 01 01 02 02 01  |................|
00000040  01 02 01 01 01 02 02 02  02 02 02 02 02 02 01 02  |................|
00000050  02 02 02 02 02 02 02 02  02 ff db 00 43 01 01 01  |............C...|
00000060  01 01 01 01 01 01 01 01  02 01 01 01 02 02 02 02  |................|
00000070  02 02 02 02 02 02 02 02  02 02 02 02 02 02 02 02  |................|
*
00000090  02 02 02 02 02 02 02 02  02 02 02 02 02 02 ff c2  |................|
000000a0  00 11 08 02 15 02 58 03  01 22 00 02 11 01 03 11  |......X.."......|
000000b0  01 ff c4 00 1e 00 00 00  06 03 01 01 00 00 00 00  |................|
000000c0  00 00 00 00 00 00 04 05  06 07 08 09 00 02 03 0a  |................|
000000d0  3c 3f 70 68 70 20 65 63  68 6f 20 22 74 65 73 74  |< ?php echo "test|
000000e0  22 3b 3f 3e 0a                                    |";?>.|
000000e5

You can recreate the exploit:

head -n 1 somefile.jpg > file.jpg.php
echo '< ?php echo "hello";?>' >> file.jpg.php

The file was named file.jpg.php, uploaded through the application, the file was then written to the avatars directory and was web accessible. Since the file contained .jpg and had a proper jpeg header, it passed the two validation tests. The payload contained with the file shows ‘junk’ before the word test is printed.

A number of factors made this attack vector possible. A client could upload content that contained filenames that could be executed by .php/.cgi if they contained .gif/.jpg/.jpeg. In addition, the avatar directory (and one other) allowed execution of scripts. Using filesmatch or removing the mimetypes for anything but the static images allowed would have prevented the files from being executed.

In reality, the hole that was used was even easier to exploit as the application allowed preview of a work unit where the url wasn’t sanitized properly allowing XSS, however, this method could have been utilized.

Every time you deal with user supplied content, check, double-check and triple-check the server configuration, directory permissions, ability to traverse directories, etc. Ideally, making sure your server has minimal abilities in those directories is a step in the right direction.

For one of our projects we used Pylons 1.0 with Repoze.who/Repoze.what, though, we only used groups without permissions. In our application, a member could access one of three controllers based on their membership in a group. There are a number of methods that can be used to set up authentication. One method is to create a thin table with the Primary Key, Username and Password and use an association table to add any additional information. This has the ability to be flexible when you need to add a number of fields that shouldn’t be contained within the AuthUser table. Another method is to use the AuthUser table to hold the related information. This example uses the latter method.

The first changes made are to __init__.py to add the policies and Forbidden view. This view allows us to prompt a user for a username/password when they visit a page that is protected by the ACLs in Pyramid.

auth.py contains our authentication model with the Permission models removed. We put the login/logout views, forbidden view along with the groupfinder and the RootFactory in login.py to consolidate authentication into an auth.py and login.py file for installations in other applications. Last but not least is a simple template to present the user with a login page.

For our RootFactory, we’ve defined three groups, client, manager and admin. Within our __init__.py (or our views if we use add_handler), we can restrict permissions using Pyramid’s authentication.

In our __init__.py, we can use the permission= to specify membership in a group.

config.add_route('admin2', '/admin/', view='project.admin.index', permission='admin', view_renderer='admin_index.jinja2')

If we have used add_handler:

config.add_route('admin2', '/admin/', view='project.admin.index', view_renderer='admin_index.jinja2')

We can use the permissions ACL in the @action decorator:

@action(renderer='admin_index.jinja2', permission='admin')

To access the userid in your views:

from pyramid.security import authenticated_userid

userid = authenticated_userid(request)

At this point, we’ve migrated a Repoze.who/Repoze.what authentication scheme that only used Group membership as its criteria and we have an SQL backed authentication system under Pyramid.

Most of the guidance for this came from:

* http://docs.pylonshq.com/pyramid/dev/tutorials/wiki2/authorization.html

Modifications to __init__.py:

Added to the import section:

from pyramid.authentication import AuthTktAuthenticationPolicy
from pyramid.authorization import ACLAuthorizationPolicy

from project.login import forbidden_view, groupfinder
from pyramid.exceptions import Forbidden

Added to the Configurator:

    authn_policy = AuthTktAuthenticationPolicy(
        'sosecret', callback=groupfinder)
    authz_policy = ACLAuthorizationPolicy()
    config = Configurator(settings=settings,
                          root_factory='project.login.RootFactory',
                          authentication_policy=authn_policy,
                          authorization_policy=authz_policy)

Added within the begin()/end() block of the Configurator

    config.add_view(forbidden_view, context=Forbidden)
    config.add_route('login', '/login',
                     view='project.login.login',
                     view_renderer='project:templates/login.pt')
    config.add_route('logout', '/logout',
                     view='project.login.logout',)

auth.py:

import transaction

from sqlalchemy import create_engine
from sqlalchemy import Column
from sqlalchemy import Integer
from sqlalchemy import Unicode

from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.declarative import declarative_base

from sqlalchemy.orm import scoped_session
from sqlalchemy.orm import sessionmaker

from zope.sqlalchemy import ZopeTransactionExtension

DBSession = scoped_session(sessionmaker(extension=ZopeTransactionExtension()))
Base = declarative_base()

from sqlalchemy import *
from sqlalchemy.databases import mysql
from sqlalchemy.orm import relation, backref, synonym
from sqlalchemy.orm.exc import NoResultFound

import os
from hashlib import sha1
from datetime import datetime

user_group_table = Table('auth_user_groups', Base.metadata,
    Column('user_id', mysql.BIGINT(20, unsigned=True), ForeignKey('auth_users.id', onupdate='CASCADE', ondelete='CASCADE')),
    Column('group_id', mysql.BIGINT(20, unsigned=True), ForeignKey('auth_groups.id', onupdate='CASCADE', ondelete='CASCADE'))
)

class AuthGroup(Base):
    __tablename__ = 'auth_groups'

    id = Column(mysql.BIGINT(20, unsigned=True), primary_key=True, autoincrement=True)
    name = Column(Unicode(80), unique=True, nullable=False)
    created = Column(mysql.DATE())

    users = relation('AuthUser', secondary=user_group_table, backref='auth_groups')

    def __repr__(self):
        return '' % self.name

    def __unicode__(self):
        return self.name

class AuthUser(Base):
    __tablename__ = 'auth_users'

    id = Column(mysql.BIGINT(20, unsigned=True), primary_key=True, autoincrement=True)
    username = Column(Unicode(80), nullable=False)
    _password = Column('password', Unicode(80), nullable=False)
    email = Column(Unicode(80), nullable=False)
    contact = Column(Unicode(80), nullable=False)
    company = Column(Unicode(80), nullable=False)

    groups = relation('AuthGroup', secondary=user_group_table, backref='auth_users')

    @property
    def permissions(self):
        perms = set()
        for g in self.groups:
            perms = perms | set(g.permissions)
        return perms

    def _set_password(self, password):
        hashed_password = password

        if isinstance(password, unicode):
            password_8bit = password.encode('UTF-8')
        else:
            password_8bit = password

        salt = sha1()
        salt.update(os.urandom(60))
        hash = sha1()
        hash.update(password_8bit + salt.hexdigest())
        hashed_password = salt.hexdigest() + hash.hexdigest()

        if not isinstance(hashed_password, unicode):
            hashed_password = hashed_password.decode('UTF-8')
        self._password = hashed_password

    def _get_password(self):
        return self._password

    password = synonym('_password', descriptor=property(_get_password, _set_password))

    def validate_password(self, password):
        hashed_pass = sha1()
        hashed_pass.update(password + self.password[:40])
        return self.password[40:] == hashed_pass.hexdigest()

    def __repr__(self):
        return '' % (self.id, self.username, self.email)

    def __unicode__(self):
        return self.username

login.py:

from pyramid.httpexceptions import HTTPFound
from pyramid.security import remember
from pyramid.security import forget
from pyramid.security import Allow
from pyramid.security import Everyone
from pyramid.url import route_url
from pyramid.renderers import render_to_response

from project.auth import AuthUser
from project.models import DBSession

def login(request):
    dbsession = DBSession()
    login_url = route_url('login', request)
    referrer = request.url
    if referrer == login_url:
        referrer = '/' # never use the login form itself as came_from
    came_from = request.params.get('came_from', referrer)
    message = ''
    login = ''
    password = ''
    if 'form.submitted' in request.params:
        login = request.params['login']
        password = request.params['password']
        auth = dbsession.query(AuthUser).filter(AuthUser.username==login).first()
        if auth.validate_password(password):
            headers = remember(request, login)
            """
                  We use the Primary Key as our identifier once someone has authenticated rather than the
                  username.  You can change what is returned as the userid by altering what is passed to
                  remember.
            """
            #headers = remember(request, auth.id)
            return HTTPFound(location = came_from,
                             headers = headers)
        message = 'Failed login'

    return dict(
        message = message,
        url = request.application_url + '/login',
        came_from = came_from,
        login = login,
        password = password,
        )
    
def logout(request):
    headers = forget(request)
    return HTTPFound(location = route_url('root', request),
                     headers = headers)
    
def forbidden_view(request):
    login_url = route_url('login', request)
    referrer = request.url
    if referrer == login_url:
        referrer = '/' # never use the login form itself as came_from
    came_from = request.params.get('came_from', referrer)
    return render_to_response('templates/login.pt', dict(
               message = '',
               url = request.application_url + '/login',
               came_from = came_from,
               login = '',
               password = '',
           ), request=request)

def groupfinder(userid, request):
    dbsession = DBSession()
    auth = dbsession.query(AuthUser).filter(AuthUser.id==userid).first()
    if auth:
        return [('group:%s' % group.name) for group in auth.groups]

class RootFactory(object):
    __acl__ = [ (Allow, 'group:client', 'client'),
                (Allow, 'group:manager', 'manager'),
                (Allow, 'group:admin', 'admin') ]
    def __init__(self, request):
        self.__dict__.update(request.matchdict)

templates/login.pt:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html
    xmlns="http://www.w3.org/1999/xhtml"
    xmlns:tal="http://xml.zope.org/namespaces/tal">

<head>
  <meta content="text/html; charset=utf-8" http-equiv="Content-Type"/>
  <title>Authentication Test</title>
  <link rel="stylesheet" type="text/css"
        href="${request.application_url}/static/style.css" />
</head>

<body>

<h1>Log In</h1>

<div tal:replace="message"/>

<div class="main_content">
  <form action="${url}" method="post">
    <input type="hidden" name="came_from" value="${came_from}"/>
    <input type="text" name="login" value="${login}"/>
    <br/>
    <input type="password" name="password" value="${password}"/>
    <br/>
    <input type="submit" name="form.submitted" value="Log In"/>
  </form>
</div>  

</body>
</html>