Random Musings of an Insane Mind

A few years ago, a very unique phone service called Grand Central was purchased by Google. As with most acquisitions that Google has made, the service was closed, existing clients maintained their current service level, but, new clients weren’t added. Grand Central had a very unique service offering and much like Picassa or Postini, you knew Google was going to take the service, twist it around and make it better and change the price model. With most of the other services that Google purchased, they were quickly revamped, branded and released. This wasn’t the case with Grand Central. Google announced Google Voice, and allowed you to submit your email address to get put on the waiting list. After what has seemed like many years, and after people on the Internet had started getting invites on June 26, 2009, I was pleasantly surprised when I opened up my email to see a notice from Google. Since I was somewhat familiar with Grand Central’s offering, I was excited to see what Google had done.

Voicemail almost becomes as easy to use as email. You can listen to voicemails, read them (if you have transcription turned on) and forward the messages to other email addresses. Once you have signed into Google Voice, you are presented with the Inbox

On the left menu, we are presented with special inboxes for voicemail:

and a number of other inboxes including SMS, Recorded, Placed, Received and Missed Calls. If you send an SMS message to your Google Voice Number, it is recorded in the Inbox and the SMS inbox, and forwarded to any phone you have tagged as able to accept SMS. You can also send SMS messages from within Google Voice by clicking the SMS button.

If a number is marked as spam, future calls from that number will be sent to voicemail immediately without ringing your numbers. You can unblock a number marked as spam later.

In the settings, you can set up how voicemail notifications should occur. You can select to have the voicemail notification emailed to you and optionally page your mobile phone through SMS.

Your email message will include a transcribed copy of the message. In several test calls, their transcription was fairly accurate. During playback, a green underline is displayed under each word as you listen to the message.

The Phones menu allows you to set up multiple phone numbers. When someone calls your Google Voice number, all of the phone numbers listed here ring at the same time. You can answer any of the phones and the first one answered receives the call.

By default, when answering an incoming call, you receive a notification that Google Voice is calling along with the name of the caller. You can enter a 1 to accept the call, 2 to send it to voicemail, 3 to send it to voicemail and listen, or 4 to accept the call and record it. There is a brief notification at the beginning of the call on both sides that the call is being recorded. The recorded call is able to be accessed in the Recorded Inbox. When someone calls your Google Voice number, they are told that Google Voice has answered the call and it asks for their name which is presented to you in presentation mode.

When you add a phone, Google places a call to the number you’ve added and asks for a two digit code to be keyed in.

There are also advanced settings:

You can set up Call Groups and have different behaviors depending on what group the caller is in. In this case, Friends are put through immediately when the phone is answered without me having the option to screen the call. A caller receives a ringing phone as you are being located or listening to the menu options during the incoming call.

Once in groups, you can set which phones will be rung, define a special greeting and whether you want to use call presentation:

Of course, you can edit your contact lists and change what group each contact is in. By default, Google Voice has already imported your Gmail contact list. There are several other import methods supported, so, importing your contact list should be easy.

Another nice feature is the Call Widget. This is a method for placing an icon on your website where a potential caller can click the graphic, enter their phone number and hit connect. Google then calls that number, establishes the connection, then proceeds to call your number. Your number is hidden within an encoded string making this a somewhat effective method for accepting callers without giving out your number.

The above 3 screens show the widget on a page, entering a name and phone number and connecting the call. When the name is entered, Google does do text to speech and announces the call. If you put a two word name, i.e. Bob Smith, the nature of the URL encoding shows through and the caller is announced as Bob plus Smith.

The last screen in the settings is for Billing. The prices for International calls are relatively aggressive compared to Vonage.

My initial impression is quite positive. Phone calls connected through the service are extremely quick and sound great. When you want to change a message prompt, Google Voice calls your phone so that you don’t need to depend on your microphone on your computer resulting in a relatively good quality recording.

Irony? The Google Voice widget is a flash widget and I haven’t been able to get Flash to install in Chrome. I haven’t been able to install Delicious for Chrome either, and of course, the Google Toolbar doesn’t work. The web interface for Google Voice is very ajax intensive and it loads very quickly and is very responsive. Since I prefer using my keyboard shortcuts over the trackpad on my laptop, Google does capture some of the shortcuts I would normally use to switch tabs.

If you don’t have Google Voice and are looking for a good way to have a single phone number that rings your house, mobile and work numbers and allows some handy features, you might want to try applying for an Invite at Google Voice.

While Google’s App Engine is a well planned service and it does work incredibly well for what it does, sometimes things break due to resource limits, etc.

While the app engine platform is still running, it appears to be an issue with this particular application’s committed resources. The App Gallery has exceeded it’s memory quota.

Many Hosting companies operate on razor thin margins trying to capture as much market share as possible. Over the long haul, many $99/month dedicated servers can be absorbed into your existing bandwidth commitments without any incremental cost. Early on, one dedicated hosting provider dumped servers on the market for $99 with 700gb/transfer per month. At the time, they were undercutting hosting providers and it was deemed impossible that they could be able to fulfill the hosting world’s needs. In reality, they knew that their average client used 2.5gb of transfer per month, so, what difference did it make if they handed their average client 700gb. By having an ‘enormous’ cap, the average consumer wouldn’t be scared about overage charges, but, there were companies that knew they would exceed that cap and the penalty rate structure forced them to go elsewhere. That hosting provider cherrypicked the clients that would make the most money, even though they were a budget provider.

Later, they offered upgrades to the hardware and bandwidth commitments leaving many of those initial customers stuck on older hardware. There was no upgrade path to get from one machine to another except for the client moving the data themselves. The hosting company was only responsible for making sure the machine had power and network. However, there needs to be an upgrade path and there needs to be enough margin in the equation to facilitate hardware and network upgrades over time.

At some point the useful life of a machine is exceeded and one is faced with upgrading the machine, or, replacing components if the machine fails. Typically, CPU fans and hard drives will fail since they are moving parts. Other times, the client installs applications that require more CPU horsepower or runs into a situation where a machine needs more RAM. Depending on the age of the machine, those upgrade costs might exceed installing a new chassis.

With today’s hardware replacing yesterday’s hardware, often times there is quite a disparity between the computing power of the existing machine and the replacement. Virtualization can allow you to put in a powerful machine and replace multiple older machines, sometimes at a much lower TCO than maintaining the older machines.

That conversion isn’t without its issues though. If you are measuring bandwidth, you can no longer use the SNMP statistics from your switch, you must use something to count the flows. Device naming becomes an issue because you need to identify the virtual machine and the physical chassis that the machine is on in case there is a hardware issue. Clients don’t always understand virtualization and want a ‘dedicated’ server, even though their CPU core can be pinned to their exclusive use. If they need extra capacity, and it is available on the chassis, they can utilize it. As a result, Virtualization of a data center can significantly decrease power consumption. An older Pentium 4/3.0ghz CPU can easily reside on a single core of a 2.4ghz Xeon with room to spare. Considering the older infrastructure, you could easily fit 8 Pentium 4/3.0ghz machines with 2GB ram on a single dual CPU Quadcore Xeon with 16gb RAM. An 8:1 consolidation considering the lower utilization machines can result in considerable density increases. Replacing those eight machines might result in using roughly one sixth the power of the previous eight, so, you can still increase the cores per rack which can increase profitabilty. Provided with a mixed infrastructure where you might be replacing single and dual core machines, again, you might lose some of the economies of scale, but, the consolidation will still ultimately increase core density.

Virtualization techniques include using Xen, Citrix, KVM, Virtuozzo and VMWare.

Intel has an interesting blog post about Optimizing Costs within the Data Center that talks about a 10:1 reduction in hardware replacing singlecore machines with virtualized instances.

In addition to the cost and power savings, they saw a processor savings as well. If you’re selling dedicated servers, it might be difficult to give someone less than a whole processor if they had been sold a single processor, but, in a corporate environment, as long as the machine has enough CPU horsepower to do its job, more than one virtual machine can be assigned per core. For example, you can install ten Virtual Machines on an eight core machine and probably still have excess CPU.

However, applications are taking more CPU time than they used to, so, even if you are able to keep a 4:1 ratio, you’re still ahead of the game.

Recently we had a client that had some performance issues with a Joomla installation. The site wasn’t getting an incredible amount of traffic, but, the traffic it was getting was just absolutely overloading the server.

Since the machine hadn’t been having issues before, the first thing we did was contact the client and ask what had changed. We already knew the site and database that was using most of the CPU time, but, the bandwidth graph didn’t suggest that it was traffic overrunning the server. Our client rescued this client from another hosting company because the site was unusable in during prime time. So, we’ve inherited a problem. During the move, the site was upgraded from 1.0 to 1.5, so, we didn’t even have a decent baseline to revert to.

The stopgap solution was to move the .htaccess mod_rewrite rules into the apache configuration which helped somewhat. We identified a few sections of the code that were getting hit really hard and wrote a mod_rewrite rule to serve those images direct from disk — bypassing Joomla serving those images through itself. This made a large impact and at least got the site responsive enough that we could leave it online and work through the admin to figure out what had gone wrong.

Some of the modules that had been enabled contributed to quite a bit of the performance headache. One chat module generated 404s every second for each person logged in to see if there were any pending messages. Since Joomla is loaded for each 404 file, this added quite a bit of extra processing. Another quick modification to the configuration eliminated dozens of bad requests. At this point, the server is responsive, the client is happy and we make notes in the trouble ticket system and our internal documentation for reference.

Three days later the machine alerts and our load problem is back. After all of the changes, something is still having problems. Upon deeper inspection, we find that portions of the system dealing with the menus are being recreated each time. There’s no built in caching, so, the decision is to try Varnish. Varnish has worked in the past for WordPress sites that have gotten hit hard, so, we figured if we could cache the images, css and some of the static pages that don’t require authentication, we can get the server to be responsive again.

Apart from the basic configuration, our varnish.vcl file looked like this:

sub vcl_recv {
  if (req.http.host ~ "^(www.)?domain.com$") {
     set req.http.host = "domain.com";
  }

 if (req.url ~ "\.(png|gif|jpg|ico|jpeg|swf|css|js)$") {
    unset req.http.cookie;
  }
}

sub vcl_fetch {
 set obj.ttl = 60s;
 if (req.url ~ "\.(png|gif|jpg|ico|jpeg|swf|css|js)$") {
      set obj.ttl = 3600s;
 }
}

To get the apache logs to report the IP, you need to modify the VirtualHost config to log the forwarded IP.

The performance of the site after running Varnish in front of Apache was quite good. Apache was left with handling only .php and the server is again responsive. It runs like this for a week or more without any issues and only a slight load spike here or there.

However, Joomla doesn’t like the fact that every request’s REMOTE_ADDR is 127.0.0.1 and some addons stop working. In particular an application that allows the client to upload .pdf files into a library requires a valid IP address for some reason. Another module to add a sub-administration panel for a manager/editor also requires an IP address other than 127.0.0.1.

With some reservation, we decide to switch to Nginx + FastCGI which removes the reverse proxy and should fix the IP address problems.

Our configuration for Nginx with Joomla:

server {
        listen 66.55.44.33:80;
	server_name  www.domain.com;
 	rewrite ^(.*) http://domain.com$1 permanent;
}
server {
        listen 66.55.44.33:80;
	server_name  domain.com;

	access_log  /var/log/nginx/domain.com-access.log;

	location / {
		root   /var/www/domain.com;
		index  index.html index.htm index.php;

           if ( !-e $request_filename ) {
             rewrite (/|\.php|\.html|\.htm|\.feed|\.pdf|\.raw|/[^.]*)$ /index.php last;
             break;
           }

	}

	error_page   500 502 503 504  /50x.html;
	location = /50x.html {
		root   /var/www/nginx-default;
	}

	location ~ \.php$ {
		fastcgi_pass   unix:/tmp/php-fastcgi.socket;
		fastcgi_index  index.php;
		fastcgi_param  SCRIPT_FILENAME  /var/www/domain.com/$fastcgi_script_name;
		include	fastcgi_params;
	}

        location = /modules/mod_oneononechat/chatfiles/ {
           if ( !-e $request_filename ) {
             return 404;
           }
        }
}

With this configuration, Joomla was handed any URL for a file that didn’t exist. This was to allow the Search Engine Friendly (SEF) links. The second 404 handler was to handle the oneononechat module which looks for messages destined for the logged in user.

With Nginx, the site is again responsive. Load spikes occur from time to time, but, the site is stable and has a lot less trouble dealing with the load. However, once in a while the load spikes, but, the server seems to recover pretty well.

However, a module called Rokmenu which was included with the template design appears to have issues. Running php behind FastCGI sometimes gives different results than running as mod_php and it appears that Rokmenu is relying on the path being passed and doesn’t normalize it properly. So, when the menu is generated, with SEF on or off, urls look like /index.php/index.php/index.php/components/com_docman/themes/default/images/icons/16×16/pdf.png.

Obviously this creates a broken link and causes more 404s. We installed a fresh Joomla on Apache, imported the data from the copy running on Nginx, and Apache with mod_php appears to work properly. However, the performance is quite poor.

In order to troubleshoot, we made a list of every addon and ran through some debugging. With apachebench, we wrote up a quick command line that could be pasted in at the ssh prompt and decided upon some metrics. Within minutes, our first test revealed 90% of our performance issue. Two of the addons required compatibility mode because they were written for 1.0 and hadn’t been updated. Turning on compatibility mode on our freshly installed site resulted in 10x worse performance. As a test, we disabled the two modules that relied on compatibility mode and turned off compatibility mode and the load dropped immensely. We had disabled SEF early on thinking it might be the issue, but, we found the performance problem almost immediately. Enabling other modules and subsequent tests showed marginal performance changes. Compatibility mode was our culprit the entire time.

The client started a search for two modules to replace the two that required compatibility mode and disabled them temporarily while we moved the site back to Apache to fix the url issue in Rokmenu. At this point, the site was responsive, though, pageloads with lots of images were not as quick as they had been with Nginx or Varnish. At a later point, images and static files will be served from Nginx or Varnish, but, the site is fairly responsive and handles the load spikes reasonably well when Googlebot or another spider hits.

In the end the site ended up running on Apache because Varnish and Nginx had minor issues with the deployment. Moving to Apache alternatives doesn’t always fix everything and may introduce side-effects that you cannot work around.

User Interface Design is about usability and presentation. When you have data, presenting the data in a usable manner that is easy to understand is much more valuable than putting the data onscreen and letting the user try to decipher it. In my previous rant about User Interface Design, I have been faced with reworking dozens of pages to fix typos, spelling errors, bad grammar and poor interface design.

In the two pictures that follow, you’ll see the original report and the modified report. Very little HTML markup has been used in either case as the site needed to be functional and was slated to have a web designer make it look nice. Seven years later and the site still didn’t get the facelift it needed.

In the first graphic, you can see a legend that details what each of the status updates for each step were. We have information repeated that seems somewhat unnecessary. There isn’t any real reason to repeat the Transaction ID, Device Name, or Job Type. It is questionable whether Job Type is a very descriptive title.

In the second graphic, we’ve consolidated the data that was duplicated in the prior graphic. The Task ID, Final Status, Task and Device are given once, and the individual status checks are given below with the status of each step enumerated. The task completion color is given at the top of the task rather than the bottom which I believe is more logical. The mind doesn’t want to look at the data that led to a status. You want to quickly look for tasks that failed and then look for the detail. Since you read top to bottom, searching for a red status would allow you to look at the results below.

In both cases, the important data is bold. I believe that is important as scanning the page draws the eye to the important data and the less important data can be analyzed if there is a need.

The two pictures that follow show how feng-gui.com believes the eye will react when looking at that page. Based on the 2nd screenshot, I believe we’ve reached the goal of making the page present the data clearly and concisely.

One page down, two hundred to go. :)