Random Musings of an Insane Mind

It seems that everyone on the blogosphere, including Divmod, is talking about “scaling” these days.  I’d like to talk a bit about what we mean ­— and by “we” I mean both the Twisted community and Divmod, Inc., — when we talk about “scaling”.

[more]

One of the issues that I continually deal with is the scaling of an application and its platform.  I believe this author makes the point rather well.

156 hours.

That’s what it took to track down a solution to a problem with some Open Source software.  The software was written in the early 2000 time frame, the last documentation update was in 2006.  The scenario that we were designing was documented on a page written in 2004.

The issue we ran into must be something that someone else has stumbled into because it is a very basic piece of the operation of this piece of software, but, in perusing all of the available documentation, using google to find any possible references, looking through all FAQs, committed code, mailing lists, etc. the solution presented itself on a page last updated in January of 2000.

A three line mention.

That’s it.

The author of the FAQ written in 2004 that describes the process and documents every step save for one very important part.  The three line mention in another FAQ, coincidentally written by the FAQ author in an email that was sent to a mailing list and included in someone else’s FAQ.

This is the inherent cost in Open Source.

We’re not using this software in an odd manner — in fact, the feature we were trying to use is one of the three fundamental uses.  The software hasn’t changed much in the last 4 years, but, it just goes to show you that documentation is easily forgotton in the Open Source world.

Would I have it any other way?  No.  I prefer open source because we can develop solutions that give us a competitive edge, and, if we need to, we can change the code to fix problems that the developers won’t fix.

Often times our requirements are based on a business case which conflicts with some of the purist open source coders.

The technology is there to make it very easy for an open source project to document.  Wiki’s, blogs, web access to revision control software, but, documentation is usually done as an afterthought, or worse, left up to the people that may not completely understand the product.

I have written technical documentation many times in my life and I evaluate a lot of open source projects to see if they fit into our organization and I can tell you, a large percentage of the documentation for many open source projects is extremely bad.

A recent project that we started to evaluate had configs on their web site documentation, which was powered by a wiki, that directly contradicted their mailing list responses from the company.  Thirty seconds later and I was able to correct the documentation to reflect the right information, but, that post which was 3 months old and directly referenced the incorrect wiki page never elicited an update.

Open Source Project maintainers — if you want people to use your product, you MUST provide good documentation.  Samples of config files with quick comments about usability are a start if you’re not going to completely document the required config files, but, a project with little to no documentation will not get adopted by the masses.

While I appreciate the fact that the coders don’t like to write documentation, if you are going to publish a project and expect people to use it, take some time to write some documentation.  When someone suggests changes or makes modifications to the wiki, be receptive rather than adversarial.

Your project will succeed much more quickly.

Also, if you have a commercial support package, and while I’m beta testing some software, I am also testing your support team’s attitude.  I know the same guys hanging out on irc, monitoring the mailing list and responding to bug and feature inquiries are the same people I’m going to be contacting for support.  Treat me wrong and I’ll find another solution.

Monetizing GPLed software isn’t easy — I know that.  But make it easy for those of us that will end up relying on your solution and are willing to pay for a support contract to make sure we get the support we need.

Years ago, I was taught that to avoid downtime, you bought the best router, the best switches, the best computers, multithousand dollar machines, the best storage servers and put everything together…. and then hoped.

Somewhere along the line, during a phone call at 3am talking with a tech from NetApp regarding a shelf failure, it seemed that the strategy had failed.  NetApp did dispatch before we even knew the shelf had failed — their system sent an email out with the status, NetApp had a shelf couriered to the facility and the phone call I received was from the facility asking if they should let the NetApp technician in.

NetApp handled the problem exceedingly well, but, we didn’t have 100% availability because the shelf failed and knocked out multiple disks in the array.  While NetApp did suggest we would have 100% uptime with the solution, based on the facts, I don’t believe there is any way we could have had 100%.  As it stood, 99.95% was what we ended up with.

The data center we were located in at the time had two 30+ minute unplanned power outages.  Multiple power feeds, single vault construction.  Another hosting company recently had a multi-day outage because they had a single power vault for their data center.  Even more ironic, a transformer failure 3 years earlier should have taught them that lesson.

So, what does one do?

HP used to want to sell really expensive hardware as did Sun when chasing five nines, 99.999%.  Google took a different approach using COTS (Commercial Off The Shelf) hardware.

Google planned to fail.  In fact, their entire distributed computer design handles failures very gracefully.  When a node crashes, the disk storage from that machine is replicated elsewhere on the network, its tasks are handed to other nodes.  This gives Google a distinct advantage.  Because their network is designed with cheaper hardware, they can put more of it online.  With more equipment comes more CPU capacity and this allows Google to do expensive calculations that other companies can only dream of.

You can buy one dual Xeon quadcore machine that is expensive and reliable for $4500, or, you can buy 3 dual Xeon quadcore machines for $4500.  Three machines on Google’s distributed computer will provide a primary and two backups for the same price, however, all three machines are available to provide Google with CPU and Disk space for the same price as the more expensive machine.

Because Google engineered their network expecting failure, their solution is obviously very robust and approaches the holy grail of five nines.  Geographically dispersed, fault tolerant network, fault tolerant distributed computer ensures you get the results you’re looking for quickly… every time.

Did Google start their mission thinking this way, or did they hire the right thinkers?  Its certainly helped our thought processes as we engineer solutions for our clients.

Greenness aside, its difficult to develop truly dense computing when you’re limited to x amount of power in a cabinet or cage.  Many data centers talk about engineering x watts per square foot.  With air cooling, 450-550 watts per square foot is supposedly the max.  One data center claims 1500 watts per square foot using air cooling.  Power converts to heat, more power = more heat.  While a facility may have the electrical capacity, most are engineering for the cooling capacity.

4800 watts is the limit at a particular data center for 1 cabinet.  Remember watts and amps and volts?

Watts = Volts * Amps

120Volt 20Amp service = 2400 Watts.

208Volt 20Amp service = 4160 Watts.

So, if you have a cabinet with 2 120V@20A circuits, and consider that dual quadcore Xeon servers come with 500 Watt power supplies, one cabinet with ten 500 watt machines exceeds the power available in a cabinet.  Exceeding 80% of the circuit is considered bad form, some companies specify a 75% maximum.

Supermicro Blade servers are slightly more efficient with 10 dual quadcore Xeon servers using 2000 watts, leaving room for support equipment.  The same blade chassis uses 1500 watts if you use 208V.

Yes, I know a server doesn’t use the max wattage all the time, but, the higher the CPU utilization, the closer to that max you get.  Startup power drain also approaches that max.  There are power strips that do staggered power-on, set max amp draw with shutoff, etc.  These aren’t problems you run into with the breadrack and minitower data centers because you just plug machines in until the circuit blows, then, remove the last one.

How much power does a device use:

• 10/100/1000 48 port switch with 4 fiber uplinks, 45 watts (peak .4 amps)
• P4/3.0ghz, 2gb RAM, 320gb SATA Hard Drive, 73.2 watts (peak 1.2 amps)
• Core2Duo E6600, 2gb RAM, 320gb SATA Hard drive, 114.8 watts (peak 1.0 amps)
• Core2Quad Q6600 64bit mode, 2gb RAM, 320gb SATA Hard drive x 2, 103.8 watts (peak 1.2 amps)
• Core2Quad Q6600 32bit mode, 2gb RAM, 320gb SATA, 110.9 watts (peak 1.0 amps)

So, even though most of these machines have 260-320 watt power supplies, their in-use wattage is less than half the stated max which allows a little more density.  So, without that metering, if you were to spec everything based on the books, you’d be seriously underutilizing your racks.

Raritan and APC both make Power Strips that allow you to monitor power with the Raritan being a little more pricey, but, allowing metering down to the individual outlet.

Granted, you still need to plan ahead, without knowing the true power use and efficiency of your power supplies, you could severely overbudget or underbudget your power needs.

What does my day have in store?  Doing a ton more power math for our next expansion.