Regenerative Braking for Services

George Reese has a missive over on O’Reilly’s site about why auto-scaling your “cloud” application is a bad idea. He starts from the naïve case where scaling your computing without bounds leads to your expenses scaling without bounds as well. Okay, that makes sense. Then he goes on to explain that setting those bounds to do the right thing is too hard, and involves humans doing capacity planning, so you should just do better capacity planning with humans and leave the automation out.

Now I’m a big fan of robots and having machines do tedious work for me, so this claim holds little truck with me. Frankly, the words “too hard” translated as I read them to “you could have a strategic advantage over competitors if you do this well.” Unsurprisingly, I’m not the only one that feels this way, and, in fact, several people chimed in with refutations and examples of how they're already doing this today to great advantage.

A response by Sam Curren, Really Bad Reasons Not To Auto-scale, refuted most of the “it’s too hard to get it right” arguments. Adam Jacob had a good comment as well, if you’re monitoring the wrong things, it is in fact easy to get wrong. In fact, one can look to Don MacAskill’s post about smugmug on EC2 to see some examples of what measuring the right things can look like. Breaking things apart into pieces that are easier to measure is an implicit piece of Don’s discussion that probably warrants more discussion another time.

One thing that hasn’t been mentioned yet in this conversation is that if you don't degrade gracefully under pressure in any of these models you've already lost. If your service is starting to degrade (or know it’s about to) the only hard part is knowing whether to grin and gracefully degrade under the temporary pressure, or bring in more capacity. Thing is, humans are quite capable of making the wrong call here, and even if they make the right call, they’ll do it much slower and they won’t do it in the middle of the night when your service suddenly gets an unanticipated spike in popularity in Japan.

Back to the mental translation, if you can develop good algorithms (or even very simple ones) to better predict when to scale up and down, you save a lot of money that is traditionally blown on idle resources in slack times. Those idle resources can be turned off, or pressed into use for non-time-critical batch work, or even sublet them to someone else to do processing with. And in fact this last one is quite probably the business that EC2 and App Engine probably represent. “Here’s some spare resources let’s sell some usage on them rather than making $0 on resources that are continuing to costing money to run.” (That other large cluster players aren’t involved this market yet indicates they either don’t have enough capacity as it is, or they aren’t in a position where they care about that idle cost yet, or they just don’t get it. It’s another interesting conversation in and of itself.)

In any case, being more efficient about resource usage represents a competitive advantage that can make a big difference. It’s like the regenerative braking on hybrid cars. Many people just afford the cost of wasting that energy as heat, perhaps not even knowing that there is a better way. However, with some initial investment and knowhow you can capture some of it and realize greater efficiency and a cost savings to boot.

Review: Issaquah Brew House

After a long day of Eileen and Jeanie studying constitutional law and me painting an interior wall, Tom stopped by and we decided to head out to the Issaquah Brew House for dinner. We were met with what I can only describe as the most untimely service I've ever received at a restaurant.

It made a good excuse to give the "Submit a review" feature of Google Maps a whirl. The review is online.

The experiences have led me to wonder what the optimum server to table ratio is. I know there's a lot of clamor on the internet that claims that the gruff service in Europe is because servers handle more tables and (as a result) have to be somewhat more efficient, and have less time to be personable. Presumably someone with experience in restaurant management would have a good idea, but 6 tables seems like a reasonable maximum from what I'm reading online.

...but how many Big Macs will it buy?

There was a conversation about investment at work that mentioned stocks, gold and the "Big Mac Index" so I decided to do a quick analysis.

Year	Big Mac	DJIA	Big Macs per DJIA	$/oz Gold	Big Macs per oz Gold
2001	$2.54	10790	4248	271	106
2002	$2.49	10259	4120	278	111
2003	$2.65	8601	3245	343	129
2004	$2.90	10409	3589	417	143
2005	$3.15	10827	3437	428	135
2006	$3.15	10883	3454	530	168
2007	$3.22	12474	3873	640	198

It didn't quite work out the way I had expected it to. Plotting the CPI (or the Euro) against these numbers might also be interesting.

Long-term or Short-term?

This post is a quick stroll down algebra lane to figure out whether holding on to something for the (more favorable) long-term capital gains rate really makes sense or not. I’ve discussed this sort of problem with a couple people and I’m recording my ideas for reference, posterity and criticism…

This evening I was thinking about the cliff in taxation between short-term capital gains and long-term capital gains. Something is “long-term” if held for a year or more, or if you’re forced to sell it because you’re hired into a position in the Bush administration. (No, I’m not making that part up.)

Short-term gains are taxed at the normal income rate which is 25 or 28% for most software developer/system administrator folks I know, and long-term is taxed at 15% in most cases. (These numbers may change at Congressional whim. And while I’m disclaiming things here: I am not your tax or investment consultant, you should consult one if you want advice you can hold someone liable for. Beware of dog. Slippery when wet.) I’m going to pretend the AMT doesn’t exist for purposes of this exercise. (If your capital came from stock options, the IRS sadly will not let you pretend the AMT away.)

This leads to a problem of deciding if the risk of waiting longer is worth the 10-13% difference in tax rate. The rate only applies to the gain, not the full amount. The amount the captial is worth in-pocket is the price minus whatever capital gains tax applies.
So a useful application of algebra:

Net = Price – (Price – Paid) × Taxrate

The net right now is probably going to be at the higher short-term rate. The market goes up and down a lot, and it’s no good if by the time the long-term rate applies the net is going to be less than it is now. You can’t necessarily time the market and know where it’s going, but some industries have pretty obvious seasonal trends. What price can the capital drop to before it’s a better deal to sell now? We know the net we can get right now because of the equation above. So now we need to find the price for another tax rate, so solve for Price:

Net = Price – (Price – Paid) × 15%

Net = Price – (15% × Price – 15% × Paid)

Net = Price – 15% × Price + 15% × Paid

Net – 15% × Paid = Price – 15% × Price

Net – 15% × Paid = (1 – 15%) × Price

(Net – 15% × Paid) ÷ (1 – 15%) = Price

So if I paid 6.50 for something that’s now worth 20.00 short-term, that’s equivalent to 18.41 long-term. That is if it is likely to drop in value more than about 2.50 by the time the one year “long-term” timer runs out,

In just 6 months time, you’ll need to net 2% more just to keep up with the current rate of inflation.

((102% × Net) – 15% × Paid) ÷ 85% = Price

So now I’m at 18.80 long-term vs. 20 now. If I think there’s an even money chance that it’ll lose 1.20 in 6 months, I’d be better off to sell now and invest in something that has a more favorable trajectory.

“Timing the market” is a foolish adventure, but blindly waiting for the long-term is an unnecessary risk if the value moves in a seasonally predictable way that’s not in your favor.

I’ve setup a Google Spreadsheet with this math in it for the curious.

Coast Starlate

NPR had a story this morning about the Coast Starlate train that runs between Seattle and Los Angeles. It’s good to see press coverage of the problems this line is having. (e.g. From October 2005 through August 2006 the train delivered its passengers on-time only 2% of the time.) Maybe with enough coverage Union Pacific can be shamed into helping the trains run on time.

Not My Dream App

There’s a new contest at MyDreamApp.com for new Mac OS X software. The format for the contest with celebrity judges and open submissions from the user community and prizes for the top 3 entrants sounds like a lot of fun. A lot of fun, that is until you read the fine print.

Participants are forbidden from making their dream app open source if it’s selected as a winner. That’s right, by participating in the contest you agree to give up all interest in your idea and any code you put behind it to the group organizing the contest. If you wanted to give back to the community by offering your awesome idea for free or, better yet, offering it as open source software, you’re out of luck.

But wait, there’s more; the deal gets even better. In return for giving away your idea they’ll offer you a meager 15% royalty from ”net income” from the Mac version of the idea. “Net” in this context means if they spend money advertising, promoting or paying developers/QA/writers to work on the Mac version of the product that gets subtracted from the amount you get 15% of. This is exactly how many musicians get screwed out of the proceeds from their music by record companies. “You know it cost a lot to promote your album, so here’s a free cup of coffee. We look forward to your next effort.”

Just to top it all off, some top finishers get free equipment. The catch? The taxes on the giveaway are your responsibility. So you have to pay out of pocket federal, state and local taxes (let’s say 28%) of the value of the “free” item. I’m going to guess that the 15% of “net income” may not be enough to cover the taxes on the free item.

Did I mention the liability release provision potentially allows the organizers to skip out and not pay you anything? And if they do pay you it will be through PayPal. And if your submission is accepted you may not even be able to claim it was your idea on your resume without their permission. And, worst of all, if the effort required for them to follow through on making your idea a shiny product is more than “reasonable” they can just walk away and develop it for Windows instead. (And pay you nothing.)

It’s an interesting way for some people to feed on the creativity of others, and I guess if people understand all this and still participate, that’s their business. I’m a little dismayed that Woz and other luminaries are participating in something that seems so potentially abusive of it’s “winners”.

This isn’t to suggest that the people organizing the contest will do any of these nasty unkind things, only that they could if they wanted to. I don’t know any of them, and I hope, for the sake of those who participate, they stay honorable.

It would be nice if someone had an open source-friendly contest (with nicer rules) along these lines for great new open source Cocoa titles in the tradition of Adium and Colloquy.

Unethical Funding at WordPress

Apparently the people behind the WordPress project’s website, which includes the project’s founder, have decided that a good way to fund the project would be to use their Google PageRank to link to a lot of canned articles on topics that generate more advertising revenue than others. Some more details on this matter are available from Andy Baio’s blog and Ars Technica

It is deeply troubling that people associated with open source projects use resources that only have value because of unpaid community contribution and cash in on them. Unless that conversion is of impeccable ethics and completely transparent you’re bound to have a problem on your hands. In the case of the “Open Projects” (now Freenode) IRC network, there was a lack of transparency that led to the conclusion that Rob Levin was in fact using the donations that were advertised to “help the network” to pay his rent and feed his family. Certainly noble purposes, except that every other IRC network is run by volunteers who do the work mainly for free and when money is going to them to pay their living expenses that much is made more clear. This made some members of the user community upset.

In the case of WordPress, there is both the lack of transparency as to where the funds were used and the fact that the funds are being raised in a way that is certainly unethical. The link and page hiding method used is a violation of the contract between WordPress.org and Google for the AdWords program which opens the site maintainer and the project itself to lawsuits from Google, should they be feeling particularly nasty. As it is, Google has reset the pagerank for WordPress.org to zero and WordPress.org links are slowly disappearing from its search results. It doesn’t help matters that the project founder who is ostensibly responsible for this decision was on an extended vacation to various countries in Europe when this story broke. The particularly cynical might wonder how the vacation was funded.

At best, this was a tragic mistake on the part of the people responsible. The offending links and articles have now been removed, but it looks as if the damage is done and it may be too late for forgiveness from Google and many users.

I’m going to have to think long and hard about how much I can contribute to the WordPress parent project without getting myself tangled up in their apparently shaky ethics. I already had misgivings about the way they used the default links in the application to promote their personal blogs rather than useful community resources. (The lead developer brags about being Google’s “#1 most important Matt in the world” on his personal page, a direct result of his link being in the default install.) I had been planning to embark on a project to make WordPress database agnostic and did not want to deal with having to be a fork from the parent project. Perhaps a fork from the parent project would be better for the community though. The coming days and how this is all handled will shine a better light on which way is best to go.

For what it’s worth, I should point out that at no time have I been paid for my work on WordPress-pg and if I do receive such payment in the future I’ll be transparent about where the money is going and what I’m doing to repay the community for any such kindness.

Commerce Clause vs. 21st Amendment

The “commerce clause” and the 21st amendment’s will face off in a case to be argued before the U.S. Supreme Court on December 7th. The commerce clause has traditionally been held to prohibit per-state restrictions on interstate commerce. The 21st amendment however contains the following language:

Section 2. The transportation or importation into any state, territory, or possession of the United States for delivery or use therein of intoxicating liquors, in violation of the laws thereof, is hereby prohibited.

So can states actually regulate interstate commerce by blocking importation of alcohol in ways that may inhibit interstate commerce? We’ll find out sooner or later…

I happened across this story in the WSJ. A few press outlets have talked about the story including The Detroit News. Search Google News for possibly relevant articles. Although there hasn’t been a lot of coverage of the ways that this sort of deregulation could influence other markets and deregulation pushes there.

Combined Federal Blacklist Campaign

Generally we have done our donations to charitable causes through the CFC. Federal employees in King County donated $2.8 million to the various charities in CFC last year. In the past, we were able to donate to groups including the Cascade Land Conservancy (a group that buys land or conservation easements on property in Snohomish, King and Pierce Counties), the ACLU (a group dedicated to preserving our ability to speak, move and act free of government interference), and the EFF (a group dedicated to preserving the Internet as free and open for sharing of all data).

Things are different this year. As we perused the CFC booklet for the folks we usually donate to, a number of organizations were missing. Try as we might, we couldn’t find the ACLU, EFF, the Sierra Club, Amnesty International or a variety of other groups. This struck us as quite odd. These are well respected non-profit groups, that is outside of claims by radical clerics that the ACLU caused God’s wrath on 9/11.

As it turns out, the CFC had added a new requirement that non-profits who received funds from federal employees’ donations into CFC had to vet their employment and volunteer rosters against a government supplied blacklist of “terrorists”. If they found any matches they were required to inform federal authorities and fire the employee until resolution. This is the same sort of blacklist that keeps Cat Stevens from visiting the United States. Both the ACLU and the EFF strongly believe that a person’s employment and volunteer efforts are a matter between that person and the organization they work for. That’s what freedom of association and presumption of innocence is all about. Being a known felon convicted of a crime in a trial is one thing, being added to a list with no opportunity to contest your inclusion is quite another.

Once on the “terrorist volunteer” list or the “no-fly” list or Sen. McCarthy’s “communist sympathizer” list, it is extraordinarily difficult to get removed. It took Sen. Ted Kennedy talking to Tom Ridge and several weeks to get the TSA to remove him from the “no-fly” list. I can’t presume that this list imposed on non-profits would be any easier to get removed from. Especially when the list is not publicly available and there is no contact for removal.

Things such as this are uniquely anti-American. They chisel away at the freedoms and values that once set us apart form the rest of the world. Federal employees and other citizens who would like to complain to the head of the CFC about this policy, can contact the Director of CFC at:
Mara Patermaster
Combined Federal Campaign
Office of Personnel Management
1900 E Street, NW
Washington, DC 20415-1000

In the meantime, you can still donate to the ACLU, the EFF, the Sierra Club, and any of the other groups that you might normally give to under the CFC.