Ireland 2009

[Sorry, but the pictures are not in chronological order]

Andy (after the tasting session) at the Jameson Factory in Midleton, Ireland.

The old Jameson Factory buildings. A new improved factory was built in the 70's next door.

A waterwheel which powered the plant and is still running today.

A small still in front of the plant.

The EU helped to fund the restoration of the old buildings.

We spent a night in Cork, Ireland where we saw this plaque in front of a local cathedral. I never knew there was an apostle of temperance before . . .

We also visited Newgrange, a massive structure built in 3200 BC. The entire structure is built of stone and there are approximately forty more mounds, of varying sizes, in the surrounding area.

These pictures are from the Southern Coast, specifically, the Ring of Dingle.

Ross Castle in the Killarney National Park.

The Killarney Riding Stables. The O'Sullivan family has owned and operated the stable since the 60's. They currently have one hundred and forty horses!

Their Percheron stallion was both friendly and beautiful. Although, he was in his own separate barn with an adjoining paddock built out of railroad ties.

Andy's mount, Ben, is a Shire Drought cross.

Andy and I on the beach in County Kerry. My horse, Aladdin, is an Irish Draught horse. We rode for two full days around County Kerry, also known as "The Ring of Kerry." We covered over twenty miles on horseback and had a fantastic time.

We have had Seven for a year!

Seven is a pure-bred female rottweiler adopted from King County Animal Care & Control on June 10, 2008.

Here are a few pictures from her training class:

Seven intently paying attention to something off in the distance while on a "down."

Seven waiting to be released from a "sit" while we walk around her.

Still Waiting . . .

Break time.

More work. Seven has to "walk" with her shoulder right next to my hip.

Still working on off-leash commands.

Almost done with the class.

Next step, AKC Obedience trials? We'll see. Good dog, Seven.

A Few Beach Pictures

Mussels.

Mussels on Rock.

Andy and Seven.

Starfish.

Book Review: Spook Country

After finishing another book by Gibson with a few of the same central characters, I decided to read Spook Country. This was a mistake.

The characters and plot are both confused, disjoint and incomplete. Short of taking notes there is slim hope of tracking what is going on with the characters, but I shouldn't have worried about it because most of them are irrelevant and the few that are relevant never gain any sense of closure. The interactions between characters are like a political-correctness-gone-crazy dystopia where giving a damn about anyone would result in a harassment lawsuit. There are some interesting settings described, however not so interesting that they compensate for the cardboard cutout characters, all of whom keep one another at more than an emotionally safe distance throughout the whole story.

The last 50 pages are very chaotic. It reads as if the author had hit his deadline for turning in the manuscript and scribbled together a plot outline that tied off a few of the threads of the story and then walked away. I found Pattern Recognition passable, but this book, set with some of the same characters, really seems like the effort was phoned in and was a disappointment.

Who knew there was a clear-cutting merit badge?

Kudos to the P-I's Investigative Reporters.  One last hurrah I suppose.  If only the P-I could have figured out how to change with the times and put out a great online paper without the costs of publishing on paper.  I would have paid for that.

Profit trumps preservation for Boy Scout councils nationwide

Reconnected

From Snoqualmie River Flooding 8-Jan-2009

As of Friday, the highway washout that had cut us off from Fall City has been fixed. Thanks to the WSDOT for their hard work in getting a big mess repaired as quickly as they did. I still need to hike down Fish Hatchery Road (pictured to the right) to see what it's looking like now.

This flooding is by far the highest I've seen the river go since we've been living here, and it was the second time this flood season that the houses down in the floodplain wound up underwater. Hopefully they have the option of moving to higher ground rather than rebuilding in a place that requires a helicopter rescue when they don't get out in time. The situation is somewhat less hopeful for the RV park that's across the river from the highway.

Snoqualmie Falls at Flood II

I made it out to take more pictures and video (see below) of the flooding along the Snoqualmie River near Spring Glen this morning.





Check out this video of Snoqualmie Falls at a normal summer time flow for comparison.

Snoqualmie Falls at Flood

After an extended period of snowfall, the freezing level has climbed drastically while we receive over half a foot of new rainfall. The result is extensive flooding.

We have a few videos from Snoqualmie Falls along with a gallery of other images from around Spring Glen.

Update: More photos and video from January 8th.

Regenerative Braking for Services

George Reese has a missive over on O’Reilly’s site about why auto-scaling your “cloud” application is a bad idea. He starts from the naïve case where scaling your computing without bounds leads to your expenses scaling without bounds as well. Okay, that makes sense. Then he goes on to explain that setting those bounds to do the right thing is too hard, and involves humans doing capacity planning, so you should just do better capacity planning with humans and leave the automation out.

Now I’m a big fan of robots and having machines do tedious work for me, so this claim holds little truck with me. Frankly, the words “too hard” translated as I read them to “you could have a strategic advantage over competitors if you do this well.” Unsurprisingly, I’m not the only one that feels this way, and, in fact, several people chimed in with refutations and examples of how they're already doing this today to great advantage.

A response by Sam Curren, Really Bad Reasons Not To Auto-scale, refuted most of the “it’s too hard to get it right” arguments. Adam Jacob had a good comment as well, if you’re monitoring the wrong things, it is in fact easy to get wrong. In fact, one can look to Don MacAskill’s post about smugmug on EC2 to see some examples of what measuring the right things can look like. Breaking things apart into pieces that are easier to measure is an implicit piece of Don’s discussion that probably warrants more discussion another time.

One thing that hasn’t been mentioned yet in this conversation is that if you don't degrade gracefully under pressure in any of these models you've already lost. If your service is starting to degrade (or know it’s about to) the only hard part is knowing whether to grin and gracefully degrade under the temporary pressure, or bring in more capacity. Thing is, humans are quite capable of making the wrong call here, and even if they make the right call, they’ll do it much slower and they won’t do it in the middle of the night when your service suddenly gets an unanticipated spike in popularity in Japan.

Back to the mental translation, if you can develop good algorithms (or even very simple ones) to better predict when to scale up and down, you save a lot of money that is traditionally blown on idle resources in slack times. Those idle resources can be turned off, or pressed into use for non-time-critical batch work, or even sublet them to someone else to do processing with. And in fact this last one is quite probably the business that EC2 and App Engine probably represent. “Here’s some spare resources let’s sell some usage on them rather than making $0 on resources that are continuing to costing money to run.” (That other large cluster players aren’t involved this market yet indicates they either don’t have enough capacity as it is, or they aren’t in a position where they care about that idle cost yet, or they just don’t get it. It’s another interesting conversation in and of itself.)

In any case, being more efficient about resource usage represents a competitive advantage that can make a big difference. It’s like the regenerative braking on hybrid cars. Many people just afford the cost of wasting that energy as heat, perhaps not even knowing that there is a better way. However, with some initial investment and knowhow you can capture some of it and realize greater efficiency and a cost savings to boot.

Book Review: Life of Pi

An interesting story about trial through adversity. The story was reasonably good, but didn't particularly grab me and make me want to keep reading. I suspect this would've been a much more compelling tale if it had been told from the point of view of the tiger, and would've allowed the commentary on religion and the human condition to be somewhat less forced.

The story had a sort of disjointed episodic feel to it which came on rather quickly after the initial bit of character introduction. My mind is not yet made up on whether this style of storytelling helped relay the descent into madness from being trapped at sea. It would have worked better for me as a literary device if it had been used a bit more subtly. As it is, it seemed unintentionally scatterbrained.

For all the review commentary I read ahead of time about the religious message in this novel, it seemed tacked on in a very forced sort of way. The protagonist was confused about what to believe, tried to believe in everything in the same time, but largely just stood in awe of nature before him and the lucky breaks he got every now and again. The awe and luck was attributed to any random belief system that seemed to best fit the moment. And when there wasn't anything interesting going on this aspect was completely forgotten.

The questions about predestiny and what kind of benevolent god(s) kill your parents, dozens of innocent people and animals and leaves you alone on a liferaft with a carnivore were left largely unaddressed. In the end, the book left me wanting with its undirected episodic nature and failure to ask hard questions that might scare off some readers.

Boggle Solver

I've been working on a pet project with App Engine to try and get a better feel for it. It’s a Boggle puzzle solver that does some AJAXy tricks to multithread the solving work. The source code is online as well for anyone that’s curious. (There is a popular knockoff called “Scramble” on Facebook that is either different enough to keep Hasbro from filing a lawsuit or Hasbro’s lawyers are waiting around for them to make some money before bothering.)

In the case of App Engine it's particularly sensitive to requests that take “too long” to process. This was a particular hassle when I was importing the dictionary that is used. In order to solve a Boggle puzzle fairly quickly you want to have all the possible words arranged in a trie. This way you can stop quickly if you’re following letters that will never spell anything as you traverse the puzzle board. It took splitting the dictionary into 5000 separate pieces to get it to load without pushing me over my quota for “long” requests. Luckily we only have to do that once.

Next came the challenge of the puzzle solving itself. Again, solving the whole board in one request takes a while to process, apparently more than is allowed without running into that “long” request quota. Even in an optimized form (see the links to Dan Vanderkam’s work at the end of this entry), the full dictionary trie is 3MB and that takes a non-trivial amount of time to load in when you’re trying to handle requests within a few hundred milliseconds.

The solution was to reload the dictionary again but this time to break it up by initial trigrams. For every initial three letters, I store the appropriate shard of the dictionary (all the words that start with that combination) as a trie. There is also a blacklist of trigrams that form no words (for instance “frw”).

The javascript calls in with a copy of the puzzle and a point on the board to solve from. The server code then finds all the trigrams starting from the specified point and loads the appropriate dictionary shards. Since we're only solving from one point on the board there won't be more than 72 shards to load for any javascript call. (9 directions from the point and then 8 directions from each of those points because we're not allowed to backtrack.) The server then traverses the board using the dictionary tries hunting for words. When it finds them it stores the word and the places on the board where the word was found.

This information is all reduced into JSON and returned back to the browser that made the javascript call. The javascript on the browser is then responsible for taking all the found words and locations and sorting them in a sane way and displaying them for the end user.

Dan Vanderkam has written some interesting code and blog posts about optimizations in solving Boggle puzzles.

Seven our new Rottweiler

Our newest addition is Seven, a 4 year old Rottweiler that we adopted from King County Animal Control in Bellevue. She’s not exactly a petite dog at 105 pounds and her snorts and grunt and growls belie her girlish charm and friendliness. Here she is with her backpack full of water bottles and dog supplies ready to go for a tromp.



Our understanding of her history is that she was bred in order to sell the puppies and one day the breeder asked the neighbors to watch the dog for a week, while they were going to be out of town and they never came back. Unfortunately the neighbors also couldn’t keep Seven because they were moving to a city with a breed ban against Rottweilers. As an aside, that's a ridiculous law since a recent study showed that the most aggressive breeds are actually Dachshunds and Chihuahuas, with Jack Russell Terriers rounding out the top three. Rottweilers and Pit Bulls were average or below average in aggression towards strangers amongst the breeds shown. Apparently some people in Seattle are clamoring for breed bans, but not on aggressive breeds like Chihuahuas, but instead on Pit Bulls. Luckily some sane owners have banded together and Pasado’s Safe Haven has also gotten involved.

After recovering from her spay surgery Seven is doing well and seems to be enjoying her new home. She trained incredibly quickly to the invisible fence and gets along with one of our cats, Jasmine, who isn’t being an idiot around her. Odie still needs to learn that waiting until the dog gets close and then running away like prey is a bad idea.

Just yesterday Seven had her first class at Cascade K9. The object lesson right now is getting her “walk” under better control so she doesn’t try to run out ahead. We also need to get a bench for her to hop up on to practice “climb” at home. As always, the most important element in dog training is the humans, so we’ll try hard to be up to the challenge and we’ll report the progress we make…

Fred is Gone

Our trusted companion of the last couple years is gone. Fred came to us from Pasado's Safe Haven, rescued from death row at a local shelter. After we heard from vets that he only had months left, he proceeded to live two years of enthusiastic tromps in the snow, trips to the beach and hikes in the woods or just around the neighborhood. Slowly though, the degenerative myelopathy which afflicted him claimed his ability to control the back half of his body. Despite this, he soldiered on determined to follow his pack leaders and have fun doing it, until recently when he turned melancholy because he simply couldn't manage any more.

In his last hours we went, sat and relaxed along a sandy rivershore, watched the waves and the trees move and felt the breeze. He was treated to a special meal out and then we let him go. It is sad to lose him, but hopefully the two years we were able to provide him were as enjoyable for him as they were for us.

Security Advisory: Norton AntiVirus for Macintosh

Synopsis

Symantec's Norton AntiVirus for Macintosh (NAV) contains a vulnerability that can lead to local privilege escalation from group admin to root (the super-user) without any of the usual password prompts Mac OS X presents for gaining super-user access. Group admin includes any users with the "Allow this user to administer this computer" box checked, this generally includes the first user created in an OS X install. This vulnerability is caused by a setuid-root binary in NAV which automatically runs another binary in a location where it can be replaced by users with group admin permissions. Since most Mac OS X users are in group admin on their computers, most NAV users will be vulnerable.

Mitigation

The easiest (and most foolproof) mitigation strategy is to uninstall NAV. (I sure don't feel very secure when a vendor allows a local privilege escalation vulnerability to fester in their security software for over 9 months. Your feelings may vary.)

Set the sticky bit on all directories between the vulnerable binary and the filesystem root that are writable by group admin. This can be done in Terminal.app with sudo chmod +t / /Library /Library/Application\ Support /Library/Application\ Support/Symantec /Library/Application\ Support/Symantec/SmallScanner.app etc. Keep in mind that running "Repair Permissions" on your disk will remove this change and leave your NAV install vulnerable once again. Apple has set the sticky bit on / and /Library by default on Mac OS X 10.5, but not /Library/Application Support and obviously not on the directories that NAV is installed into.

What Symantec Had to Say

The last time I heard from Symantec was August 29th:

As you know, Symantec developers reviewed the issue concerning NAV for the Mac that you sent to us, and agreed that there is cause for concern. However, they felt that the same problem could potentially affect other vendor’s[sic] software as well. We contacted Apple Product Security, and suggested some changes that could improve security for everyone.
...
Symantec does not currently plan to make changes to our existing products to directly address the problem you reported. The changes would require considerable architectural modifications for those products, and the changes could cause other problems for those products if Apple subsequently release an OS update to address the underlying problem.
...

Timeline

Just in case people think the vendor didn't have enough time to address this, here's the timeline:

• 2007-Jan-16 - Reported issue to Symantec


• 2007-Jan-17 - Symantec will "review as soon as possible"


• 2007-Jan-31 - Symantec "working on planning a fix", will coordinate release when product update has ETA


• 2007-Apr-20 - Symantec thinks it would be better if Apple made changes to Mac OS X to fix the problem in NAV


• 2007-Jun-21 - Contacted Apple for status


• 2007-Jun-22 - Apple communicates the changes they're going to make for Leopard. These changes are not enough to workaround Symantec's vulnerable software.


• 2007-Aug-29 - Symantec says they've made suggestions to Apple, but otherwise aren't going to do anything to fix this issue (see the quote in the previous section)


• 2007-Oct-01 - It turns out that Apple's Leopard builds at the time didn't do enough to cover up the vulnerability in NAV


• 2007-Oct-11 - Contacted Symantec and Apple to let them know I was going to post this Nov-1


• 2007-Nov-1 - Today.

Details

SmallScanner is run as root by the setuid-root binary
/Library/Application Support/Symantec/AntiVirus/DiskMountNotify.app/Contents/MacOS/DiskMountN
otify. E.g. after inserting a data cdrom:
root 8734 28.9 -2.1 616152 43596 ?? U 7:51PM 0:16.13
/Library/Application Support/Symantec/AntiVirus/SmallScanner.app/Contents/MacOS/SmallScanner

Unfortunately /Library/Application Support/ is writable by group admin and by
mv'ing the Symantec subdirectory out of the way and installing a tree of
symlinks in its place except for malware in place of SmallScanner arbitrary
code can be executed as root. That is you can trampoline from group admin to
user root. (There were a variety of other ways to do this that were coming out
of the woodwork back in January.)

Using the symlink method described above to replace SmallScanner with the following shell script...

#!/bin/sh

touch /tmp/uhoh.this.is.very.bad.news
exec /Library/Application\ Support/Norton\ Solutions\ Support/Norton\ AntiVirus/SmallScanner.
app/Contents/MacOS/SmallScanner


And then inserting a disk (or using hdiutil to mount a disk image) results in normal looking behavior along with a new file...
-rw-r--r-- 1 root wheel 0 Jan 15 20:10 /tmp/uhoh.this.is.very.bad.news

Etc.

I'd like to thank the Apple Product Security team for being more forthright in their communication with me on this issue, it was a lot better than my previous interactions. I'd like to apologize to all the users that have been unknowingly insecure for the past 289 days, I think I like full disclosure better too given the way vendors seem to drag their feet.

Review: Issaquah Brew House

After a long day of Eileen and Jeanie studying constitutional law and me painting an interior wall, Tom stopped by and we decided to head out to the Issaquah Brew House for dinner. We were met with what I can only describe as the most untimely service I've ever received at a restaurant.

It made a good excuse to give the "Submit a review" feature of Google Maps a whirl. The review is online.

The experiences have led me to wonder what the optimum server to table ratio is. I know there's a lot of clamor on the internet that claims that the gruff service in Europe is because servers handle more tables and (as a result) have to be somewhat more efficient, and have less time to be personable. Presumably someone with experience in restaurant management would have a good idea, but 6 tables seems like a reasonable maximum from what I'm reading online.

How Long Do You Wait For Responsible Disclosure?

A vocal minority of people were critical of the timeline last time I found a security flaw so let's try this: (names changed to protect the innocent and the guilty)

• 2007-Jan-16 - Reported local privilege escalation issue to Vendor D
• 2007-Jan-17 - Vendor D will "review as soon as possible"
• 2007-Jan-31 - Vendor D "working on planning a fix", will coordinate release when product update has ETA
• 2007-Apr-20 - Vendor D thinks it would be better if Vendor J made changes to fix the problem in Vendor D's software
• 2007-Jun-21 - Contacted Vendor J for status
• 2007-Jun-22 - Vendor J says precisely which changes they're going to make for their next release (slated for Q4). These changes are not enough to protect Vendor D's software.
• 2007-Aug-29 - Vendor D says they've made suggestions to Vendor J, but otherwise aren't going to do anything to fix this issue
• 2007-Oct-01 - It turns out that Vendor J's recent builds indeed don't do enough to cover up the vulnerability in Vendor D's software

Beyond that, people running current versions of vendor J's software will be vulnerable forever because vendor D says they aren't going to do anything about it: (from Vendor D's August 29th email)

[Vendor D] does not currently plan to make changes to our existing products
to directly address the problem you reported. The changes would require
considerable architectural modifications for those products, and the changes
could cause other problems for those products if [Vendor J] subsequently
release[sic] an ... update to address the underlying problem.

My immediate thoughts are that a disclosure deadline late this month would be appropriate at this point. That'll give me time to prepare a third-party patch to make the fix that Vendor D should've made ages ago, and give the parties some additional PR prep time (beyond the 9 months they've already received). I have to say that Vendor J really isn't at fault here, Vendor D made a stupid mistake and now doesn't want to take the time to fix it.

But this is blog land, so maybe people have thoughts.

Update: I've told the vendors November 1 (or earlier at their discretion).

...but how many Big Macs will it buy?

There was a conversation about investment at work that mentioned stocks, gold and the "Big Mac Index" so I decided to do a quick analysis.

Year	Big Mac	DJIA	Big Macs per DJIA	$/oz Gold	Big Macs per oz Gold
2001	$2.54	10790	4248	271	106
2002	$2.49	10259	4120	278	111
2003	$2.65	8601	3245	343	129
2004	$2.90	10409	3589	417	143
2005	$3.15	10827	3437	428	135
2006	$3.15	10883	3454	530	168
2007	$3.22	12474	3873	640	198

It didn't quite work out the way I had expected it to. Plotting the CPI (or the Euro) against these numbers might also be interesting.

Data Validation Rant

I just signed up to renew my ACM membership for the first time since I was a student back at WWU. I have to admit to feeling kind of bad about having finally caved in to the direct mail they've been bombarding me with, but it did offer a reasonable discount. So I'm busy filling out their online form that asks for all sorts of information they can use to direct additional marketing at me and I click the "continue" button, but alas I cannot continue.

So why can I not continue? Everything I entered is correct, I double-checked. It turns out I put a dash in my zip+4 code, and for them that's an error. The US Postal Service disagrees, and in fact insists that xxxxx-xxxx is the "standard format" for a zip code. One would hope that the ACM of all organizations could figure out how to validate data without looking stupid, but apparently that's too much.

This is a real frustration in lots of other online settings as well. There are quite a few online merchants that refuse to process your payment unless you leave the spaces out of your credit card number. They display stern warnings like "Important: Please do not put spaces or dashes between credit card numbers." If they can't even figure out how to remove extraneous spaces or dashes from the information I give them, should I be trusting them to get the rest of my order correct and not have my personal information stolen? Dealing with that input is maybe 4 lines of code if you're trying to make it hard. It's usually not the merchant's fault, they're just buying some third party service. Any service that merchants are having to pay for that can't manage to get this right should be publicly shamed for their lack of competence.

Email addresses are also apparently a tricky beast for people who are building online tools. There are a lot of characters that are valid in an email address that various online services tend to choke on. There are numerous examples online disguised as "how-tos" that are actually "how-not-to-dos". The example I'm picking on in this case doesn't accept addresses with a + in them. Addresses like learn+to+read@rfc2822.int are in fact perfectly legitimate email addresses, as are addresses with hyphens, underscores, carets, tildes, equals and a lot of other somewhat obscure characters. Mail servers seem to deal successfully with these things, one would hope most
web application writers could cope with it too.

Come on all you folks that are building tools for the web, it's not hard to get these things right by just being little bit more pragmatic about it.

Moved to Blogger

As you may have noticed, I’ve moved the blog part of the site over here to Blogger’s new custom domain feature. Some old links to the blog pages may not work any more, but I've tried to get the majority of them to redirect correctly. My old custom hacked up Typo install was getting frayed around edges and its easier to let a free service handle this stuff for me as long as I can extract my data later.

It turns out that it's not too hard to take the data from Typo articles by getting them by id number and then finagle it into something that the Blogger API will take and do the migration programmatically. I did have to clean up the XHTML in places where tags hadn’t quite lined up, but it wasn't all that much work.

BOM Shelter: MoAB 5, 8, 15 Permissions Fix

Mac OS X, and a number of programs by third parties, have some risky permissions by default. I’ve taken the work I did a couple weeks ago and updated it to cover over more of these problems (MoAB days 5, 8, and 15). It also has a new more clever name: bom-shelter.py (sig)

To get this script, simply save the bom-shelter.py link to your disk.

To verify the script you get is one I wrote, you can download the signature and gpg --verify bom-shelter.py if you happen to have GnuPG installed and have a reason to trust my public key (0x4185664C).

To use this script, you must, from an admin account, run sudo python bom-shelter.py in Terminal, iTerm, or some other reasonable equivalent.

This script does the following for each of the MoAB advisories listed above:
#5: The permissions on BOM files are made more secure and /Library/Receipts (and important descendants) get a sticky bit to prevent shenanigans.

#8: /Library/Frameworks gets a sticky bit to prevent potential adversaries from being able to replace components that Application Enhancer runs as root inside Application Enhancer.framework.

#15: The three setuid root programs that can be overwritten by members of the “admin” group in /Applications/Utilities mentioned in the advisory are changed to not be “admin”-writable. This is also done to /Applications/System Preferences.app/Contents/Resources/installAssistant which has a similar vulnerability.

For all of these things, the script also edits the BOM files in /Library/Receipts to ensure that if you “repair permissions” on your disk these vulnerabilities will not reappear. The BOM file format is not very well documented so these edits may or may not work for you, but they should not corrupt the file. The editing function is careful to only change values if they are what is expected, otherwise it’ll print a warning and not make a change. Backup versions of your BOM files are saved as part of this process.

If you happen to have Application Enhancer installed, in order to secure your machine with Landon Fuller’s awesome MoAB Fixes or any other reason, please take the time to secure /Library/Application Enhancers outside your home directory and ~/Library/Application Enhancers inside your home directory. Malicious code can write things there without your permission and if Application Enhancer uses those patches without asking you, it might make you sad.