Regenerative Braking for Services

George Reese has a missive over on O’Reilly’s site about why auto-scaling your “cloud” application is a bad idea. He starts from the naïve case where scaling your computing without bounds leads to your expenses scaling without bounds as well. Okay, that makes sense. Then he goes on to explain that setting those bounds to do the right thing is too hard, and involves humans doing capacity planning, so you should just do better capacity planning with humans and leave the automation out.

Now I’m a big fan of robots and having machines do tedious work for me, so this claim holds little truck with me. Frankly, the words “too hard” translated as I read them to “you could have a strategic advantage over competitors if you do this well.” Unsurprisingly, I’m not the only one that feels this way, and, in fact, several people chimed in with refutations and examples of how they're already doing this today to great advantage.

A response by Sam Curren, Really Bad Reasons Not To Auto-scale, refuted most of the “it’s too hard to get it right” arguments. Adam Jacob had a good comment as well, if you’re monitoring the wrong things, it is in fact easy to get wrong. In fact, one can look to Don MacAskill’s post about smugmug on EC2 to see some examples of what measuring the right things can look like. Breaking things apart into pieces that are easier to measure is an implicit piece of Don’s discussion that probably warrants more discussion another time.

One thing that hasn’t been mentioned yet in this conversation is that if you don't degrade gracefully under pressure in any of these models you've already lost. If your service is starting to degrade (or know it’s about to) the only hard part is knowing whether to grin and gracefully degrade under the temporary pressure, or bring in more capacity. Thing is, humans are quite capable of making the wrong call here, and even if they make the right call, they’ll do it much slower and they won’t do it in the middle of the night when your service suddenly gets an unanticipated spike in popularity in Japan.

Back to the mental translation, if you can develop good algorithms (or even very simple ones) to better predict when to scale up and down, you save a lot of money that is traditionally blown on idle resources in slack times. Those idle resources can be turned off, or pressed into use for non-time-critical batch work, or even sublet them to someone else to do processing with. And in fact this last one is quite probably the business that EC2 and App Engine probably represent. “Here’s some spare resources let’s sell some usage on them rather than making $0 on resources that are continuing to costing money to run.” (That other large cluster players aren’t involved this market yet indicates they either don’t have enough capacity as it is, or they aren’t in a position where they care about that idle cost yet, or they just don’t get it. It’s another interesting conversation in and of itself.)

In any case, being more efficient about resource usage represents a competitive advantage that can make a big difference. It’s like the regenerative braking on hybrid cars. Many people just afford the cost of wasting that energy as heat, perhaps not even knowing that there is a better way. However, with some initial investment and knowhow you can capture some of it and realize greater efficiency and a cost savings to boot.

Boggle Solver

I've been working on a pet project with App Engine to try and get a better feel for it. It’s a Boggle puzzle solver that does some AJAXy tricks to multithread the solving work. The source code is online as well for anyone that’s curious. (There is a popular knockoff called “Scramble” on Facebook that is either different enough to keep Hasbro from filing a lawsuit or Hasbro’s lawyers are waiting around for them to make some money before bothering.)

In the case of App Engine it's particularly sensitive to requests that take “too long” to process. This was a particular hassle when I was importing the dictionary that is used. In order to solve a Boggle puzzle fairly quickly you want to have all the possible words arranged in a trie. This way you can stop quickly if you’re following letters that will never spell anything as you traverse the puzzle board. It took splitting the dictionary into 5000 separate pieces to get it to load without pushing me over my quota for “long” requests. Luckily we only have to do that once.

Next came the challenge of the puzzle solving itself. Again, solving the whole board in one request takes a while to process, apparently more than is allowed without running into that “long” request quota. Even in an optimized form (see the links to Dan Vanderkam’s work at the end of this entry), the full dictionary trie is 3MB and that takes a non-trivial amount of time to load in when you’re trying to handle requests within a few hundred milliseconds.

The solution was to reload the dictionary again but this time to break it up by initial trigrams. For every initial three letters, I store the appropriate shard of the dictionary (all the words that start with that combination) as a trie. There is also a blacklist of trigrams that form no words (for instance “frw”).

The javascript calls in with a copy of the puzzle and a point on the board to solve from. The server code then finds all the trigrams starting from the specified point and loads the appropriate dictionary shards. Since we're only solving from one point on the board there won't be more than 72 shards to load for any javascript call. (9 directions from the point and then 8 directions from each of those points because we're not allowed to backtrack.) The server then traverses the board using the dictionary tries hunting for words. When it finds them it stores the word and the places on the board where the word was found.

This information is all reduced into JSON and returned back to the browser that made the javascript call. The javascript on the browser is then responsible for taking all the found words and locations and sorting them in a sane way and displaying them for the end user.

Dan Vanderkam has written some interesting code and blog posts about optimizations in solving Boggle puzzles.

Security Advisory: Norton AntiVirus for Macintosh

Synopsis

Symantec's Norton AntiVirus for Macintosh (NAV) contains a vulnerability that can lead to local privilege escalation from group admin to root (the super-user) without any of the usual password prompts Mac OS X presents for gaining super-user access. Group admin includes any users with the "Allow this user to administer this computer" box checked, this generally includes the first user created in an OS X install. This vulnerability is caused by a setuid-root binary in NAV which automatically runs another binary in a location where it can be replaced by users with group admin permissions. Since most Mac OS X users are in group admin on their computers, most NAV users will be vulnerable.

Mitigation

The easiest (and most foolproof) mitigation strategy is to uninstall NAV. (I sure don't feel very secure when a vendor allows a local privilege escalation vulnerability to fester in their security software for over 9 months. Your feelings may vary.)

Set the sticky bit on all directories between the vulnerable binary and the filesystem root that are writable by group admin. This can be done in Terminal.app with sudo chmod +t / /Library /Library/Application\ Support /Library/Application\ Support/Symantec /Library/Application\ Support/Symantec/SmallScanner.app etc. Keep in mind that running "Repair Permissions" on your disk will remove this change and leave your NAV install vulnerable once again. Apple has set the sticky bit on / and /Library by default on Mac OS X 10.5, but not /Library/Application Support and obviously not on the directories that NAV is installed into.

What Symantec Had to Say

The last time I heard from Symantec was August 29th:

As you know, Symantec developers reviewed the issue concerning NAV for the Mac that you sent to us, and agreed that there is cause for concern. However, they felt that the same problem could potentially affect other vendor’s[sic] software as well. We contacted Apple Product Security, and suggested some changes that could improve security for everyone.
...
Symantec does not currently plan to make changes to our existing products to directly address the problem you reported. The changes would require considerable architectural modifications for those products, and the changes could cause other problems for those products if Apple subsequently release an OS update to address the underlying problem.
...

Timeline

Just in case people think the vendor didn't have enough time to address this, here's the timeline:

• 2007-Jan-16 - Reported issue to Symantec


• 2007-Jan-17 - Symantec will "review as soon as possible"


• 2007-Jan-31 - Symantec "working on planning a fix", will coordinate release when product update has ETA


• 2007-Apr-20 - Symantec thinks it would be better if Apple made changes to Mac OS X to fix the problem in NAV


• 2007-Jun-21 - Contacted Apple for status


• 2007-Jun-22 - Apple communicates the changes they're going to make for Leopard. These changes are not enough to workaround Symantec's vulnerable software.


• 2007-Aug-29 - Symantec says they've made suggestions to Apple, but otherwise aren't going to do anything to fix this issue (see the quote in the previous section)


• 2007-Oct-01 - It turns out that Apple's Leopard builds at the time didn't do enough to cover up the vulnerability in NAV


• 2007-Oct-11 - Contacted Symantec and Apple to let them know I was going to post this Nov-1


• 2007-Nov-1 - Today.

Details

SmallScanner is run as root by the setuid-root binary
/Library/Application Support/Symantec/AntiVirus/DiskMountNotify.app/Contents/MacOS/DiskMountN
otify. E.g. after inserting a data cdrom:
root 8734 28.9 -2.1 616152 43596 ?? U 7:51PM 0:16.13
/Library/Application Support/Symantec/AntiVirus/SmallScanner.app/Contents/MacOS/SmallScanner

Unfortunately /Library/Application Support/ is writable by group admin and by
mv'ing the Symantec subdirectory out of the way and installing a tree of
symlinks in its place except for malware in place of SmallScanner arbitrary
code can be executed as root. That is you can trampoline from group admin to
user root. (There were a variety of other ways to do this that were coming out
of the woodwork back in January.)

Using the symlink method described above to replace SmallScanner with the following shell script...

#!/bin/sh

touch /tmp/uhoh.this.is.very.bad.news
exec /Library/Application\ Support/Norton\ Solutions\ Support/Norton\ AntiVirus/SmallScanner.
app/Contents/MacOS/SmallScanner


And then inserting a disk (or using hdiutil to mount a disk image) results in normal looking behavior along with a new file...
-rw-r--r-- 1 root wheel 0 Jan 15 20:10 /tmp/uhoh.this.is.very.bad.news

Etc.

I'd like to thank the Apple Product Security team for being more forthright in their communication with me on this issue, it was a lot better than my previous interactions. I'd like to apologize to all the users that have been unknowingly insecure for the past 289 days, I think I like full disclosure better too given the way vendors seem to drag their feet.

Data Validation Rant

I just signed up to renew my ACM membership for the first time since I was a student back at WWU. I have to admit to feeling kind of bad about having finally caved in to the direct mail they've been bombarding me with, but it did offer a reasonable discount. So I'm busy filling out their online form that asks for all sorts of information they can use to direct additional marketing at me and I click the "continue" button, but alas I cannot continue.

So why can I not continue? Everything I entered is correct, I double-checked. It turns out I put a dash in my zip+4 code, and for them that's an error. The US Postal Service disagrees, and in fact insists that xxxxx-xxxx is the "standard format" for a zip code. One would hope that the ACM of all organizations could figure out how to validate data without looking stupid, but apparently that's too much.

This is a real frustration in lots of other online settings as well. There are quite a few online merchants that refuse to process your payment unless you leave the spaces out of your credit card number. They display stern warnings like "Important: Please do not put spaces or dashes between credit card numbers." If they can't even figure out how to remove extraneous spaces or dashes from the information I give them, should I be trusting them to get the rest of my order correct and not have my personal information stolen? Dealing with that input is maybe 4 lines of code if you're trying to make it hard. It's usually not the merchant's fault, they're just buying some third party service. Any service that merchants are having to pay for that can't manage to get this right should be publicly shamed for their lack of competence.

Email addresses are also apparently a tricky beast for people who are building online tools. There are a lot of characters that are valid in an email address that various online services tend to choke on. There are numerous examples online disguised as "how-tos" that are actually "how-not-to-dos". The example I'm picking on in this case doesn't accept addresses with a + in them. Addresses like learn+to+read@rfc2822.int are in fact perfectly legitimate email addresses, as are addresses with hyphens, underscores, carets, tildes, equals and a lot of other somewhat obscure characters. Mail servers seem to deal successfully with these things, one would hope most
web application writers could cope with it too.

Come on all you folks that are building tools for the web, it's not hard to get these things right by just being little bit more pragmatic about it.

BOM Shelter: MoAB 5, 8, 15 Permissions Fix

Mac OS X, and a number of programs by third parties, have some risky permissions by default. I’ve taken the work I did a couple weeks ago and updated it to cover over more of these problems (MoAB days 5, 8, and 15). It also has a new more clever name: bom-shelter.py (sig)

To get this script, simply save the bom-shelter.py link to your disk.

To verify the script you get is one I wrote, you can download the signature and gpg --verify bom-shelter.py if you happen to have GnuPG installed and have a reason to trust my public key (0x4185664C).

To use this script, you must, from an admin account, run sudo python bom-shelter.py in Terminal, iTerm, or some other reasonable equivalent.

This script does the following for each of the MoAB advisories listed above:
#5: The permissions on BOM files are made more secure and /Library/Receipts (and important descendants) get a sticky bit to prevent shenanigans.

#8: /Library/Frameworks gets a sticky bit to prevent potential adversaries from being able to replace components that Application Enhancer runs as root inside Application Enhancer.framework.

#15: The three setuid root programs that can be overwritten by members of the “admin” group in /Applications/Utilities mentioned in the advisory are changed to not be “admin”-writable. This is also done to /Applications/System Preferences.app/Contents/Resources/installAssistant which has a similar vulnerability.

For all of these things, the script also edits the BOM files in /Library/Receipts to ensure that if you “repair permissions” on your disk these vulnerabilities will not reappear. The BOM file format is not very well documented so these edits may or may not work for you, but they should not corrupt the file. The editing function is careful to only change values if they are what is expected, otherwise it’ll print a warning and not make a change. Backup versions of your BOM files are saved as part of this process.

If you happen to have Application Enhancer installed, in order to secure your machine with Landon Fuller’s awesome MoAB Fixes or any other reason, please take the time to secure /Library/Application Enhancers outside your home directory and ~/Library/Application Enhancers inside your home directory. Malicious code can write things there without your permission and if Application Enhancer uses those patches without asking you, it might make you sad.

Counting Trigrams for Fun

There is a thread over at the forums for the xkcd comic strip with a puzzle game to find trigrams. To help out my brain which has been full of number-related work lately, I whipped up a Python script “trigramtastic.py” to help find more challenging trigrams for the game. Feeding it things like /usr/share/dict/web2 can produce some helpful results, although there are a lot of other potential data sources.

Universal Binary GPG 1.4.6

I’ve made a universal build of GnuPG 1.4.6 (sig). This version is not vulnerable to an attack described in a December security announcement. Copy the contents of this zip into /usr/local/bin to replace the vulnerable binaries.

The source is available from the GnuPG project.
The build was made by making separate directories for Intel, PowerPC and PowerPC 64-bit builds and then using lipo to stitch them all back together again.

The PowerPC 64-bit code may be somewhat slower since certain operations are not optimized in assembly for that platform.

Installing using the MacGPG installer and then copying in the binaries provided above should result in an up-to-date install that is not vulnerable.

MoAB Day 5 Fix Script

The fix for day 5’s bug sadly can’t be affected through runtime patching alone. It requires changing the permissions on disk of a number of files and directories which are vulnerable to being edited by default. I’m providing scripts you can run in Terminal.app to change these permissions to safe values.

• bom-safety.py (sig) changes the permissions to safe values.


• bom-unsafety.py (sig) changes the permissions back to Apple’s original (unsafe) values.

You can run these scripts in Terminal.app as root using sudo:

sudo /usr/bin/python bom-safety.py

It’ll print a message when it is done or if it encounters a problem.

Finlay Dobbie pointed out that certain Apple installers like X11.app or XCode Tools may change these permissions back to vulnerable values again.

In order to re-run the bom-safety script you must rename the backup it creates at /Library/Receipts/BaseSystem.pkg/Contents/Archive.bom.orig.

• Set the sticky bit on /Library/Receipts


• Set the sticky bit on the paths down to each of the critical BOMs


• Unset the group-write bit on the critical BOMs


• Create root-owned 0-length place holders for critical BOMs/paths
  that don’t exist


• Backup /Library/Receipts/BaseSystem.pkg/Contents/Archive.bom


• Make a 1-bit change to the
  /Library/Receipts/BaseSystem.pkg/Contents/Archive.bom file that causes
  repair permissions to keep the sticky bit set on /Library/Receipts
  rather than removing it.


• Print a completed message

As always, you shouldn’t run code when you can’t understand it yourself, trust someone who understands it, or trust the author of the code. For those wondering if I actually wrote the code downloadable above I am providing GPG signatures above for your review. Ironically, the current available GnuPG for Mac has a code execution security hole since early December. I’m building new universal binaries of GPG now and will post them later today.

Month of Apple Bugs

An old friend is posting runtime fixes for the bugs of the Month of Apple Bugs.

Protect yourself. Hopefully Apple will have improved their response time to these sorts of issues, and hopefully the user community will not try to blow smoke at people about real problems.

Update: Some people have asked for my thoughts on response time to these sorts of issues after my own experience interacting with Apple’s product security team.

My experience was over 3 years ago now and presumably Apple has improved from that and similar experiences with others in that time. Three years is an eternity in the software industry, even in large organizations with a lot of inertia. My experience way back then was struggling to get any verifiable confirmation that they were working on the problem.

It was a frustrating experience; trying to overcome any situation of mutual distrust is very difficult. While it is understandable that it takes a while to develop and test changes, that can’t be used as a blanket excuse to keep the people reporting issues in the dark about progress on them.

On the other hand, it sounds like the Month of Apple Bugs folks are giving no advance notice at all to Apple and the open source project VLC. It is impolite of them, even supposing that one or both were notoriously reticent about releasing fixes. I don’t think their actions rise to the level being negligent under current social standards, but then, there’s no law against being a jerk.

Making a Time-Lapse Movie

Our webcam now has a time-lapse movie for today and yesterday. Or rather it will have one for yesterday after it’s been running for more than one day. This was achieved by saving off a copy of every webcam image into a directory for the day and then processing them with some free tools. In this post I walk through the process we use to make this happen.

First, my Gawker image fetch project grew a new -d directory option to save JPEG files into a directory. We have to save the images if we expect to make a movie out of them at some point. It should be noted this requires a bit of disk space. With a new image every 30 seconds, a day of images occupies nearly 150 megabytes of disk.

Now that we have the images, we want to turn them into a movie. For this we start with netpbm.
Using jpegtopnm we convert the series of JPEG images into a series of raster images that the next software can deal with.

Next in line is mjpegtools. It has a tool ppmtoy4m that takes the series of raster images and renders it into a “Y4M” video file. The Y4M format adds useful information to traditional YUV video data, such as the size of the image and the framerate. It’s worth noting that this Y4M data is mainly raw video frames, which results in a fairly large file if we write it to disk.

Lastly, we want to use x264 to convert the Y4M data into a highly-compressed movie that your can view with your web browser. The only problem is that x264 treats it’s input as raw YUV data by default and has no switch to make use of Y4M data, it only uses Y4M data if the input file is named something.y4m.

But we don’t want to save a very large Y4M file to our disk every 10 minutes when we’re generating the movie, only to throw the file away when we’re done. We want to just send it the input directly. All the other tools allow us to pipeline the commands together so as one command processes some data it passes it’s output straight on to the next command. So now it was time to fix x264 to let us pipeline the way we want:

I sent this message to the x264-devel list this morning.

This patch adds the long option “--y4m-input” which sets b_y4m. It
also avoids any evaluation that might result in setting both b_avis
and b_y4m.

In order to politely handle pipelines from mjpegtools and other y4m
sources, b_y4m must be set even if the filename is ”-”. Sadly, that filename
does not end in ”.y4m”.

The patch is available here.

In the longer term it would be useful to offer options for all
accepted input and output types on stdin and stdout to avoid having to
create (rather sizable) temp file everywhere.

So with that fix in place now we can send a series of JPEG files through the pipe of jpegtopnm, ppmtoy4m, x264 and wind up with a time-lapse movie you can watch.

Update: The files are now also mangled by the qt-faststart program (part of ffmpeg) which will let QuickTime (and possibly other players) start playback immediately before the whole file is loaded. I’ll try to get this change committed to svn soon.

Cooler MacBooks

There was just an article on Slashdot about the new tool Hendrik Holtmann developed, smcFanControl, that allows more control over the minimum fan speed in Apple’s MacBook and MacBook Pro computers.

Apple sets the fan speed to 1000 RPM by default. This choice results in a surface temperature well in excess of what’s comfortable on both the bottom and top of the “laptop”. In fact it’s so warm that Apple advises users to “not leave the bottom of your MacBook Pro in contact with your lap or any surface of your body for extended periods” in the owner’s manual.

By using this new tool and setting the minimum to a more reasonable 2000 or 2500 RPM you can have a still quiet laptop that can actually sit in your lap comfortably. The effect on battery life should be negligible compared to the big power consumers (e.g. the display backlight, the hard drive).

Best yet, it’s open source, not another $10 piece of “shareware”, so everyone can use and improve it.

Update: There’s an even better program for handling this called Fan Control that is also free software, and has a handy pane in System Preferences.

Not My Dream App

There’s a new contest at MyDreamApp.com for new Mac OS X software. The format for the contest with celebrity judges and open submissions from the user community and prizes for the top 3 entrants sounds like a lot of fun. A lot of fun, that is until you read the fine print.

Participants are forbidden from making their dream app open source if it’s selected as a winner. That’s right, by participating in the contest you agree to give up all interest in your idea and any code you put behind it to the group organizing the contest. If you wanted to give back to the community by offering your awesome idea for free or, better yet, offering it as open source software, you’re out of luck.

But wait, there’s more; the deal gets even better. In return for giving away your idea they’ll offer you a meager 15% royalty from ”net income” from the Mac version of the idea. “Net” in this context means if they spend money advertising, promoting or paying developers/QA/writers to work on the Mac version of the product that gets subtracted from the amount you get 15% of. This is exactly how many musicians get screwed out of the proceeds from their music by record companies. “You know it cost a lot to promote your album, so here’s a free cup of coffee. We look forward to your next effort.”

Just to top it all off, some top finishers get free equipment. The catch? The taxes on the giveaway are your responsibility. So you have to pay out of pocket federal, state and local taxes (let’s say 28%) of the value of the “free” item. I’m going to guess that the 15% of “net income” may not be enough to cover the taxes on the free item.

Did I mention the liability release provision potentially allows the organizers to skip out and not pay you anything? And if they do pay you it will be through PayPal. And if your submission is accepted you may not even be able to claim it was your idea on your resume without their permission. And, worst of all, if the effort required for them to follow through on making your idea a shiny product is more than “reasonable” they can just walk away and develop it for Windows instead. (And pay you nothing.)

It’s an interesting way for some people to feed on the creativity of others, and I guess if people understand all this and still participate, that’s their business. I’m a little dismayed that Woz and other luminaries are participating in something that seems so potentially abusive of it’s “winners”.

This isn’t to suggest that the people organizing the contest will do any of these nasty unkind things, only that they could if they wanted to. I don’t know any of them, and I hope, for the sake of those who participate, they stay honorable.

It would be nice if someone had an open source-friendly contest (with nicer rules) along these lines for great new open source Cocoa titles in the tradition of Adium and Colloquy.

Ignore the pkgsrc disk image instructions

The pkgsrc distribution suggests (and offers tools for) making a fixed-size disk image for OS X and Darwin users. This not the best approach with OS X 10.4 and later.

The following command line incantation can create a resizable HFS+ disk image that is case-sensitive and will grow to fit what you need:

hdiutil create -fs HFSX -fsargs "-c c=64,a=16,e=16 -s" -volname pkgsrc -type SPARSE -stretch SizeOfYourDisk -size SizeOfYourDisk -ov /SomewhereOnYourDisk/pkgsrc

You can avoid silliness with your PATH by adding symlinks:

ln -s /Volumes/pkgsrc/pkg /usr/pkg

ln -s /Volumes/pkgsrc/pkgsrc /usr/pkgsrc

ln -s /Volumes/pkgsrc/pkgdb /var/db/pkg

You can compact this image later to reclaim disk space with:

hdiutil compact /SomewhereOnYourDisk/pkgsrc.sparseimage

You can later use hdiutil resize if you move it to a disk that is a different size. Mounting can also likely be automated to a large degree, maybe the lazyweb can chime in with ideas along those lines.

Update: Don’t, for the love of your files, put the disk image inside a FileVault home directory (or any other sparseimage disk image). I had to borrow a PowerPC Mac and buy a copy of DiskWarrior to get my files back. DiskWarrior apparently doesn’t work with ICBMs yet.

Gawker Image Fetch Script Improved

I’ve made some improvements to Gawker Image Fetch and moved it over to Google Code Hosting.

If you checkout the copy from Subversion over there you’ll gain the ability to specify the Gawker instance with Bonjour/Zeroconf, as well as having a new --foreground mode that fills your terminal with exciting debugging messages like:

Saw service ‘Unprivileged Clown._lapse._tcp.local.’


Source moved to 1.2.3.4:7548


Replacing connection


Connecting to 1.2.3.4:7548

The new Zeroconf features appear to work in both OS X and FreeBSD. Theoretically it should be able to follow the camera to a new IP if the camera machine reboots.

I’m still seeing an occasional bug that requires restarting the script in the install for the webcam. Hopefully the additional debugging information will help me get to the bottom of that.

Let me know if it works for you…

Thanks to the developers of pyzeroconf for that library. It has a few bugs which I’ve fixed for this script and haven’t submitted back yet.

Update: I sent a patch along by email to pyzeroconf interested parties.

Gawker Image Fetch Script

There’s not a lot more to it than the title suggests. I’ve made a script you can use to fetch webcam images from any host on the Internet that’s running Gawker.

The script takes arguments of the host and port to connect to as well as the file to overwrite with the latest exciting image. Together with some javascript you can do neat things like have a live updating webcam image.

Silly things like half-written images are dealt with, but that means the directory that has the image file in it needs to be writable. If the camera sends images faster than your local machine can write out to disk, that’s bad. So don’t do that.

In the future, I’ll make changes that fulfill desires I have, like producing a timelapse movie of the last 24 hours. And more complicated, a timelapse movie that skips the dark frames in the night that are less interesting.

After a few minutes of thought: Future changes like reconnecting in the event of a failure spring to mind as well. For now though, this is scratches-an-itch-ware.

Update: My script now resides at Google Code with the code available in Subversion at http://gawker-image-fetch.googlecode.com/svn/trunk/. It now supports finding cameras with Zeroconf and reconnecting automatically.

Later Update: New features are appearing, like support scripts for doing time-lapse movies. Check the code site for the latest information.

Oh... And There's One More Thing

I hope you’ll pardon the dearth of posts recently. In case you hadn’t heard, I’ve started at a new job as a Site Reliability Engineer for Google.

After a handful of weeks in Mountain View, I’ve made the 875 mile drive back home. Some highlights of the trip down and my free time down there will trickle into the gallery over the coming days.

Testing A New Webcam

We have a new test webcam pointed at Mount Si. I’ve always been
bothered by the bizarre behavior of the QuickCam when it catches the morning
sun. Using an iSight may help this since it theoretically is smart enough to do
some aperture adjustment on its own. On the other hand, it’s ability to pick out
details in the dark has proven to be somewhat more sketchy.

It is worth noting that the new webcam setup I’m testing is using only open
source software (aside from the operating system). Or at least it will be once
I’m happy with the performance and release the script I’m using to fetch images
that are served with Gawker.

The resolution of this camera is about the same as the QuickCam, but the image is a lot sharper. I have a feeling that the image tag on that page isn’t shaped quite right for the size of the image that is output from that camera. But that’s why it says “testing” all over the page. A chance to work the kinks out.

def ParseDescription(self, desc_buffer):
  pass

We’ll see if that’s premature optimization some time soon…

A Little Bit Faster Now

The DSL link was upgraded from 1.5Mb/s down and 256kb/s up to 3Mb/s and 512kb/s respectively this morning. Which reminds me that getting the most out of the link requires some work. Daniel Hartmeier has a nice write-up on prioritizing ACKs which is good idea on PPPoE links so that you can upload and download simultaneously without killing your throughput one way or the other.

PPPoE especially seems to have issues with empty ACKs getting caught in buffers and having to wait 1 second or more to be fed through and out. The following user PPP setting in FreeBSD is of particular interest.

set ifqueue packets




Set the maximum number of packets that ppp will read from the
tunnel interface while data cannot be sent to any of the available links. This queue limit is necessary to flow control outgoing data as the tunnel interface is likely to be far faster than
the combined links available to ppp.





If packets is set to a value less than the number of links, ppp
will read up to that value regardless. This prevents any possible latency problems.





The default value for packets is “30”.

If we’re uploading a lot of data, those 30 default packets will tend to be the maximum MTU size (generally 1492 for PPPoE). That’s 44KBytes of data waiting around to be transmitted, or 358kbits. If the link is only pushing 256kbits each second, new items in the queue have to wait nearly a second and a half to get out. If your TCP ACKs for downloads start getting delayed by that long, the throughput on that connection is going to suffer and VoIP or gaming is simply out of the question.

With faster links, this default is somewhat less problematic. Our new 512kb/s link still has a buffer of 700ms or so. Still more than enough to confuse the heck out of the in-flight packet management in TCP when that buffer wait starts fluctuating wildly.

I suspect if ifqueue was set very small and some sort of weighted fair queuing was used to manage its input, that total performance would be better. And it would avoid having to guess the optimal upstream bandwidth the way you do with the suggested approach with pf.

Universal Build of PostgreSQL 8.1.3

Traditionally I’ve used DarwinPorts for managing the packages installed on my Mac laptop. Sadly, there has been some falling out between the OpenDarwin folks and Apple recently. For this and other reasons having DarwinPorts build Mac OS X-friendly Intel binaries cleanly (let alone universal binaries) seems highly unlikely.

In order to produce things that’ll work on both PowerPC Macs and ICBMs (Intel-CPU based Macs, what an acronym) you have to build it yourself or find someone else who has taken the time to do it.

The Mac Python crew have been working for a couple months now on a universal Python 2.4.2 install that passes its own regression testing (unlike the one that Apple ships). They have a copy of the installer .dmg available online. (There is also a universal Ruby online if you swing that way.)

One of the problems I’ve encountered, after using these builds, is that the universal Python really wants its C-based modules compiled as universal builds also. It’s not such a big deal when the modules are standalone (and pay attention to the CFLAGS that the Python build was made with). It is more problematic when they want to link against other things, which then must also be built universal.

In today’s adventure, I wanted to build the psycopg module that is a way to access PostgreSQL databases from inside Python. This meant making a universal build of PostgreSQL. Marc Liyanage has some universal builds available, but they were for 8.1.2 rather than 8.1.3. His ViewCVS is install is broken right now, but I was able to get the Perl snippet from the Google cache to get the build to succeed.

PostgreSQL actually builds remarkably easy aside from an oft-repeated bit where it takes the myriad object files (.o) from each smaller component and makes them into one larger object file. Apparently ld doesn’t know how to consolidate “fat” object files. To workaround this you have to run ld once for each architecture (ppc, i386) then lipo the resultant files together. Marc’s one-liner takes care of this change in the various makefiles.

find . -name Makefile -print -exec perl -p -i.backup -e 's/\Q$(LD) $(LDREL) $(LDOUT)\E (\S+) (.+)/\$(LD) -arch ppc \$(LDREL) \$(LDOUT) $1.ppc $2; \$(LD) -arch i386 \$(LDREL) \$(LDOUT) $1.i386 $2; lipo -create -output $1 $1.ppc $1.i386/' {} \;

Psycopg ignores CFLAGS given to it during configure and took a slight bit of monkeying around to use the magic flags that are in the Apple documentation to build. Other than that it went smoothly though.

I stuck the PostgreSQL build into /Library/Frameworks/PostgreSQL.framework for good measure. I’m not sure if that’s a good or bad idea, but it does keep it from getting stomped on by native builds done in OpenDarwin or Fink or whatever.

It’s about 8MB all wrapped up so I’m not going to post it here, but if people really want it maybe I can work something out. That’s all for now…

USB Audio Out Broken on Intel Macs

Yes, the joys of having a Rev. A hardware product from Apple are boundless. My new MacBook Pro seems to have a problem dealing with USB audio output devices. According to a reply I got on Apple’s discussion boards this affects Intel-based iMacs as well.

Thanks to the fact that Apple won’t open the xnu (Mac OS X kernel) sources for the Intel systems, I can’t effectively chase down the issue myself (potentially saving some beleaguered Apple engineer some work rounding up a suitable test rig). So I’ve resorted to filing a bug with the Apple’s developer feedback interface: 4460973 – USB Audio Output Fails on Intel Macs. I’ve never used this before and my previous experience with Apple’s bugfix process was lackluster.

1. plug in a USB audio output device


2. select it for output in System Preferences


3. click Play in iTunes

Feb 25 21:57:09 macbook kernel [0]: USBF: 89320.749 IOUSBPipe[0x4244600]:ClosePipe for address 6, ep 1 had a retain count > 1. Leaking a pipe

I always figured there would be some sort of minor issue like this, being Rev. A hardware and all. On the whole the machine is great, but I imagine this would be a serious bit of sadness for DJ types that weren’t just using the built-in outputs. USB audio output devices work if, and only if, the device was plugged in at boot and the machine hasn’t been in “sleep” mode between then and now. So just reboot, over and over and over again! I’d just as soon leave that fun usability experience to the folks with that other operating system.

It’ll be interesting to see how quick Apple can get a fix for this. Since it affects iMac users as well it’s most likely a problem with the USB controller or the driver for it. However, it’s hard to say for sure without digging a lot deeper, and dig_deeper() is returning ENOXNUSRC.