A Little Bit Faster Now

The DSL link was upgraded from 1.5Mb/s down and 256kb/s up to 3Mb/s and 512kb/s respectively this morning. Which reminds me that getting the most out of the link requires some work. Daniel Hartmeier has a nice write-up on prioritizing ACKs which is good idea on PPPoE links so that you can upload and download simultaneously without killing your throughput one way or the other.

PPPoE especially seems to have issues with empty ACKs getting caught in buffers and having to wait 1 second or more to be fed through and out. The following user PPP setting in FreeBSD is of particular interest.

set ifqueue packets




Set the maximum number of packets that ppp will read from the
tunnel interface while data cannot be sent to any of the available links. This queue limit is necessary to flow control outgoing data as the tunnel interface is likely to be far faster than
the combined links available to ppp.





If packets is set to a value less than the number of links, ppp
will read up to that value regardless. This prevents any possible latency problems.





The default value for packets is “30”.

If we’re uploading a lot of data, those 30 default packets will tend to be the maximum MTU size (generally 1492 for PPPoE). That’s 44KBytes of data waiting around to be transmitted, or 358kbits. If the link is only pushing 256kbits each second, new items in the queue have to wait nearly a second and a half to get out. If your TCP ACKs for downloads start getting delayed by that long, the throughput on that connection is going to suffer and VoIP or gaming is simply out of the question.

With faster links, this default is somewhat less problematic. Our new 512kb/s link still has a buffer of 700ms or so. Still more than enough to confuse the heck out of the in-flight packet management in TCP when that buffer wait starts fluctuating wildly.

I suspect if ifqueue was set very small and some sort of weighted fair queuing was used to manage its input, that total performance would be better. And it would avoid having to guess the optimal upstream bandwidth the way you do with the suggested approach with pf.

Universal Build of PostgreSQL 8.1.3

Traditionally I’ve used DarwinPorts for managing the packages installed on my Mac laptop. Sadly, there has been some falling out between the OpenDarwin folks and Apple recently. For this and other reasons having DarwinPorts build Mac OS X-friendly Intel binaries cleanly (let alone universal binaries) seems highly unlikely.

In order to produce things that’ll work on both PowerPC Macs and ICBMs (Intel-CPU based Macs, what an acronym) you have to build it yourself or find someone else who has taken the time to do it.

The Mac Python crew have been working for a couple months now on a universal Python 2.4.2 install that passes its own regression testing (unlike the one that Apple ships). They have a copy of the installer .dmg available online. (There is also a universal Ruby online if you swing that way.)

One of the problems I’ve encountered, after using these builds, is that the universal Python really wants its C-based modules compiled as universal builds also. It’s not such a big deal when the modules are standalone (and pay attention to the CFLAGS that the Python build was made with). It is more problematic when they want to link against other things, which then must also be built universal.

In today’s adventure, I wanted to build the psycopg module that is a way to access PostgreSQL databases from inside Python. This meant making a universal build of PostgreSQL. Marc Liyanage has some universal builds available, but they were for 8.1.2 rather than 8.1.3. His ViewCVS is install is broken right now, but I was able to get the Perl snippet from the Google cache to get the build to succeed.

PostgreSQL actually builds remarkably easy aside from an oft-repeated bit where it takes the myriad object files (.o) from each smaller component and makes them into one larger object file. Apparently ld doesn’t know how to consolidate “fat” object files. To workaround this you have to run ld once for each architecture (ppc, i386) then lipo the resultant files together. Marc’s one-liner takes care of this change in the various makefiles.

find . -name Makefile -print -exec perl -p -i.backup -e 's/\Q$(LD) $(LDREL) $(LDOUT)\E (\S+) (.+)/\$(LD) -arch ppc \$(LDREL) \$(LDOUT) $1.ppc $2; \$(LD) -arch i386 \$(LDREL) \$(LDOUT) $1.i386 $2; lipo -create -output $1 $1.ppc $1.i386/' {} \;

Psycopg ignores CFLAGS given to it during configure and took a slight bit of monkeying around to use the magic flags that are in the Apple documentation to build. Other than that it went smoothly though.

I stuck the PostgreSQL build into /Library/Frameworks/PostgreSQL.framework for good measure. I’m not sure if that’s a good or bad idea, but it does keep it from getting stomped on by native builds done in OpenDarwin or Fink or whatever.

It’s about 8MB all wrapped up so I’m not going to post it here, but if people really want it maybe I can work something out. That’s all for now…