mod_dav Quota Patch Redux

I haven’t written about my patch to Apache httpd’s mod_dav module to show free disk space lately, and it apparently fell off the website when I moved things to the new server with built-in UPS. Thanks to the Internet Wayback Machine and TNPI, I’ve recovered the text and patch on the new server.

Anyone who has used Mac OS X’s iDisk tool may have noticed that the window shows how much space is available for writing unlike most WebDAV servers. I noticed this and wanted this feature on my own WebDAV server and created a patch to mod_dav that adds the needed properties. The specification for sending this information to clients is not finalized so this patch may stop working for some clients some day when the specification changes and clients change with it.

The patch is available here and should apply cleanly to the source for recent versions of Apache 2.0 (it was originally written for 2.0.47). This patch shows disk free, not quotas if you happen to have them enabled. And even then, the only the quota that applies to Apache is the quota for the user it runs as.

If you notice in the patch there are a couple places where real big numbers get divided by 512. This is because for some strange reason, when MacOSX’s mount_webdav bits calculate disk free they want to be getting the stats in 512-byte blocks. All the documentation I’ve found about <d:quota> and <d:quotaused> seems suggest that this number should be in bytes. But then again in those specs it was <d:quota-bytes>. /sbin/mount_webdav also does some really funny stuff when these numbers get large, I wonder in my mind if there is a buffer overflow or something lurking under the surface in mount_webdav’s quota handling.

The patch is much nicer now than it was when I first tried it in the handles the quota property. It handles this property through the avenues it is supposed to rather than pasting a quota onto every single PROPFIND that comes along. Whether that PROPFIND is looking for DAV:quota or not.

Further down the correct path than I am right now is implementing something along the lines of statfs(2) or statvfs(2) in the Apache Portable Runtime. Arguably it might be a better idea to just have functions that return the total disk space and available disk space. Especially since Windows may or may not support statfs() in its POSIX API. But in any case, the raw system calls in repos.c right now in this patch aren’t really appropriate.

It would be nice if this patch could be integrated into the Apache httpd project. That however will depend on the statfs calls being integrated into the APR.

Unethical Funding at WordPress

Apparently the people behind the WordPress project’s website, which includes the project’s founder, have decided that a good way to fund the project would be to use their Google PageRank to link to a lot of canned articles on topics that generate more advertising revenue than others. Some more details on this matter are available from Andy Baio’s blog and Ars Technica

It is deeply troubling that people associated with open source projects use resources that only have value because of unpaid community contribution and cash in on them. Unless that conversion is of impeccable ethics and completely transparent you’re bound to have a problem on your hands. In the case of the “Open Projects” (now Freenode) IRC network, there was a lack of transparency that led to the conclusion that Rob Levin was in fact using the donations that were advertised to “help the network” to pay his rent and feed his family. Certainly noble purposes, except that every other IRC network is run by volunteers who do the work mainly for free and when money is going to them to pay their living expenses that much is made more clear. This made some members of the user community upset.

In the case of WordPress, there is both the lack of transparency as to where the funds were used and the fact that the funds are being raised in a way that is certainly unethical. The link and page hiding method used is a violation of the contract between WordPress.org and Google for the AdWords program which opens the site maintainer and the project itself to lawsuits from Google, should they be feeling particularly nasty. As it is, Google has reset the pagerank for WordPress.org to zero and WordPress.org links are slowly disappearing from its search results. It doesn’t help matters that the project founder who is ostensibly responsible for this decision was on an extended vacation to various countries in Europe when this story broke. The particularly cynical might wonder how the vacation was funded.

At best, this was a tragic mistake on the part of the people responsible. The offending links and articles have now been removed, but it looks as if the damage is done and it may be too late for forgiveness from Google and many users.

I’m going to have to think long and hard about how much I can contribute to the WordPress parent project without getting myself tangled up in their apparently shaky ethics. I already had misgivings about the way they used the default links in the application to promote their personal blogs rather than useful community resources. (The lead developer brags about being Google’s “#1 most important Matt in the world” on his personal page, a direct result of his link being in the default install.) I had been planning to embark on a project to make WordPress database agnostic and did not want to deal with having to be a fork from the parent project. Perhaps a fork from the parent project would be better for the community though. The coming days and how this is all handled will shine a better light on which way is best to go.

For what it’s worth, I should point out that at no time have I been paid for my work on WordPress-pg and if I do receive such payment in the future I’ll be transparent about where the money is going and what I’m doing to repay the community for any such kindness.

Can You Hear Me Now?

We’ve had one more hiccup in VoIP land. At first, we had only one provider for connecting long distance calls, SimpleTelecom. I’m not really sure what is wrong for them right now, but they aren’t completing calls. The IAX2 connector says “circuit-busy” and their call detail record says “Forbidden”. In any case, my call is not being connected and I want to talk to people. Good thing there are plenty choices and I’m not tied down.

In 15 minutes I setup an account with a competing long-distance provider, Teliax. Then with a small amount of change on the machine here, I was able to setup this new provider to be used in the event the first provider failed to complete my call. Voices didn’t transmit correctly when G729 compression was used, but forcing the call to G711-u seemed to fix things.

It was nice that adding new providers went relatively easily. On the other hand, people who get the “ready for your mother to use” services like Vonage, Packet8, and AT&T’s CallVantage are locked out of ever using another provider the same way people are traditionally used to with their normal plain old telephone service. When they have problems, there is no shopping for someone else that is working in the meantime. Vendor lock-in can be a bummer.

Population: Tire

The bicycle riding experiment came to an abrupt stop Wednesday night as my front tube popped rather loudly just before 5th & Mercer and my wheel rim sliced the tire up on the pavement as I came to a stop. I’ve got replacement tubes and tires in place now and should be ready to go for tomorrow, which will, according to forecasts, be a jaunt in the rain.

It Came From Progress Report on WordPress-Pg 1.5

I just checked in a little more progress on making things work. Lots of stuff to report and more general thoughts here as well…

• Fix the Category List order strangeness in wp-admin/categories.php


• Change the default links in the install to not include every person who ever wrote a line of WordPress code and instead point to the project pages and useful resources like the WordPress Codex.
  
  
  
  • I do realize this is a little on the gratuitous side, but I think it’ll provide helpful resources to new users. I don’t think new users want to see my blog.
  
  

• Page Parent in wp-admin/page-new.php fixed to work properly


• Pages list on visitor facing pages working


• Viewing of static pages working


• Fixed various uses of SQL year() to use extract(year from col) instead
  
  
  
  • It’s worth noting that this isn’t necessary if the ODBC functions are installed in PostgreSQL. There is probably some room for argument that we should just require that instead.
  
  

I think Keenan has been working on the upgrade scripts as well. Certainly a braver man than I, the way the parent project handles upgrading the schema scares me a fair bit. Before I saw today’s changes I almost wanted to suggest putting the kibosh on the whole “upgrade” script thing and just having an import script that takes care of it instead. In any case I think an “import from WordPress-MySQL 1.5” script probably has enough utility that it should be added to the to do list.

• Install from scratch


• Upgrade from 1.2 (?)


• New blog entries work


• New comments work


• New static pages work


• New categories work


• New links work

The redirect stuff doesn’t work for some pages due to more year() issues. It throws a database error that details the problem, so that should be easy enough to fix. The XMLRPC code still needs to be examined and from a mile high view it looks like a hornet’s nest or a hive of villany or something like that.

There is, of course, plenty left to be addressed, but once the WP-Pg 1.5 work is done, hopefully work can begin towards a happier database agnostic future. And even more hopefully, the developers of the WordPress parent project would embrace those sorts of changes. Right now I get the impression there is ambivalence about the kinds of changes that would be required. It may turn out that, like so many technical hurdles, the hardest work required is in soothing the feathers of the people involved.

Preparing to Bicycle to Work

With some motivation from listening to a coworker in Las Vegas, I’ve decided to start bicycling the last few miles to work instead of catching a bus or the monorail. Apparently some parts of the government are getting worked up about the general lack of exercise people get nowadays. The exercise will supposedly help with midday fatigue, as well as the obvious weight benefits from burning the calories.

Taking your bicycle into the city is a different sort of adventure than my usual mountain biking back in the sticks or on country roads within reach of our home. Getting the bicycle into the city has to be dealt with, that old U-lock just can’t be trusted to keep a bike in one place in the city nowadays, and having knobby tires isn’t a lot of use on city streets. So here’s a quick run through things I’ve had to find to make this all happen…

Roof Rack

A trip to the Seattle REI took care of the bicycle rack part. My old Cannondale mountain bike frame has a pretty hefty center tube that normal wheels-on claw type bike racks just can’t seem to cope with. I can recall more than one occasion driving down the freeway with the bike up top and hearing a loud clunk as the old claw rack lost its grip and the bike fell sideways hanging over the side of the car. That didn’t exactly do wonders for the wheels it was hanging from at that point either. Wheels-off racks won’t work either, the hydraulic caliper brakes make wheel changes a real fiasco. They do however make braking on muddy, wet, or otherwise slippery rims real easy.

Street Tires

I have a pair of slick street tires that I had purchased for the purpose of riding around town when I was in college. There are some bicycle messengers on the web that recommend mountain bikes with click tires to people getting started doing that work. So hopefully I’ll be set along those lines.

Bike Lock

The trip to REI also included getting a more serious way to attach the bike to fixed objects so it doesn’t just wander away when I’m not around to pay attention. Luckily Seattle isn’t quite New York in terms of bicycle theft, but I’m still not going to trust the old Kryptonite U-lock I have which is vulnerable to the cold spray trick, the Bic pen trick, the crowbar trick, and heaven only knows what else. To solve this problem, I’ve got a new Kryptonite chain which is made with squarish links that apparently are not friendly to bolt-cutters and a padlock with a guard around the shackle to keep it from getting cut. They have some product information about it online.

Helmet

I already had this, but it probably bears mentioning. Bicycling around downtown Seattle without a helmet would be insane, not to mention illegal. I can’t count the number of times I’ve had some idiot in a car or truck nearly run me down when I was walking across a crosswalk or even just walking on the sidewalk. Being attentive while driving apparently isn’t a requirement nowadays, so it’s best to have some protection for my valuable intellectual property.

We’ll see how this adventure goes and I’ll probably say more about it at some point in the future.

The Return of Progress Report on WordPress-pg 1.5

I spent some quality time in a local coffee shop this evening pounding out more work on the port. Without further rambling here is the good and bad side of where we’re at now:

• Installs from scratch works in my test environment
  
  
  
  • The logo image on the install page depends on a remote image, probably should get fixed to be local
  
  
  • The table creation statements in step 2 interleave wrong in my browser
  
  
  • There are some errors thrown by the index manipulation code in step 2
  
  
  • Haven’t tested an upgrade scenario of any sort (this probably won’t work)
  
  
  • Haven’t tested an import scenario of any sort (this certainly won’t work)
  
  
  
  


• Creating new posts works


• Creating new comments works
  
  
  
  • Comment status code is probably broken (badness related to comment_approved growing a new “SPAM” status which I’ve wound up implementing using NULL since this was made a boolean in the 1.2.x ports)
  
  
  
  


• Creating new categories works


• Creating new links works


• The front page as well as individual post pages display correctly


• The admin pages display mostly correct
  
  
  
  • There is some weirdness with the categories list in Manage Categories where root items are displayed twice except for the id 0 “Uncategorized” entries.
  
  
  
  


• I’m seeing some database error messages on creating a post but can’t determine where they are coming from due to wp-db being obtuse and various bits of WordPress code turning error reporting off for no good reason


• Haven’t even begun to look at xmlrpc.php yet. Making that go seems like it is going to be a bit of an adventure

• Got the posts and categories displaying correctly on the end user parts


• Fixed places where LIMIT M,N needed to become LIMIT N OFFSET M for PostgreSQL


• Fixed more badness where “SHOW TABLE STATUS” was used to get the auto_increment counter for new categories, which resulted in new category creation working properly.


• Default post category is now the special “Uncategorized” category. Some weird stuff was happening where the post_category would be 0 but wp_post2cat would have 1 in it for some reason

• Cleanup database error messages to be able to know when things go wrong without tailing PostgreSQL’s logs


• Make install.php use local images only


• Fix install.php step 2 to use reindex rather than dropping and recreating indexes (which may be necessary for MySQL but not us)


• Figure out and fix what is throwing the SQL syntax errors in post.php on “save as draft”


• Fix the Category List order strangeness in wp-admin/categories.php


• Change the default links in the install to not include every person who ever wrote a line of WordPress code and instead point to the project pages and useful resources like the WordPress Codex.


• Change the default hello world comment and text to have some “Lorem Ipsum” copy in them to fill out space better.

So I’m going to be checking this all into CVS in a few minutes. I also took the time to add my thoughts about database abstraction approaches on the Using Alternative Databases page over on the WordPress Codex wiki. The WordPress-specific abstraction layer is going to work out best in the long haul. Highly generic abstraction layers for database access generally do poorly from optimization, performance and ease of coding points of view. And as the time spent on the 1.5 port demonstrates, the existing database code is a total fiasco for trying to add new databases to.

Brave New VoIP World

We’ve made the plunge into the VoIP (Voice over IP: transmitting telephone calls over the Internet) world. After a few months of paying $20 a month on long distance I figured there must be better way. Our local phone company’s best plan without a monthly fee is 18¢ per minute. AT&T has a no monthly fee plan for 10¢ per minute, except that you pay a much higher rate for in-state long distance. MCI doesn’t even offer a no monthly fee plan any more.

Enter Internet long distance. A variety of VoIP companies offer long distance to the anywhere in the US, Canada, and most of western Europe for 2¢ per minute or less. No monthly fee. The only catch is that you have to get your call onto the internet. There are services that bundle a whole telephone line over the internet with long distance, a new phone number, and an interface box. These services also generally cost at least $20 per month. Worse yet, being a DSL customer out in the sticks, we can’t get DSL without a plain old telephone line. No use paying for two phone lines.

But you aren’t stuck with a package deal, you can always get your own interface box and buy your own long distance and still use the existing phone line you already have. In our case we got a Sipura SPA-3000 from AsteriskMall and setup Asterisk on a spare computer. With some amount of configuration work, we’ve got outgoing local calls still traversing the POTS line, outgoing long distance going over the Internet to our VoIP long distance provider (saving us 8-16¢ per minute), and incoming calls coming in over the POTS line. There are also some special numbers we can use to make free VoIP calls to other folks on the Internet like IAXtel’s 1-700 numbers and Free World Dialup. Free World Dialup in particular has free calling to anyone with the most popular bundle providers (Packet8, Vonage, etc.).

In the long run, our investment in the interface box will pay for itself in long distance savings in half a year. And since we still have our normal phone line which the interface box will fall back to automatically when there is a power outage we still get the E911 service, even when the power and/or Internet connection are not working.

This isn’t to say that this hasn’t been a bit of an adventure. The SPA-3000 has over one hundred configurable parameters and getting it configured correctly involved a lot of trial and error. (By the way, thanks to our parents who have been very kind about our phone calls as we try and get things working. “Can you hear me now?”) Voice quality has been fairly good though. The delay hasn’t been quite as bad as I had expected. It’s slightly better delay-wise than satellite calls across the Atlantic, but this depends on how Internet traffic is between our home and

• Calls going out the POTS line are too quiet for both parties
  
  
  
  • Solved by increasing the PSTN Line “SPA to PSTN Gain” and “PSTN to SPA Gain” settings to 5 (dB presumably)
  
  
  
  


• Calls going out the VoIP long distance have an echo for the people we call
  
  
  
  • Possibly solved by setting SIP “RTP Packet Size” to 0.010 (seconds)
  
  
  
  


• Calls either way on the POTS line can’t make DTMF tones (touchtones) in call
  
  
  
  • No resolution yet
  
  
  
  


• Calls either way on the POTS line rarely have an echo back to our phone when both sides are talking
  
  
  
  • No resolution yet
  
  

There is still a long ways we can go with this too. Now that we have Caller ID incoming calls can get special treatment depending on who they are and we can get some more powerful voice mail service as well.

Your call is important to us. Press 1 to learn about why paying for long distance service from the old communication monopolies is going to be obsolete in a dozen years…

The Bride of Progress Report on WordPress-pg 1.5

More PHP and SQL…

$id_result = $wpdb->get_row("SHOW TABLE STATUS LIKE '$wpdb->posts'");
$post_ID = $id_result->Auto_increment;

…then after 20 more lines…

$postquery ="INSERT INTO $wpdb->posts
(ID, …) VALUES ('$post_ID', …)";

This is a race condition. While being formally defined by FOLDOC as “anomalous behavior due to unexpected critical dependence on the relative timing of events”, in this case it can be more easily thought of as a race between two individuals trying to do the same thing. A pair of people racing to make it to a revolving door when the door will only fit one of them. A group of cars racing to make the ferry when only one more car will fit on the boat. That sort of idea…

This code replaced old code that was doing something similarly vulnerable to a race condition. Except that the old code also protected against incidents where the auto_increment counter is out of touch with reality somehow. (More weirdness with auto_increment is documented at Ian Barwick’s MySQL Gotchas page.) In both the old and new code, we’re determining what the new post id is going to be, then going off to do other things, then we’re inserting the post with that new post id. That is unless someone else who got that new post id the same way we did, did the other things, and then did the insert before we got our chance, in which case there is an error and the post creation fails (with an error that the average user is not going to understand since there is nothing looking for this sort of failure). There can be only one (successful post with a given id).

Frankly, I don’t actually know if there is a way to handle this safely in MySQL without either recognizing the error that comes back and trying again with a different ID that (hopefully) will work or just locking the table through the whole process. Both of which are far from elegant. Luckily, the frequent posting that would trigger this bug isn’t likely with your average blog and the commenting scheme uses different code which doesn’t need to obtain the comment id ahead of time.

In PostgreSQL, there are utility functions for dealing with the sequences used for auto-incrementing that let you get the next value safe from possible collisions or races. The downside to this approach is that if your transaction fails for some other reason the value you received will never be used by anyone. However since it is an internal id number that is never actually visible to the end users in most cases, it doesn’t really matter if there are gaps in the ids. There will be gaps in the ids if you delete posts anyway.

All we have to do for the PostgreSQL port is run this query instead:

SELECT next_val('${wpdb->posts}_id_seq')

With this we wind up both safe from the race condition for obtaining the next post id and with a very elegant looking solution to the potential problem. I’m sure those guys in Sweden will get around to implementing a feature akin to this soon. Maybe it’s there and I just haven’t heard of it yet.

Some editorializing follows, please cover your eyes if discussion about censorship by corporations bothers you…

Since this writing could be taken by some as “disparaging” the MySQL product, I’m apparently not authorized to use any MySQL AB’s trademarks in this blog entry, including the trade name “MySQL”. Don’t believe me? Read their rather draconian trademark policy for yourself. Good thing this country still has fair use laws, so I don’t need authorization from the manufacturer to write critical evaluations of things.