Technical Details

When it comes to the merits of many open source software systems I’m not the most generous guy (in terms of compliments), but in general open is always better then closed!

I had to release a new version of rTunes this weekend because of a wincom id change with iTunes 7.4.2.4. There was this nagging issue with rTunes and really slow web serving when I was using a remote computer, this was really irking me and since I was updating the source I thought I’d track this down. So off I went and started dropping in some profiling code, after the first test I had a pretty good idea of what was going on (and also why no one had complained about this issue yet.) Python’s BaseHTTPServer generates a log entry when you send a response, the problem as it turns out, the log entry wants to print the host name which means it does a DNS “fully qualified name lookup” and there’s the rub; I’m running DNS at home (and yes it’s not properly setup where Windows is concerned but it works for my purposes) but these name lookups were taking 4 secs on my network which is ridiculous performance, so it was simple to override the address_string to simply return the IP. If you’ve ever setup Apache you know they strongly recommend turning off name lookups, no kidding, so why this library has this as a default is kind of weird and a little bit dumb. After changing this behaviour performance improved by two orders of magnitude, woohoo.

The whole point of this story is that given the documentation of the BaseHTTPServer library and most other libraries of all sorts (free and commercial), without the source I would have spent significantly more time tracking down this issue with a closed source opaque library.

Vive la open source!

I’m happy to get this out the door along with a major fix for DNS Fully Qualified Name look ups, “use the IP Luke”.

The remote versions for windows will follow shortly.

Enjoy, and let me know if there are any new features you’d like to see.

Get rTunes here:  http://agwego.com/rtunes

I just got the first report in that iTunes 7.4.2.4 breaks rTunes, here we go again Apple, can’t you maintain a stable interface for crying out loud. The good news, is I have no problems connecting with the raw source, it’s only the compiled version that is having difficulties right now. If you’d like the instructions on installing python and the necessary packages to get rTunes working leave a comment and I’ll post them.

Update:

The release of 7.4.2.4 changed the com signature which means the hard-coded com signature in the binary package is out of sync, unfortunately there doesn’t  seem to be a better way at this time when it comes to py2exe.

I should have a new version out this evening, I just need to package and test, and I also found a very annoying “bug/design feature” in the http lib which was making rTunes very slow when DNS is being used on a local network, like my home network. 

• .zip – you’re very popular but a little shallow, you get invited to most of the parties but you’re not considered part of the really really cool crowd. You’re also accident prone.
• .Z – you’re a dinosaur, get out of your mother’s basement and mingle, explore the world and upgrade your life.
• .zoo,.lzh – you secretly are still a kid, time to put down the Amiga/Atari ST and move on, I’m sure your significant other would be much happier if you let go of the past
• .gzip – You’re the cool kid, you can get into most parties and no one will sneer at your decision.
• .rar – You are a rebel, girls look at you like you’re pirate and you don’t mind sharing the fact, you like porn.
• .bz2 – You are very serious, detail oriented and concerned with optimizing anything no matter how small the return, you spend much of your time alone
• .hqx – You’re an iconoclast, you love swimming against the current, you want everyone to know it but you’re not willing to share.
• .uue – You’re a lowest common denominator kind of person, KISS – keep it simple stupid, and you like porn.
• .sit – You’re an iconoclast with money and you like to flaunt it, the only thing is, you don’ really know what’s going on in the world.

This is a pretty handy article on automating your subversion backups good stuff to know and implement, seen via dzone.com

I’m porting the Blogmatrix platform to Solaris and of course all kinds of little things are popping up, like expat’s install is broken for include files, and I’m assuming it’s a Murphey’s law sorta thing that it hasn’t been discovered before, at least I haven’t found the problem Googling for it.

Here’s the gist, the installer drops a number of files into assorted places, for instance a library, a man page and two header files. Looking at the Makefile.in we have the following, in particular the last line.

APIHEADER = $(srcdir)/lib/expat.h $(srcdir)/lib/expat_external.h

installlib: $(LIBRARY) $(APIHEADER)
        $(mkinstalldirs) $(libdir) $(includedir)
        $(LIBTOOL) --mode=install $(INSTALL) $(LIBRARY) $(libdir)/$(LIBRARY)
        $(INSTALL_DATA) $(APIHEADER) $(includedir)

The last line translates to the following on Solaris:

conftools/install-sh -c -m 644 ./lib/expat.h ./lib/expat_external.h  /your/install/dir/include

Which seems reasonable enough, except for the fact that install-sh ignores anything between the first file and the last directory directive on the command line, nice. On other platforms configure uses a different install script provided by the system instead of install-sh included in the package, bummer, it would make sense to fix install-sh but it won’t be pretty. So I’ve patched the Makefile instead, ugly but much smaller and it should work every where.

80c80
< installlib: $(LIBRARY) $(APIHEADER)
---
> installlib: $(LIBRARY) installinc
83c83
<       $(INSTALL_DATA) $(APIHEADER) $(includedir)
---
>       #$(INSTALL_DATA) $(APIHEADER) $(includedir)
84a85,89
> installinc: $(APIHEADER)
>       for exHeader in $(APIHEADER); do\
>               $(INSTALL_DATA) $${exHeader} $(includedir);\
>       done
>

The first rule is to never assume anything is where you think it is, /usr/bin/env is your friend, especially in combination with #!

Use commands in preference to environment variables, although it’s nice that environment variables like $PWD and $USER maybe set they aren’t always what you are looking for. For instance on FreeBSD if you are in your home directory, $PWD will probably read /home/user_name, but if you want the absolute path you should use $(pwd -P) or $(/bin/pwd)  (shell built-in vs command) which will yield /usr/home/user_name, two very different results. On Solaris $USER isn’t set, you need to use $LOGNAME instead or better $(/usr/ucb/whoami)

I’m not sure why the command line double-dash option, for example”–something” came into being, but it is not very portable when it comes to basic UNIX commands like “ln”, “tar”, or anything else for that matter, always use the short form for your scripts,  so “ln -s -f” instead of “ln -s –force”

Use indirection, create your own set of environment variables/functions that have the correct platform dependent references set upon initialization and then you can change your platform without modifying your main scripts. Your script may be a little uglier but it will run much better.

Pipelining is still your friend, many commands like tar have shortcuts to avoid piplining for instance “tar zxf” to automatically decompress the file before untaring it, nice and easy to write but not very portable. Instead use the long form “gzcat filename | tar xf – “, also you will probably need to “cd dirname” to your directory instead of using “-C  dirname” which isn’t supported on all platforms.

For cross platform testing (if you care about such things or need to worry about it) get yourself a copy of VMWare and set up your target test environments and start testing as destructively as you like, this beats dual booting or running multiple machines, you can have one virtual QA environment setup, VMWare Server is particularly good for this.

http://anthology.spacemonkeys.ca/archives/821-64-bits….html 

Are you prone to fits of rage, do you get easily frustrated? If so, avoid OpenBSD. Just when I thought an operating system couldn’t get any worse than FreeBSD, OpenBSD lowered the bar, yah!

The installer is pretty lame, instead of having a directory with a bunch of files and other crap in it, including very poorly written documentation, what would be nice is the entire release in one CD image, not spread all over hell’s half acre. Of course there is no BASH shell included, oh there is a binary package but there are a number of shared libraries required, and the package doesn’t really indicate any dependencies that bash has, so off to play detective, folks I have better things to do with my time than figure out what packages contain what shared libraries that are needed. Just to add insult to injury the binary package is linked with a very old version of libintl, thanks for coming out, this is easy enough to cheat.

For completeness sake, your need the following two packages to complete the bash install libiconv and gettext which contains libintl, install in their respective usual places (/usr/local) and add a link for libintl.so.3.0 -> ./libintl.so.8.1 or whatever version is in the current gettext package.

For the most part OpenBSD is a lot like FreeBSD but there are lots of things OpenBSD doesn’t support like bzip2 natively, this also means tar doesn’t support bzip either, all of which is very frustrating.

There is no MySQL Binary install for OpenBSD, which isn’t their fault, but it is an indicator of how important an OS OpenBSD is, not very.

ldconfig doesn’t work as expected as on other OS’s for non-root users, bummer.

There will probably be other updates to come, but I’m getting ready to abandon OpenBSD.

I kept with it, but I’m ready to take my LCD display and keyboard and throw them out the window I’m so frustrated with the incompetence that is OpenBSD. Apparently, IPv6 is installed by default and if you read the OpenBSD mailing lists there is no possible reason you would ever want to disable v6, since it is the “future.” Well OpenBSD’tards with Apache installed it wants to bind only to the IPv6 interface and since I don’t have any other IPv6 machines I cannot connect to Apache using our default configuration, I have to explicitly bind it to the IPv4 interface which means someone has to manually do this, what a joke. If I could disable IPv6 I could have figured out this issue much sooner, I don’t know if this is an Apache problem or strictly and OpenBSD but I’ve never run into this on any other OS.

I’ve been pretty critical of VMWare, and I’ve been using Virtual PC since before Microsoft got their grubby little hands on it. Virtual PC in my opinion is generally easier to use and worked very well up until recently and for the most part supported Linux very well.

This weekend I tried to install Fedora Core 7 on Virtual PC 2004 (on my laptop) and of course it didn’t work, so I figured, hey I’ll upgrade to VPC 2007. After upgrading to 2007 the Virtual PC networking no longer worked so VPC was no longer of any use, after trying to fix the problem and jumping through all the stupid MS’isms I gave up and thought I’d downgrade to 2004, of course the network was totally screwed for 2004 as well. During this I also tried to install Fedora Core 5 on 2007 which I have working on another VPC instance on another machine, of course it doesn’t work anymore. Now if I was a conspiracy nut I’d be thinking MS has purposely tweaked VPC to prevent Linux from running, the jury is still out on this one.

So VMWare to the rescue, having finally setup VMWare server for Linux and getting all manner of OS’s running and being quite happy, I switched to VMWare and it worked perfectly, bye bye VPC.

So MS you may be giving VPC away, but if it can’t do the job what’s the point, I’d rather pay VMWare for a working product any day, MS’s loss is VMWare’s gain.