Nathan's Weblog

Fame and Fortune!

Ah, my first press release. This is the most attention anything I've done has ever received. Part of me wishes it was for something less mundane than writing a driver for some off-the-shelf consumer hardware, but I guess I can't complain. It's not often that Slashdot posts about driver releases.

What has been interesting is the response from the community that the press exposure has generated. Criticism that I was expecting—lack of kernel 2.4 support, for example—hasn't come up at all, and the response on Slashdot was decidedly positive which is almost unheard-of. The only (reasonable) complaint so far, besides the missing features like VBI and PAL which I'm planning to remedy anyway, is the price of the Plextor devices compared to similar products. The "Plextor quality costs extra" argument only goes so far, especially considering that Lifeview and Georgia Tech are selling substantially identical GO7007-based units for 25-50% less.

Plextor has been kind enough to donate a TV402U to Isaac Richards, author of MythTV, so maybe soon Myth will support these guys. Ultimately, I'd like to see ODMs develop self-contained PVRs running MythTV with WIS hardware on-board, but we'll have to see. For now, I'll be content to just get some community support around the GO7007.

Lots of webcams

Purdue is apparently well-covered. Really.

Some ideas on Spook

I've just released the first new version of Spook in almost six months. Embarrassing that I've let things go for so long, but at least all the features that have been accumulating in private trees since then have mostly been added. The network-side code has been overhauled which not only made things more efficient but paved the way for multiple media streams in the same RTSP session. Audio. w00t.

Since I started adding the audio support, the limitations and flaws in the current stream implementation have been bothering me. It kind of made sense at first to declare "the output from this grabber is named 'foo', and the input to this encoder is named 'foo' and the output will be named 'bar'," etc, but it's not only confusing but it prevents me from implementing a number of more useful features. It would be nice, for example, to have Spook automatically open all the video and audio capture devices it could get its hands on, compress them to some default reasonable format, and put up a template webpage with a list of everything it found so that the user could test out Spook without configuring anything at all. Not possible if all capture and transformation actions have to be declared explicitly.

There's two major changes I'd like to make to the way Spook creates media streams. The first is to change the media compression and format specification from an imperative form to a declarative form. There's no good reason (other than Unix tradition, I suppose) to force the user to explicitly list the various filters and encoders and the order they should be used, when all that's really necessary is for the user to specify the desired end result. "I want a 320x240 384kbps MPEG4 stream" is enough information to set up the entire module pipeline automatically. The main problem that arises is getting all the nuances of each type of stream correct—providing two different streams, one of 30 fps and one of 10fps, can be done by dropping frames after compression with JPEG, but with MPEG4 the frames must be dropped before they are encoded.

The other change is to convert the stream namespace into a hierarchy instead of using free-form strings. This sounds minor, but it allows modules to create new streams on the fly without causing confusion. Through the magic of sysfs, the V4L module can automatically discover all your USB webcams, configure streams for them, and create meaningful, static names for them. The two cams plugged into the hub on the lower USB port on the front of your system will be always be named something like "Device::Video::USB::2-1:1.0" and "Device::Video::USB::2-2:1.0" no matter when they were connected, because their name is based on their position on the bus. The same devices can be accessible with their device path, like "Device::Video::/dev/video1" or whatever, if you prefer that.

The real advantage of putting the streams into a tree is being able to perform transformations over an entire subtree rather than on one at a time. This would allow one part of the tree to contain a "mirror" the same streams as another part, but having passed through one or more modules. For example, the "Low Quality::" tree could contain the above streams as "Low Quality::Video::USB::2-1:1.0", etc, compressed as 100 kbps 10 fps video, rather than the native 30 fps uncompressed YUV format received from the hardware. These mappings from an existing subtree to a new one would be specified in the configuration file, of course, so the user could control which subtrees were available and the exact parameters used.

At the end of the whole chain, there still needs to be some sort of connection to make the stream available to clients across the network. This could be done through explicit pairings ("rtsp://*/webcam" will serve video from "Low Quality::Video::USB::2-1:1.0") or by exporting entire subtrees ("rtsp://*/cams/" will serve anything below "Low Quality::Video", such as "rtsp://*/cams/USB::2-1:1.0") as the user sees fit. As I mentioned, template webpages could optionally enumerate the streams in exported subtrees to simplify the initial configuration.

By now, you're probably wondering why I'm putting so much effort into automated support for large numbers of streams. It's not like most people have so many webcams that they can't configure them manually, right? Well, the ultimate goal is to turn Spook into more of a generic network media mixer, capable of importing video from any video source local or remote, performing some set of transforms on it, and exporting it over the network using a variety of formats and protocols. Re-encode video from the DCS-900 watching your koi pond into MPEG4 and relay it through another, more well-connected server via RTSP? No problem. Webcast press conferences from the room's AV system while simultaneously recording them for later retrieval over streaming RTSP or AVI download over HTTP? Makes sense to me. "Tune in" to the physics class lecture you're skipping by dialing a number from your mobile phone? Why not?

Putting as many media sources as possible into a common namespace doesn't get much closer to convergence, but it's a necessary step.

They took AIM

AOL suspended my instant messenger account today for supposed violations of their terms of service. No idea if I'll get it back. If not, I'll probably go register for Yahoo's IM service, or something similar.

Spammers and Drivers

The spambots finally found my weblog, so the comment plugin is gone. Hope y'all don't miss it too much. Feel free to email me directly if you're unable to contain your rage over my scathing political commentary.

I guess I've been kind of anti-social for the past few months. I've been doing some work for Wis Technologies, who makes the world's most bad-ass real time video encoder ASIC, filling in some gaps in their reference software. They've been funding work on Spook, and although most of it hasn't yet gone back into the public tree, it's come a long ways. I also have some stream-relay code that I've yet to check in. Hm, Spook is looking kind of abandoned these days. I should poke it a bit more in the near future.

January has been consumed with a mad rewrite of the Linux driver for the WIS encoder. The old driver sucked mightily, so when Plextor wanted to their ConvertX product line to have Linux support (which didn't exist before only because it didn't occur to anybody), trashing the old code and starting from scratch was pretty much the only option. It came out pretty well though. It's in QA now and should be released via Plextor in a week or so.

Um...

So I was searching Google for "Gondwanaland", the huge southern supercontinent from long ago, and noticed that somebody had actually bothered to buy ads for that keyword. Maybe a museum or a science book publisher? Nooooooo....

Somebody needs to scrub their geographical location database a little better, methinks.

For the low, low price

Those of us on the X Prize mailing list have received an invitation to the X Prize award ceremony. Truly a historic event to be a part of. And what would you expect to pay for such a once-in-a-lifetime opportunity? Tickets range from $275-$750 per person, depending on how close you'd like to sit to the world's newest astronaut.

I guess that explains how they got the "backing" for the prize money. They'll tell you to just bring the cash to the event, which they'll put into a big box and present to Burt Rutan during the ceremony. Clever, eh?

How can people stand this??

For consulting I decided that I really should have a working Microsoft Office workstation, so I bought a Dell Dimension 3000 with WinXP Pro and Office SBE for just under $1000. (At retail, XP Pro and SBE would cost $710, so I figured I'd spend the extra $250 and get a rather nice PC to run it on.) I ordered seven days ago, Dell shipped it the next day, and I received it today on the third delivery attempt.

Having not owned a Windows system for almost four years, I was rather surprised at the amount of hassle it takes to get an out-of-the-box PC up and running. I can understand having to update the virus database, install a couple patches to Windows, etc (although the drive was imaged at the factory a week ago) and I can even tolerate "Windows Product Activation", but just to get the preinstalled copies of XP, Office, and McAfee running, I was not expecting:

Clicking "I Agree" on eight different EULAs, not counting Dell's infamous "Press any key to indicate acceptance of all EULAs for all software preinstalled on this system" screen that pops up two seconds after the first boot.
Seven required reboots, two of which occurred after nothing obvious had happened.
A 23-meg "critical update" for XP.
Harassment from Windows Security Manager (whatever that is) every time I boot because I have set Windows Update to only notify me when updates are available, but not to download and install them automatically.
Harassment from "Dell Support", telling me it wants to upload "anonymous usage statistics". No, there's no way to turn it off. The only relevant option lets you toggle between asking before sending statistics or simply sending statistics without notifying you at all.
Having to open the help browser to figure out how to update my McAfee virus definitions, even though the McAfee control panel popped up on every boot to alert me that my definitions were out-of-date.
Language settings in all my Office applications default to "Chinese (Taiwan)". (I'm pretty sure Dell ships legal software...)
And, of course, having fifty different preinstalled apps that I didn't ask for, telling me to sign up for music downloads, dialup Internet access, etc.

Honestly, how do people put up with this nonsense? Suffering through all this crap makes setting up Gentoo look easy!

I thought the attraction was the coffee

So, every time you see a blog post or magazine review for some new tech device with onboard 802.11, the inevitable suggestion is that the usefulness of the device is substantially increased because now it can get online from the nearest coffee shop. "With onboard Wifi, the newest iPods can download tunes from iTunes Store while you're at the coffee shop!!" Huh? How is this a selling point? I'd much rather be able to download new music while I'm laying on the couch feeling too lazy to pick my laptop up off the endtable, which is a much more likely place to find Wifi anyway.

Does the coffee shop aspect of 802.11 connectivity really make that much of a difference to people? How much time do ordinary iPod-toting folks spend sipping lattes and wishing they could download the 839th Dave Matthews concert CD without having to leave the lively, stimulating environment of their local Starbucks?

Sheesh. Clearly these people need help to overcome their unhealthy addition to Wifi access and caffeinated beverages. I recommend a Bluetooth-enabled 3G cell phone and some Penguin mints.

When upgrading Gnome...

OK, so I figured if I shut down my X session entirely, I could let emerge upgrade Gnome to 2.6 and start up a new session afterwards and everything would work fine. I even checked to make sure gconfd and friends were gone, as they have a habit of sticking around after a session ends.

Turns out that's not good enough. Apparently, during the upgrade process, some test suite started up the Bonobo activation server, and when I started my first Gnome 2.6 session there was much weird behavior, like gnome-settings-daemon failing to start.

Probably the best way to make sure everything's reset is to reboot the system entirely, but damn, that feels waaaaay too un-Unixy.

In related news, Gnome 2.6 is pretty much identical to 2.4, although that annoying dialog box in Epiphany asking whether to open or save the playlists from Shoutcast is fixed.

Unpleasant discovery #329

The outlets in our living room and dining room are controlled via a light switch. All of them. In both rooms. From two switches. And I don't mean that the two switches control two separate sets of outlets—it's a 3-way configuration so all of the outlets can be switched together from either the living room or the front door.

I guess I know what tomorrow's project will be...

Like a hot knife through butter

I just finished pulling network cable through the house from the bedroom, office and living room into the basement where the server farm will live. With any luck, all the runs will be good enough for gigabit.

I also had to install a new phone jack in the basement, because, oddly enough, nobody thought to put a phone down there before. Eventually, it will be the only jack hooked up to the Verizon loop once I get the VoIP service setup running, but for now I just twisted the new line onto the big wads of wires that are bypassing the nice 66 punch-down block next to the electrical panel. Everything was all nasty and corroded, so I figured I'd have to go back and fix it later.

After I finished installing cable, I plugged a phone into the new jack to see if I had wired it to the right pair. Even with a DSL filter, the dialtone sounded pretty noisy, but I tried plugging the DSL modem into it anyway. Synced right up. Huh. So I hooked up the server and booted. PPPoE came up. Pings worked; even flood pings only lost a few packets. Download speeds were around 180 KB/sec, right where they should be. Who'd have thought that a line too sketchy for analog voice would work fine with 1.5 Mbit DSL?

Spelunking

Webcams are surprisingly useful for discovering where heating ducts go. They are small and can be dangled by the cord for a good 15 feet down a vertical duct or look around corners under the floor. I posted a typical picture looking at the hidden side of a light fixture inside one of our returns.

Posted so I don't forget

Charter One Bank has pre-paid MasterCards available online. Lots of people do that, I know, but Charter One does not charge a service fee, and they don't even charge you for shipping & handling. You choose an amount from $20 to $500, select a message to be embossed into the card, and enter the recipient's name and address. The payment is funky though—your only option is to take the money off an existing credit card, which presumably will be charged as a cash advance. Maybe using a debit card will work?

Another day, another tarball

I've got my process down. Update the README, `make distcheck`, upload the tarball, update the webpage, submit to Freshmeat, update the weblog.

Tonight's release adds Video4Linux support, so now the other 95% of the Linux community can use it. V4L is the API used by frame grabber cards and USB webcams, so I guess I should support it. USB webcams are pretty pathetic quality in comparison to Firewire webcams, and are also annoying to support because every USB webcam driver has its own funky API to set framerate and what-not. But it's what most people seem to have, unfortunately. Just for fun, I threw in support for the Philips webcams, but since I don't have one myself I have no idea if it will work.

Maybe tomorrow somebody will actually subscribe to the mailing list, too.

Return from the land of the lost

Been a while since I posted here, eh? As it happens, I've been deeply engrossed in a little toy that grew into a real project.

I've always been disappointed in the lack of decent streaming webcam software. Everyone and their dog has written a "grab an image and FTP it somewhere" app, usually complete with a crappy, useless GUI. (All you people who remember Cyclops are nodding your heads and thinking, "Yeah Nathan, you're one to criticise crappy webcam servers.") Some people go so far as to write a custom Java/ActiveX applet to pull down images real fast for kind of a redneck "streaming" effect. (Carson calls this "faux streaming".) The closest I've seen to live video is the HTTP multipart/jpeg streaming implemented by camserv and the Axis network cameras. This works, to some extent, but there's just something repulsive about shovelling JPEGs down a TCP socket as fast as possible and calling it "live video".

Come on, people! For decades we've had video compression techniques that are specifically designed to push live video down small pipes. And protocols for streaming real-time multimedia over IP networks have been standardized for over 10 years. The folks lucky enough to have multicast IP access have been holding multi-user video conferences since before most of the world had even heard the word "Internet". So why on Earth should we put up with the equivalent of mailing Grandma a stack of Polariods and telling her to flip through them really fast?

Before you say anything, don't give me that "widest possible audience" crap. I know you want your webcam to be accessible to anybody with a web browser, but how many people do you know who don't have QuickTime installed? Seriously, everybody who didn't have QuickTime installed already went and installed it to watch the Lord of the Rings trailer. And if they didn't care about the LotR trailer, they don't deserve to watch your webcam anyway.

OK, so where was I? Right—my point was that we already have the technology, standards, infrastructure, and most importantly the viewing software in place to start feeding people quality live video from our homes and offices. All we need is the server software. So, properly motivated by such an obvious need, I wrote some.

Spook is my attempt at a decent live streaming video solution. It captures video from Firewire cameras, compresses it to MPEG4 with XviD, and streams it (via unicast, no multicast required) to anybody who asks for it using RTSP. The result: true live video at 30 frames per second with a resolution of 320x240. In half the upstream bandwidth of the average cable modem. Now that is what I'm talking about.

Today was the first release of Spook. It only supports Firewire cameras for the moment, so I'm sure I'll get a few more folks interested once I support Video4Linux and all the TV capture cards and USB webcams it brings. The problem with V4L is that configuration for each camera chipset is different, so it may be a bear to support configuration of FPS and all that. I'll find out soon, I guess.

Bittersweet victory

Sad is the world of Linux wireless Ethernet. Sure, the "traditional" chipsets like Orinoco/WaveLAN, Prism, and Aironet work as well as they always have, but newer hardware like the Atheros chipset, the new MPI-350 card from Cisco, and the infamous Centrino chipset are all only supported by broken/outdated closed-source drivers, or are not supported at all.

My new Thinkpad is "wireless-ready" meaning it has a built-in antenna and a mini-PCI slot for a wireless card. Nifty, right? The niftiness of having internal wireless hardware combined with the advantage of being able to select a NIC with good Linux drivers. But naturally, IBM doesn't make it this easy. For "regulatory" reasons, which I guess are understandable considering the anal retentiveness of the FCC, the BIOS is programmed to accept only one of three cards in the mini-PCI slot. You can guess which three cards they are based on the previous paragraph.

The Thinkpad shipped with a Centrino card, which wasn't looking promising as Intel has been promising Linux drivers for over a year now without delivering anything. The best option seemed to be the Aironet card, since older Aironet cards have been supported under Linux since the beginning of time. The "official" Cisco Linux driver for the MPI-350 was a closed source module last updated over a year ago, but kernel 2.6.3 was released with experimental support for the card in the airo module, so I figured my odds of getting it working were good.

I priced out the MPI-350 in the usual places, coming up with about $70, but then it occurred to me that the IBM OEM cards might have a different PCI ID and only the IBM-supplied MPI-350s would make it past the Nazi BIOS. So I bit the bullet and ordered an Aironet MPI-350 for $120 from IBM.

The day before my new card showed up, Intel announced the first release of their Linux Centrino wireless driver. Open-source and everything. I tried it out and it worked. But a bit buggy and no WEP, so I decided not to return the Aironet card.

So today I finally got around to installing the new card. Hardware installation was painless, of course. Then I booted up and found that the airo module had been loaded, but the card wasn't associating with the access point. A Google search revealed that my card's firmware was too new, and that I needed to downgrade to 5.00.03 to get it to work. No problem, I can download that from Cisco, and flash it into the card with... oops.

The open-source driver won't talk to the new firmware at all. The Cisco Linux driver also requires an older firmware, so there were zero options for flashing the card on Linux.

Two hours later, Windows XP was restored on the laptop. Three minutes after that, the card had been downgraded. An hour later, Linux was back.

One final issue. Although Linux could now talk to the card, it wouldn't associate with an access point. It seemed to be a problem with the WEP configuration due to messages like:

Setting key 0
airo: Max tries exceeded waiting for command
PC4500_writerid: Write rid Error 65535
PC4500_writerid: Cmd=0121
airo:  WEP_PERM set ffff

The solution was to download the Cisco Linux Aironet Configuration Utility (ACU), which actually works with the open-source kernel module as well as Cisco's kernel module, and use that to set the WEP key. Since the card stores the key in non-volatile memory, this only needs to be done once. Now when I boot the system, I use iwconfig eth1 enc on to enable WEP without trying to set a key, and this uses the stored key to associate with the access point. Success!

I believe at this point, my laptop is now fully functional. All I need to do now is figure out how to do some of the fancier power management to squeeze the maximum life possible out of my battery.

Figures

I finally got around to fixing my Perl script to forward email to my cell phone, which Verizon broke last month, and the first email that came through was a worm with a forged address from a whitelisted domain. That really says something about the Internet today.

Syndication formats

I guess now's as good a time as any to jump into the Atom vs. RSS debate. (I've been meaning to post this for a while...) For those of you who don't follow the exciting world of weblog syndication, the venerable RSS syndication format is under attack by a young upstart format named Atom. While the development of Atom was started only eight months ago, it has caught on quickly in the weblog world, and most weblog software now supports it as well as a number of prominent sites such as LiveJournal and Blogger.

Why a new syndication format? Mostly the issue seems to be over the control of the RSS spec. While RSS was originally developed by Netscape, it was abandoned as the company lost market traction, and Dave Winer took over maintenance of the specification. The problem is, Dave Winer doesn't seem to understand how standards are supposed to be maintained and developed, and many people feel he is exerting undue control over the future of RSS.

So, to take advantage of starting from scratch, the Atom standard goes beyond just defining a "syndication" format. A format for accessing archived entries was created, and there are even standardized mechanisms for posting new entries and updating old entries. Soon, the variety of weblog update clients that LiveJournal users have available will be able to update and maintain weblogs running on any server software using the Atom API.

The Atom spec is cleaner and better designed than RSS, and as a result can support a much wider range of uses. Wiki sites are already using Atom to publish entries and accept new submissions. Want to publish XML-formatted security updates with Atom? You can embed your XML directly into the feed without having to entity-escape it, allowing much simpler implementation of the parsers. Atom supports feeding information from machine-to-machine, instead of just the machine-to-person format that RSS imposes by its poor design.

Hopefully, more sites will discover nifty uses for Atom, and it will become the starting point for growing the semantic web beyond the current content syndication rut we're stuck in now. The sooner we discard RSS in favor of Atom, the sooner we can move on.

I'm glad I'm not the only one who hates MT

Some dude posted a tirade about the uselessness and banality of most weblogs, which obviously touched off some sentiments throughout the "blogosphere". (Yeah, it's from February 3; I'm not exactly Mr. Current Affairs, am I?) The article starts off attacking bloggers in general, then degrades to harsh criticism of Movable Type. I don't agree with his attacks on weblogging, which I believe rivals instant messaging as a revolutionary communications technology, but I'm more than happy to add my piece about Movable Type.

This sentence sums it up: "Movable Type is written by web designers, not programmers." We all know how well web designers can program. Most of them are even honest enough to admit that they are not capable of producing code any more than I am capable of producing a decent-looking webpage. Not the oblivious folks at Six Apart though. To make matters worse, they've taken it upon themselves to lead the creation of new weblog "standards", such as the disgustingly-broken TrackBack mechanism. I could live with ignoring their bad code, but creating bad standards that other people have to conform to is inexcusable.

Maybe one day some sort of standards group will tackle the problems confronting weblog interoperability. It has already started, in fact, with the intention of developing Atom through the IETF. Until then, we'll be forced to put up with "standards" defined by a single, incompetent vendor.