WEBlog -- Wouter's Eclectic Blog

Thu, 26 Aug 2010

include povanim.mk

I was finding myself to be running the same povray command lines over and over again. And then ffmpeg2theora. And occasionally forget to update one or two .png files in the mean time, because I was trying and trying and trying some stuff all the time, and forgot that one frame in the final render. That's silly; that's why we have make.

So I wrote me a makefile.

START := $(shell grep 'Initial_Frame' $(INIFILE) | cut -d'=' -f 2)
STOP := $(shell grep 'Final_Frame' $(INIFILE) | cut -d'=' -f 2)
NUMBERS := $(shell seq -w $(START) $(STOP) )
POVFILE := $(shell grep 'Input_File_Name' $(INIFILE) | cut -d'"' -f 2)
ifeq ($(OUTPUT),)
OUTPUT := $(basename $(shell grep 'Output_File_Name' $(INIFILE) | cut -d'"' -f 2))
endif
ifeq ($(OUTPUT),)
OUTPUT := $(basename $(INIFILE))_$(basename $(POVFILE))
endif
PNGFILES := $(foreach nr, $(NUMBERS), $(OUTPUT)$(nr).png)
LEN := $(shell echo "$(STOP)" | wc -m)

$(OUTPUT).ogv : PATTERN := $(OUTPUT)%$(LEN)d
$(OUTPUT).ogv : $(PNGFILES)
	ffmpeg2theora -f image2 -i '$(PATTERN)'.png -o $@

$(OUTPUT)%.png : ONAME := $(OUTPUT)
$(OUTPUT)%.png : INAME := $(INIFILE)
$(OUTPUT)%.png: $(POVFILE) $(INIFILE)
	povray $(INAME) -D +SF'$*' +EF'$*' +O$(ONAME)

ifeq ($(CLEANFILES),)
clean:
	$(RM) $(CLEANFILES)
endif

CLEANFILES := $(CLEANFILES) $(PNGFILES)

The first part, to just write the makefile so that it would work, was easy. The second bit, write it so that you can include it in the same Makefile several times and have it generate patterns for the files you need, without having them conflict with eachother, was... slightly harder. But not impossible, apparently, with GNU make.

Use like this:

INIFILE=foo.ini
include povanim.mk
INIFILE=bar.ini
OUTPUT=bar
include povanim.mk

with povanim.mk containing the above Makefile

Now just run 'make' or 'make bar.ogv', wait for the rendering and encoding to complete, and you have an animation film. You can even use make's '-j' option to speed up processing by using multiple processors. Which will no longer be useful when povray 3.7 comes out (since that uses multithreading to use all cores with a single povray instance, anyway), but it is for now.

/en/computer/play

PermaLink

Wed, 04 Aug 2010

Something's very wrong...

I maintain NBD

I have an irrational emotional dislike for python. (I know, I know).

I needed to build nbd in a clean environment. So I set up the environment, install openssh-client (to get the sources across) and then do apt-get build-dep nbd.

That pulls in python.

Something is wrong with this world.

/en/computer/debian

PermaLink

Tue, 27 Jul 2010

NBD: authentication and named exports

I just did some work on two NBD features: authentication and named exports.

The first was written by Brad Allen back in 2008. I'd been thinking about an authentication scheme back at DebConf 8, and discussed it on the nbd-general mailinglist. Brad picked it up and ran with it, but it never got finished. Basically, what I did today was forward-port the code so that it would compile (and work) with today's version of the NBD utilities.

I tested it and it works, but I'm not sure what to do with it. NBD is not encrypted; so there's no protection against TCP hijacking or some such. Oh well; I might still integrate it into nbd mainline.

The second feature is something way more interesting. It had been on my long-term TODO list for quite a while, but I never sat down to implement it. Until today, that is. This patch has been compile-tested, but I haven't functionally tested it yet; I'll do that tomorrow.

The main reason it took so long to implement, was that initial NBD handshake made that quite hard. Upon connecting, nbd-server as it currently exists will open the file or block device that it needs to export, and—before nbd-client has sent anything—sends information on that export (its size, whether or not it is read-only, etc) to the client. Unfortunately, this does mean that I cannot add new features to nbd-server while retaining full backwards compatibility if these features may involve changing what nbd-server sends during the initial handshake, as is the case here. This could be worked around by having the server first send information on a "default" export to the client, and then variate that if the client informs the server that it actually wants something else. But this would have been rather ugly, since it means that errors in open() or stat() calls for the initial "default" export would have to be non-fatal until the client actually tries to read or write to or from the device, at which point they would suddenly become fatal. This would open a can of worms near which I didn't want to tread.

The way I fixed it in the end, however, is fairly simple. If names are used to specify an export, nbd-server can live with just one port to export all block devices that it supports, rather than needing many. As such, I can "reserve" one port for the new-style named exports. On this port, nbd-server would expect that clients send a name, and would not function correctly with clients that don't understand names. At the same time, nbd-server could still export the same block devices on other ports as well, using the old-style protocol, for clients that do not support named devices.

This would of course require that both the client and the server know perfectly well what the port is, and that it does not change too easily. As such, I've just put in a request for an assigned port number over at IANA, so that I can be sure that the port will not change unexpectedly. This also means I won't make a public release with this change until the port number has been assigned, so that I don't need to support two 'default' nbd-server ports in the future (one "temporary" and one IANA-assigned port).

For now, however, I think I'll go grab some food.

/en/computer/nbd

PermaLink

Mon, 26 Jul 2010

Maintainer stupidity

As a buildd admin, you get to see the myriad of ways in which packages can fail to build. Sometimes this is due to interesting technical reasons about the architecture in question; sometimes... not so much.

cc1: unknown option: -mmmx

For reference, I maintain powerpc and m68k buildd hosts.

configure: error: pkg-config: command not found

Checking build-dependencies is hard. Right?

ccache: failed to create /home/buildd/.ccache (No such file or directory)

No, I don't manually install ccache inside the buildd chroot, which must mean some build-dependency pulled it in. Why one would want to do that, I dunno—it's not like a build that takes longer would produce a different package, so it clearly is not required for the build.

(BTW, the invalidity of $HOME is on purpose -- packages are not allowed to write outside their build directory during builds, which includes home directories)

... and more. Sometimes, I wonder how people get the right to upload. But, well.

/en/computer/debian/buildd

PermaLink

Mon, 19 Jul 2010

Stream-of-image video data and cinelerra

Just so I won't forget, but also to make sure others with similar problems to me don't need to search for several hours before finding it's so simple that it's silly:

If you have a bunch of image files (PNG, JPG, whatnot) that you want to treat as a single clip in cinelerra rather than hundreds (if not thousands) of individual files, the trick is to use a "list file": you create a file with some metadata and a list of all the files that make up the video clip, and import that, rather than the individual files.

If said individual files are PNG files with alpha information, doing so also makes it ridiculously easy to create an animation that only covers part of the screen.

The file format is pretty basic; but over at this page on the cinelerra.org website they list two scripts (one python with GUI, one bash and command-line only) that can generate such a file, given a bunch of image files. That makes it even easier.

I'd also like to note, for the record, that cinelerra outperforms kdenlive in everything but 'transcoding flexibility'. Kdenlive also has support for image lists (and you can create them in the gui rather than having to muck about with external programs or editors), but it has the rather nasty misfeature of doing the equivalent of "Oh, look, what's that? We don't need that, let's throw it away" whenever it encounters alpha information in PNG files.

For full reproducability, it's also possible to generate PNG files with alpha information from povray if you use the +UA command-line parameter, and say something like

sky_sphere {
	pigment {
		color rgbt <0, 0, 0, 1>
	}
}

somewhere in your scene file. Yes, I know, povray is non-free, but it works, and—as opposed to blender—I do know how to use it. Hey, I'm not RMS.

/en/computer/play

PermaLink

Monty at it again...

Apparently Michael Widenius isn't quite about to give up on his quest against Oracle.

The latest news is that he's appealing the EC decision. Wonder what the use of that is, since the merger seems pretty final already. I also don't think it's a very good cause, at any rate. After all, the worst that can happen is that MySQL will fork. Or that Oracle will pour some resources in it, and actually make it a database, rather than the POS toy it currently is. Or that it dies out, and that the world switches over to PostgreSQL or Firebird for Free Software database requirements. Wouldn't be a big loss, would it?

Give up already, Monty!

/en/computer/legal

PermaLink

Wed, 07 Jul 2010

Pauly site: redux

A few years ago, when I was still a member of the Jozef Pauly ensemble from Ekeren, at some point I took over maintaining their website. One of the things I did was converting it to drupal, mainly because that would allow other members of the ensemble to help maintaining it.

Of course, such things do not happen automatically; and had I spent some more time on the project, it might've actually worked. But I didn't, so it didn't. Oh well.

One of the main issues with the site as I set it up, was that I hadn't really done a very good job integrating everything. The standard drupal 5 theme looked pretty good, I thought. Slap on a few modules, and done. Unfortunately, I skipped a bit of testing, and as a result, the site didn't really function the way it should've. When this became apparent, I didn't have the time to properly fix it.

During the last few years that I was a member of the ensemble, someone else stepped up to help maintain it. I told him that one of the things that I wanted to do was to upgrade the site to Drupal 6, so as to make i18n a bit easier. But some of the modules which weren't in drupal core made this somewhat harder than needed, and I didn't have the time to investigate in-depth what the issues were.

In the end, I set up a secondary site on a different vhost on the same server that ran drupal6, and gave that guy admin rights there to help convert things to the new site. I planned to lead the effort to make things work; but when I came back not so much later, it turned out that he'd done a very good job already, and that it was probably a good idea to let him finish.

This took much, much longer than expected; but eventually, he's finalized the setup, and it's now live (since a few weeks) on their official site.

This site has content in both Dutch and English, though there is slightly more in the Dutch version: since they're doing a concert trip again this year, there's a blog for their families. Also, some people individually have decided to blog about the whole thing, too, which is... interesting.

At any rate, I think the site looks very, very cool. Go check it out!

I guess we may have to start thinking about doing an upgrade to Drupal 7 some time soon. But, hey.

/en/computer/play

PermaLink

Thu, 24 Jun 2010

On LDAP objectClasses

Petter suggests to make the dhcpHost LDAP objectClass an auxiliary one, to make it possible to combine it with the dNSDomain objectClass in one and the same object.

Unfortunately, you can't do that. Well—not if you want Debian-edu to remain compatible with everything else under the sun. You see, one of the features of LDAP is that each and every objectClass is well-defined, and that you cannot change it. As such, once a tool knows about an objectClass, it can make assumptions about objects in that class, for any such object, in any LDAP-server anywhere. Change an objectClass, and some tools will break.

The proper fix is, first, to verify that you are using the proper objectClasses. To me, an objectClass called dhcpHost and one called dNSDomain don't sound like they were meant to be used together; I would suspect that there is another DNS-related objectClass that is meant for hosts rather than domains. If that isn't the case, then one can create a new objectclass—say, auxiliaryDhcpHost or some such—that is the same as dhcpHost in all but name, OID, and the fact that you make it an auxiliary rather than a structural objectClass. Because luckily, what's true for an objectClass is also true for an attributeType: if it is assigned to an LDAP object, tools may assume that their meaning is what they think it is...

/en/computer/cluebat

PermaLink

Fri, 04 Jun 2010

Re: beid

Sometimes help is just around the corner, and all you have to do is ask.

Not so long ago, I asked for help on the support software for the Belgian electronic ID card. I'd been packaging that since it came out back in 2004, but recently I've had some issues with getting the most recent version of this software to work properly. Since Debian 6.0 "squeeze" is fairly near, and since I don't want a horribly outdated version of beid to be part of that release, I realized that something needed to happen, and rather soon.

My request got quite some response; much more than I'd expected. There have been answers from people with various amounts of experience in Debian packaging; one of the people to reply to my request was Laurent Bigonville, who's also a Debian Developer. So we've now set up a mailinglist, a git repository, and work seems to be proceeding nicely.

I've also asked for the removal of the old belpic source package. However, due to a mistake on my end, the beid source package rather than the belpic one was removed, which means that it'll have to be re-added to the archive now. That should happen sometime during the next few days, after the worst problems have been fixed. However, since I now asked for belpic to be removed, too, this means that for the time being, there is no eID software in unstable. We'll fix that in due time; stay tuned.

/en/computer/debian

PermaLink

Sat, 22 May 2010

Help!

Yes, there is something like an 'RFH', or 'Request For Help', in the WNPP in Debian, but I thought I'd go through this blog, first, since I believe many more people who will feel inclined to help here will read this blog.

I've been maintaining the beid packages in Debian since, eh, quite a while now; and while it's not always been the package that I have spent the most time on, it's not a package that I want to give up.

However, in recent times, I've had issues with the code. I've not really been able to get the 3.5.2 release to compile properly as a package, which means that for the moment, squeeze is still stuck with a 2.x release. This is a problem, since that old release isn't supported anymore upstream, and what's worse, it does not support newer-issued eID cards. So if your card is less than about two or three years old, you can't even use it anymore. For this reason, the beid packages really, really should be updated before squeeze releases.

I've been trying to spend some time doing so, but real life kept intervening. And I haven't even been able to start looking for the 3.5.3 code that I'd heard rumours would exist.

So, this is a request for help. If you want a more recent beid package to appear in Debian (and, by extension, Ubuntu) any time soon, and you can spare some extra time on maintaining a somewhat convoluted package, you'd be an ideal candidate for comaintainer of this package. You don't need to have any experience working on Debian, but you do need experience with build systems and Makefiles. If you don't have any experience in working on Debian, but would like to learn, this might be an interesting opportunity. If you do have experience working on Debian and are as interested as me in getting this to be up-to-date in the next Debian release, your help would be very much appreciated. Even if you only have time for a little while and would like me to take over once everything is going fine again, that's helpful.

At any rate, if you want to help, contact me at wouter@debian.org. We'll work out the specifics then.

Thanks in advance,

/en/computer/debian

PermaLink

Tue, 18 May 2010

Can I have my Debian back?

So someone thought it might be a good idea to switch the default from the time-honoured Debian style ("booting foo... ok") to the RedHat style ("<left-align>booting foo... <right-align>[ok]"). I don't think the switch adds any value (it's not like it looked bad or anything in the past); in fact, I think it's a bad idea. And I even have a technical reason.

On my 26" 1920x1200 screen (when the i950 driver detects it properly and actually sets the framebuffer to that resolution), the output looks like this:

Now imagine you get to see that screen for about two to five seconds. Please tell me which two initscripts failed. And no, you cannot use a ruler. Bonus points for if you manage to do so while the text is also moving upwards, which tends to happen when systems boot.

Also note that because kernel mode-settings has become the default, a lot more people are going to be using framebuffers.

Am I the only one who thinks this is a bad idea?

/en/computer/cluebat

PermaLink

Fri, 23 Apr 2010

Touching up a video with Cinelerra

I found myself with a problem: trying to linearly edit a movie clip with a non-linear editor isn't very simple, nor is it meant to be.

Case at hand: I have about 1 hour and 40 minutes of film material that was recorded with dvswitch, and which is 90% fine; but given what we wish to do with it, we really wish to remove the errors from the recording—things like failed transitions, rogue camera movements, and so on. This is why we recorded the individual video streams on tape too, so that we could indeed do some retouching afterwards.

Cinelerra, however, is not really made for that. If you want to create a transition, you need to have two segments of clips that are stuck together on the timeline, and the transition is then applied to the border between the two. You can create multiple video tracks, but you cannot cut from one track to another; you really have to add a clip to the timeline. Doing that has obvious issues if you have multiple video streams that were recorded on the same time and that can be time-synced on multiple tracks. You would prefer to just select one track or the other on a certain point in time, but cinelerra just doesn't do that.

So the workflow that I found is the following:

Add all video streams to the time line, each on a separate track. Make sure the to-be-touched-up stream is on top; when multiple tracks are in use, and the are not faded to 50% or so, cinelerra will only show the top stream in the final rendition.

Synchronize the streams: first, create a rough alignment by moving them with the mouse. Then, press arrow-down until you can see the streams on a frame-by-frame basis. Now for each of your camera streams, do the following:

1. In the 'main' stream, search a section that was filmed by the camera which we are trying to align.
2. Hit 'l' to create a label
3. Search for the exact same frame in the camera stream. It may help to disable playback for the 'main' stream temporarily to visually verify that the frame is the same in the compositor window.
4. Move the cursor to that frame, and create another label.
5. If the streams are indeed desynced, then one of the tracks will lead, and the other will follow. If the leading track is the 'main' track, arm the main track and any tracks that you have already synced before, and make sure to remember the audio tracks. If, on the other hand, this is the camera track, then only arm that camera track. At any rate, make sure to disarm the track(s) that are following.
6. Use the control and arrow keys to select the first label. Use control, shift, and the arrow keys to select the section up to the second label.
7. At the bottom right of the program window, you can see three entry boxes with time indexes. The first is the time index to your first label; the third the time index to your second label; an in between those, the second is the length of the interval from the first label to the second label. Set the contents of the first box to all-zeroes, to move the selection to the beginning of the time line; then press tab (to move to the second entry box) and enter (to confirm that you wish to create a selection of that length at the beginning of the timeline).
8. Press Shift-Space to insert silence at the location of the selection. This will move the armed tracks to the right, and not touch the not armed tracks.
9. If you did everything right, the two tracks should now be aligned. Move the cursor to the second label, and verify that this is true by disabling 'play' for the main track and any tracks in between the main track and the track you're syncing. The frame in the compositor window should not change if you re-enable the main track. If it does, use 'z' for undo, remove your labels and try again.

Once your video tracks are aligned, you need one final video track that comes all the way on top. This is an empty, 'override' track. Anything you put in this track will override the main track. This way, you do not have to cut from the main track to another camera; instead, you just copy from a camera track and paste on the override track.

It is, of course, possible to paste onto the main track, but that has the disadvantage that you cannot easily undo such edits after a 'save' operation. Using a separate 'override' track does allow that.

To copy from one track to another track without moving in time, use the following method:

1. Select the region where you wish to select another camera.
2. Arm the camera track that you wish to copy from, and make sure not to arm any other track.
3. Hit 'c' to copy the clip at that point.
4. Disarm the camera track, and arm the 'override' track
5. Without moving or removing the selection, hit 'v' to paste onto the override track.

All this may be basic knowledge for people who are more flexed in nonlinear editors than me; but I couldn't find this procedure on the Internet, so I thought I'd put it out there so that the next person who tries this doesn't have to try for days on end to come up with this, as I had to do.

/en/computer

PermaLink

Mon, 22 Mar 2010

N900 back!

So, a short while ago, the battery of my N900 had run flat. As in, completely, utterly flat. In itself, that's no big deal -- you connect it to a power outlet, wait until it's gone past the low-battery charging bits, then boot and use as usual.

Except that it wouldn't. I'd recently installed a firmware update from Nokia (which contained some bug fix that I really was waiting for), and now the bloody thing wouldn't boot. Rather than do the low-battery charging dance, it thought the battery had enough power that it could just go ahead and boot.

Of course it was wrong, so whenever I powered it on, it would sit there fore 30 seconds, try to boot, fail, power off for about a second, and start over. I left it like that for a night, but it didn't recharge.

Needless to say, that was quite annoying.

So I sent it in for repair, and today there was a package in the mail, containing my (repaired) N900.

They'd reflashed it. A reasonable course of action, I guess, but of course that meant all data was gone. Good thing it has a backup application, and good thing I stored those backups on an external micro-SD card.

Restoring those back-ups was a breeze; they'd also stored the contents of sources.list and have done something akin to "dpkg --get-selections", since now the application manager kindly asks me whether I wish to reinstall the applications that I had installed. Cute.

At any rate, I'm very happy I finally got rid of this awful Samsung "cellphone". It's a SGH-C140 that I had as a replacement until my N900 got back, but there was something very very wrong with it; I don't know whether it's a design flaw or just a wear-and-tear issue (these replacement phones aren't usually handled in the best possible way), but it managed to lose connection to the network every once in a while. So people who would call me would immediately get my voicemail, and then they'd be angry with me later because I was unavailable. Well, duh.

Anyway, that's over now. Whee.

/en/computer/hardware

PermaLink

Fri, 05 Mar 2010

Netgear WNDR3700 and OpenWRT

I wanted a machine on which I could easily run OpenWRT. So I'd went to the #openwrt channel on freenode a while back, and just asked for suggestions; people suggested to me that the Netgear WNDR3700 was a good choice, so I ordered that.

I assumed that it would be easy enough to install OpenWRT on this device, but hadn't actually looked into it, planning to wait with that until the device had arrived. Little did I know that the machine actually comes with OpenWRT preinstalled. Now there's an interesting twist.

Now you do need to run some "telnetenable" thingy to be able to get a shell, after which "telnet <device>" gets you a root shell (with no username or password by default). Supposedly you should update that by using "passwd", but they managed to break that in the firmware that comes with the device.

I am missing a few things, though.

root@WNDR3700:/bin# dmesg
/bin/ash: dmesg: not found
root@WNDR3700:~# uname -a
/bin/ash: uname: not found
root@WNDR3700:~# hexdump /bin/config |more
/bin/ash: less: not found

Unh?

root@WNDR3700:~# alias
more='less'
vim='vi'
root@WNDR3700:~# 

Aahh.

And for those who were wondering: no, it does not have any 'vi' installed, either.

Oh well.

The fun thing is, this device has a USB connector, too; so it should be possible to connect a USB storage device, install Debian, and use it as a very potent home server/router/switch/whatever. That'd require me to understand how hostap works, though, which I haven't played with yet. I'm sure I'll figure that bit out -- at some point.

/en/computer/hardware

PermaLink

Wed, 03 Mar 2010

dpkg vs RPM

Thomas blogs about some issues he had with his N900's facebook plugin. This post isn't about that, as I don't use facebook.

But as part of his blog post, he mentions the following:

This reminded me of a pet peeve I have with those people who claim Debian’s packaging system to be far superior to rpm – apparently dpkg doesn’t have any equivalent of rpm -qv which allows you to verify that the files that should be installed by a package are indeed on disk

True, probably because the script would be so trivial:

for i in $(cat /var/lib/dpkg/info/nbd-client.list)
do
	[ -f "$i" -o -d "$i" ] || echo "$i missing"
done

There, that wasn't hard, was it?

Now I'm not sure whether rpm's -qv option actually checks the checksum of the files, too. If it does that, then the semantically similar way would be:

(cd / && md5sum -c var/lib/dpkg/info/nbd-client.md5sums)

... except that MD5 is totally and utterly useless these days, and that we should be changing to something else. And that md5sums is an optional feature, provided by some, but not all, packages. And it may also be that maemo packages don't have md5sums (which would make sense). But, anyway.

/en/computer/cluebat

PermaLink

Wed, 17 Feb 2010

Local kernels

LC_ALL=C debian/rules debian/control
md5sum --check debian/control.md5sum --status || \
		/usr/bin/make -f debian/rules debian/control-real
make[1]: Entering directory `/home/wouter/debian/other-peoples-source/linux-2.6-2.6.32'
chmod +x debian/bin/gencontrol.py
debian/bin/gencontrol.py
Traceback (most recent call last):
  File "debian/bin/gencontrol.py", line 331, in 
    Gencontrol()()
  File "debian/bin/gencontrol.py", line 14, in __init__
    self.process_changelog()
  File "debian/bin/gencontrol.py", line 305, in process_changelog
    (distribution, version))
RuntimeError: Can't upload to unstable with a version of 2.6.32-8~local1
make[1]: *** [debian/control-real] Error 1
make[1]: Leaving directory `/home/wouter/debian/other-peoples-source/linux-2.6-2.6.32'
make: *** [debian/control] Error 2

The issue at hand was that I'd created a version of '2.6.32-8~local1', to document that I'd locally branched version 2.6.32-7 with that one config option turned on, but that my version should not be deemed larger than 2.6.32-8 (signalled by the ~ and whatever follows that). However, something somewhere in the Debian kernel build system (I was unable to figure out what, exactly) disliked that version number and told me I could not upload to unstable with that version. So it borked, and told me I had to fix my version.

Well, no, I don't want to do that. Instead, I wanted to disable that check somehow. Turns out that isn't too hard; gencontrol.py not only checks the version number, it also checks the target distribution. So if you don't want to upload to unstable, you just need to tell the tool that:

-linux-2.6 (2.6.32-8~local1) unstable; urgency=low
+linux-2.6 (2.6.32-8~local1) local; urgency=low

Simple, but you'd need to know.

/en/computer/debian

PermaLink

Mon, 15 Feb 2010

Hum.

I wonder how these people will react to this.

At least it's going to be somewhat more open, I guess. Which is good.

/en/computer

PermaLink

Sat, 16 Jan 2010

ACCEPTED

My mutt said this last night:

 894   + Jan 15 Archive Adminis (0,4K) ipcfg_0.1_amd64.changes ACCEPTED

This obviously means that if you wish to use it, you no longer need to go through git; you can just add experimental to sources.list and run 'apt-get install ipcfg'. A few notes, though:

• I missed a few build-dependencies, so it's only available on amd64 for now. Yes, I should use pbuilder more (I did for the 0.3 upload, which went in today).
• The documentation is somewhat lacking. I just found out that the documentation isn't actually installed in the binary package, so you have to download the source package (or check out the git repository). This will be fixed for the next release. The documentation also currently consists of a few .txt files rather than (a) proper man page(s). This will likely take some more time.
• ipcfg Conflicts:, Replaces: and Provides: ifupdown. This is because there is no proper virtual package name for it to conflict/replace/provide. I have filed a bug, #554194, against debian policy, to request it be made, but so far this is not yet finished (and indeed was somewhat dead during the period that I had no time to work on ipcfg). This may confuse some packages that really do depend on ifupdown.
• ifdown works, but not reliably so. This will be my next main point of focus.
• If you use the insserv automatic ordering system for initscripts, then you must either purge ifupdown or remove its initscript before installing ipcfg; otherwise, the availability of two initscripts that provide network configuration will confuse the hell out of insserv.

And in case you wonder why the hell I went from 0.1 to 0.3:

ipcfg (0.2) experimental; urgency=low

  * Rebuild without .git directory. D'oh.

 -- Wouter Verhelst   Tue, 12 Jan 2010 17:43:09 +0100

srsly

/en/computer/code/ipcfg

PermaLink

Wed, 13 Jan 2010

Using ipcfg, now

Yesterday, I had some time to debug ipcfg some more. The main blocker for me to upload it to unstable was the fact that I could not get WPA security to work; therefore, I could not use it myself. In the interest of "eat your own dogfood", I did not feel that uploading some experimental code to Debian that I'm not using myself is a good idea.

That problem is now fixed, albeit through something of a hack, one that I hope will not be necessary forever: I decided to write a plugin to run ifupdown extension scripts (found in /etc/network/if-*.d). It does require some set-up, and there are still some severe issues; but as of now, I am using ipcfg rather than ifupdown on my laptop.

Those interested in trying it out can either wait for ftp-master to ack the upload and then install from unstable, or just fetch it from the git repository.

/en/computer/code/ipcfg

PermaLink

Tue, 29 Dec 2009

Perl on the N900

Since about a week, I have a Nokia N900. They're not actually on sale in Belgium yet, but I managed to get hold of one through a grey market shop.

The device is quite nice. I do have a few irks with the interface here and there, but Nokia has a public bugzilla that I'll surely use to file bugs whenever and wherever appropriate. And after using it for about a week, I thought of migrating this small application which I once wrote on half an afternoon using perl, postgres, and Gtk2, to the device. It's mainly a cataloguing system that I store my DVD collection in (which has gotten large enough that this is necessary), nothing earthshattering. The ideal candidate to get my feet wet in developing for the N900, so to speak.

That effort got stopped dead in its tracks pretty quickly.

Nokia-N900-42-11:~# perl
use Gtk2;
Can't locate Gtk2.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.8.3 /usr/local/share/perl/5.8.3 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.8 /usr/share/perl/5.8 /usr/local/lib/site_perl .) at - line 1.
BEGIN failed--compilation aborted at - line 1.
Nokia-N900-42-11:~# apt-cache search perl
perl - Larry Wall's Practical Extraction and Report Language.
libpcrecpp0 - Perl 5 Compatible Regular Expression Library - C++ runtime files
po4a - tools for helping translation of documentation
libpcre3 - Perl 5 Compatible Regular Expression Library - runtime files
libgdbm3 - GNU dbm database routines (runtime version)
perl-base - The Pathologically Eclectic Rubbish Lister.
liblocale-gettext-perl - Using libc functions for internationalization in Perl

Or, in other words: they only ship the perl bits needed to make dpkg run.

Hrmpf. This would make stuff slightly more... involved.

Oh well.

/en/computer/code/n900

PermaLink

Sun, 20 Dec 2009

Re: Grrr

I received a number of comments on my "Grrr" post, all of which missed the point:

Yes, I am aware that there are many more ways to fix this issue beyond a memcpy. However, the example code is legal and would not crash the application, if not for the fact that libc thinks I am doing something wrong. On top of that, this kind of overflow "protection" only kicks in when the code is compiled with -O2 rather than with -g -O0. While I am not sure whether the difference is due to the absense of debugger symbols, or rather due to the different optimization levels, fact is that software which runs fine in debugging should also run fine in production.

There are good arguments for compiling all C code with -Wall -Werror, and I do that as a matter of course for all C software that I write. However, sometimes automated tools are just wrong in their compile- or run-time bug detection, in which case such it should be possible to disable that detection. This is one such case, and my blog post was more about ranting about the inability to do so, rather than about the fact that I had to memcpy when in fact there were other options available.

But yeah, perhaps I should have been clearer about that. Forgive me for not being clear after having fought with compilers for far too long.

/en/computer/code

PermaLink

Good, not evil

There is a bit of a fluff online currently about the following clause in the "jsmin" code (whatever that is):

The software shall be used for good, not evil

This seems to have started when Google rejected a project based on that code due to its license being not free or open source according to their standards, and therefore not welcome anymore.

The arguments then quickly degenerated into things like 'when did google stop being against evil'. But those are all besides the point.

One of the most important properties of free and open source software is that anyone can use it for any purpose; there are no restrictions to using them. The DFSG (and hence, the OSD which was derived from the DFSG) encode this as follows:

No Discrimination Against Fields of Endeavor
The license must not restrict anyone from making use of the program in a specific field of endeavor. For example, it may not restrict the program from being used in a business, or from being used for genetic research.

You may think "how is 'evil' a field of endeavor", but that is the wrong question. To "use software for evil" can mean any of a number of things, including "nuclear research", "weapons development", "abortion", or, heck "the cash register in a butchery shop", depending on the ethical and moral framework through which the person writing the license sees the world.

The ability to give someone a CD or DVD with a bunch of software on there, being able to tell them that they can just use this in any which way they see fit, is a very strong and important feature of the free and open source community. Every time someone comes up with a clause like the above, however, this ability is diluted somewhat; and if it is readily accepted within the greater free and open source movement, then eventually everyone interested in using a piece of software must first check whether they're not trying to use software that forbids someone's pet evil, and we lose one of the greatest strenghts that does exist for free software, but not for proprietary software.

The sad thing is, the jsmin author seems to agree. From a video/transcript on which he talks about his absurd license clause is the following quote:

Also about once a year, I get a letter from a lawyer, every year a different lawyer, at a company – I don't want to embarrass the company by saying their name, so I'll just say their initials – IBM…
[laughter]
…saying that they want to use something I wrote. Because I put this on everything I write, now. They want to use something that I wrote in something that they wrote, and they were pretty sure they weren't going to use it for evil, but they couldn't say for sure about their customers. So could I give them a special license for that? Of course. So I wrote back – this happened literally two weeks ago – "I give permission for IBM, its customers, partners, and minions, to use JSLint for evil."

Or, in other words, all you have to do if you want to use this software for evil is set up a second company, tell Douglas that this second company wants to sell software that uses his software to people who might use it for "evil", even though the first company won't, and you're in business. Because Douglas doesn't really oppose evildoers.

So the question is, why is that clause in there in the first place? There are only two possibilities; either Douglas didn't really think about those issues, in which case I hope he will one day see the light and remove the clause; or he did, and decided to go ahead and put that clause there anyway. And that would be evil.

/en/computer/legal

PermaLink

Wed, 16 Dec 2009

Grrr

struct {
	char str[4];
	char sc1;
	char str2[3];
	char sc2;
} foo __attribute__((packed));

snprintf(foo.str, 5, "%04X", data);
foo.sc1 = ';';
snprintf(foo.str2, 4, "%03X", otherdata);
foo.sc2 = ';';

Yes, I know that both snprintf() calls in the above snippet will overflow their immediate buffer. Yet this code is safe; the network protocol for which this code is written does not actually need nor expect NUL bytes; instead, it wants semicolons. I could of course use a "char foo[9]" rather than a struct as above, but I find this to be slightly more convenient than to count offsets.

However, this code does not work with glibc, because the buffer overflow detection kicks in.

Solution:

char buf[5];

snprintf(buf, 5, "%04X", data);
memcpy(foo.str, buf, 4);

In other words: add a stupid and useless memcpy, because someone thinks they're smarter than me. Stupid morons.

/en/computer/code

PermaLink

Tue, 08 Dec 2009

Hardware RNG

So, as many people probably know by now, the nice folks over at Simtec Electronics, some of whom were at DebConf9 last summer, have created a nice small device that plugs in a USB port and, with a userland daemon and some encryption for security, generates entropy (randomness) for the Linux kernel to use (through /dev/random and friends).

I've plugged one in my server today, and suddenly my server's entropy pool was full. This is a really nice thing. For a simple example of what happens when you insert such a thing into a server, check out my munin graphs.

Very nice for such a cheap device...

/en/computer/hardware

PermaLink

Sun, 15 Nov 2009

NaN currency

The Daily WTF, an excellent website that I recommend everyone remotely involved in computers has a look at every now and then, has one series of posts where it shows 'strange' or 'funny' error messages from not just desktop computers, but also ATMs and other embedded devices.

One of the more recent entries of these Error'd series contained a picture that made me cringe; it showed 'NaN' for a currency value. Obviously this is an unhandled floating point error, but that's not what made me cringe.

What did make me cringe is that people who are supposed to work with shitloads of money are using floating point numbers to store monetary values. This is stupid. By their nature, floating point numbers are imprecise and inaccurate. The representation of a floating point number is an approximation. This is not the kind of thing one would like for bank transfers. "Mr Foo would like to transfer approximately €100 from his account to that of his neighbour." Err, no; it doesn't work that way.

Instead, whenever you use currency, you should use integer numbers. If your local currency is the US Dollar or the Euro or another similar currency where zero-point-something amounts of money are regularly used, work on the convention that "1" in your variable actually means ".01", or ".1", or whatever is most convenient, and make the right conversion whenever you need to do some output. Do not think you can deal with rounding errors in floating point, because you cannot.

COBOL (I can't believe I'm saying this) actually makes the latter easy, because you can create a statement that says "to output this variable, show the first three digits, then a point, then the last two digits". I believe it doesn't even have support for floating point numbers. Unfortunately this kind of thing is harder in C (and similar languages), but that doesn't mean you should just go ahead and assume that floating points are a good idea...

This is the kind of error I could accept from a first-year graduate student, but not from someone who's supposed to work with money on a daily basis.

Oh well. At least the ATM is about US Dollars, so it's not likely that it's going to be my problem any time soon.

/en/computer/cluebat

PermaLink

Sun, 11 Oct 2009

Fuck you, Sony.

I had wanted to buy a PS3 ever since I learned about this interesting processor that is called the Cell. Not that I'm very much into console gaming or any such thing; I'd have settled for any kind of affordable Cell hardware, really, but that basically is 'the PS3' these days.

What I had missed, however, was the fact that recent PS3 machines don't support running Linux anymore. Apparently this was all over the interwebs, but unfortunately I didn't see that.

That's €300 I won't see again, for a useless (to me) piece of hardware.

Stupid morons.

/en/computer/hardware

PermaLink

Thu, 24 Sep 2009

RedHat is so medieval

So there's three systems. One of them runs CentOS 4, the other two run RedHat EL 4. One of the RedHats has been in use for quite a while, but the other two are freshly installed, and would need to have the same functionality as the first, but don't yet, mainly because they miss a number of packages.

So we do this.

[wouter@working-host ~]$ rpm -qa --qf "%{NAME}" | sort -u > working-host
[wouter@CentOS ~]$ rpm -qa --qf "%{NAME}" | sort -u > CentOS

Next, we compare the two:

diff -u working-host CentOS | grep '^-' | tail -n +2 | sed -e 's/^-//' > CentOS-missing

At this point, the file CentOS-missing contains some 403 lines. So how does one install packages on CentOS4?

[root@CentOS ~]# yum install $(cat CentOS-missing)

It starts thinking about it, then after several minutes comes back to me with "foo is already installed in an older version. I can't do what you're asking me"

[root@CentOS ~]# grep foo CentOS-missing
[root@CentOS ~]#

In other words, it decided that, to satisfy one of the dependencies of one of the packages I'm asking it to install, it needs to have some other package installed, too, so adds it to the set. Then figures out that it's already installed, at an older version than what it was trying to install. Then bombs.

Okay Wouter, what were you thinking? Surely you can't expect a package manager to suddenly update packages when you ask it to install something else? That would mean upgrading and installing in the same command execution! Oh, the horror.

So, okay, we run yum update first.

Do you think that fixes it? Of course not!

It now comes back with... a message saying that a newer version of what we're trying to install is already installed.

Sigh.

So I'm now running 'for i in $(cat CentOS-missing); yum -y install $i; done' on this host. Since it takes yum several minutes to resolve dependencies, that means it's going to take a while to have it install 403 packages.

The not-yet-working RedHat 4 installation seems to have the exact same problem, except that up2date takes even longer to resolve dependencies, so we're not beyond the 'upgrade' step there yet. I'm sure it will come up with all kinds of other interesting failure modes. Luckily the RedHat-missing file is much smaller (83 packages).

How do RedHat users not eat their keyboard in frustration?

Note: yes, I'm aware of apt-rpm. Unfortunately, that is not supported on 64bit installations, because apt (in neither its dpkg nor its rpm variant) understands the notion of multiarch. At this time.

/en/computer/cluebat

PermaLink

Fri, 18 Sep 2009

Announcement

I was really happy a few years back, when Martin announced he'd start working on netconf. Not just because I agreed with his assessment that ifupdown needed replacement; also because I had been thinking about a good way to implement such a thing, and had been planning to start writing 'soon'. With Martin's announcement, I made a few suggestions to give him some input, and then put my plans away—I had more important things to take care of, anyway.

Unfortunately, netconf did not manage to reach the ambitious goals that Martin set out, mainly due to lack of contributors. I wanted to help out; I really did. But Martin's choice of python to implement a 'prototype' which then had to be reimplemented in C or C++ didn't get me very thrilled. Not just because I have a severe dislike of python; also because implementing something twice is, in my humble opinion, not quite the best way to do something like this.

But hey, who am I, if I don't put my code out there.

Never the less, I tried helping. I really did. During debconf8 at Mar del Plata, I approached Martin with the suggestion that I start reimplementing those bits that were unlikely to change anymore in C, as a first step toward something that could be in the base system. Unfortunately, that didn't happen. After trying for about two weeks, I just gave up. I thought I could read python code and reimplement that so that it would work the same way, but I couldn't. I thought I could implement python modules in C without knowing python, but as it turns out that's a laughable idea.

For several months then, I didn't think about it anymore. In March of this year, however, I got a little bored. And what does one do when they get bored? Right, you find something to do. I my case, that was 'figure out how bison works'. I'd of course known about parser generators for quite a while, but just never had the time to dive in deep and figure out how they work. When I needed to do a config file for nbd, I instead hand-wrote my parser and used a lexer from libglib. But I wanted to learn how to do things properly, so I sat down.

Long story short, when I had to choose something somewhat more complex to implement, I thought 'how about a hypothetical config file for a network configuration utility', and I started implementing that. It turned out to be fun, so I didn't stop after I knew enough about bison. And just last week, for the first time I successfully used it to bring up my wired interface using DHCP -- and bring it down again. It can also already manage static interfaces through netlink, though it isn't quite able to the equivalent of 'ip addr flush' yet.

Fun.

The system is quite flexible. It has a plugin interface (which needed some refinement that I did using some help on IRC just today), which should allow developers to implement extra functionality by just dropping a shared object in the right directory. In fact, I'm working on the wireless support (hence my blog post of last week) as a plugin.

Obviously the code isn't remotely ready yet. The wireless code has a long way to go. The firewall module needs to be started. It will probably crash and burn if put to the test. But even so, it already has one feature that ifupdown, in its 10 years of existence, never acquired: it will not try to DHCP off an interface if it finds out that there is no cable connected to it—unless you wrote a configuration that asks you to do so.

I wanted things to be as simple as possible, and therefore the minimal valid configuration file that will do something useful is the following:

That's right, it's empty. This will cause the system to bring up the 'lo' interface at bootup, and do nothing else. If you say 'ifup eth0', where eth0 is a valid and existing interface, it will first check whether there is a link, and if so, DHCP off of it.

There's much to do still, but I have now reached a point where I feel the system is ready to meet all challenges it should be able to meet (or so I hope), and where it might be nice if interested people check it out.

The code is in a git repository that's mirrored on git.debian.org, github.com, and my own git.grep.be. The latter is my --mirror repository, but is not available for cloning to anyone else. If you're interested in checking it out, you may want to start with doc/tech.txt, and/or examples/ipcfg.cfg. I updated the former just yesterday, so it's quite up-to-date. On the other hand, while the latter shows the direction I would like the system to take, the code is a far cry from implementing everything shown there, and it has been a while since I last checked that file out, so the comments in there are, in some cases, somewhat outdated. But that's good, because I want to keep the file as a reference of the goals that I originally had, before I'd written this whole corpus of code, so that I can go back at some undefined point in the future and fix things.

Comments are (more than) welcome.

/en/computer/code/ipcfg

PermaLink

Thu, 17 Sep 2009

#include <iwlib.h>

Writing a network configuration tool is... fun.

Well, mostly anyway.

/***************************** SCANNING *****************************/
/*
 * This one behave quite differently from the others
 *
 * Note that we don't use the scanning capability of iwlib (functions
 * iw_process_scan() and iw_scan()). The main reason is that
 * iw_process_scan() return only a subset of the scan data to the caller,
 * for example custom elements and bitrates are ommited. Here, we
 * do the complete job...
 */

/*------------------------------------------------------------------*/

(from iwlist.c)

Not. The. Way. To. Do. It. Especially not if you're the person who writes both libiw and iwlist. Grmbl.

Also, documentation for libiw seems to be, well, nonexistent.

Fun. Not.

/en/computer/cluebat

PermaLink

Thu, 30 Jul 2009

Passwords

The Debian System Administrators decided, apparently, that disabling password logons is a good thing that warrants a 'Good News' post.

Allow me to politely disagree, for two reasons:

First, an SSH key is a password that is stored on the hard disk, while a 'regular' password is only stored inside someone's brain. While torturing someone to get at their password is arguably possible, it is not possible to do so without this person noticing. The same cannot be said about someone secretly stealing a file from someone else's hard disk; and while it is certainly possible to protect an SSH key with a password, it is not at all required to do so in order to use such keys. As such, on the server end you have no way to know whether a remote client is in fact the person whom they claim to be, just because they happen to have a SSH key that just happens to match the original.

Second, security is not accomplished by forcing people to use things they do not want to use. If you do that, they will find ways to work around your security—leaving you with no security at all.

But oh well, it's not my call to make, so whatever.

/en/computer/cluebat

PermaLink

Tue, 28 Jul 2009

Extramadura: gnuLinEx and NBD

So obviously I already knew that the region of Extramadura uses a version of Debian they call gnuLinEx, but I didn't know the specifics. As such, it was nice and interesting that they offered us the option of going to a local school, where we could see an installation of gnuLinEx in action. Obviously I went there.

This was an interesting experience, for sure. When I arrived in Cáceres, I learned from Vagrant Cascadian that the school installations make extensive use of LTSP. This, in turn, uses NBD. Since they run this on 80.000 computers, it's quite likely that they're the largest NBD installation in the world. I had no clue.

So, today, when noticing that nbd-client was used on the local machines, I had a short little chat with José, one of the guys from gnuLinEx who's a Debian Developer, about the fact that I've been wanting to do some work on NBD's performance (mostly profiling runs etc), but that I don't have the setup to do this efficiently. To keep a long story short: I now have a test lab the size of which is several times the country I live in. Whee.

I had my camera with me, and took some pictures. I'll upload them soon, but have some other, more urgent, matters to attend to now (such as 'eat').

Later.

/en/computer/debian

PermaLink

Sat, 25 Jul 2009

RC NMUs

A few days ago, during DebCamp9, someone NMU'ed belpic to close #525593, a 'failed to build from source' bug that was filed against it on april 25th, 2009.

Since such bugs (for good reason) are deemed release critical, my package was facing the prospect of being thrown out of testing, and this person (who shall rename nameless in this blog post, since it is not about fingerpointing; but if you must, check the bugreport to find out who) did an upload to prevent that from happening.

In and of itself, this is a good thing. I don't like it when my packages have bugs, and generally try to keep the number as low as possible. There used to be a time when my maintainer bug listing had zero open bugs for most of the time, and though I long ago gave up trying to keep it that way, it is still a goal I would like to reach at some undefined point in the future. Yet, with this particular upload, I was quite unhappy, even going so far as to cancel it, preventing it from going into the archive; and when, over lunch, I discussed the situation with Adeodato Simó, he seemed to feel that I improperly blocked this upload; that I instead should have allowed it to proceed. I felt as if he was almost hostile to my notion that the uploader should have coordinated more with me than had happened. I tried to explain why I felt the uploader had done something wrong, but I do not think I convinced him.

After having thought about it for a few days now, I still don't feel I was wrong in my actions; and I don't like the idea of possibly being a bad maintainer. So here's my position on the whole thing:

First of all, the bug was indeed open for a long time. The reason is that I didn't have much spare time in May to look at it. I also had no clue as to what the problem was, which makes it kinda hard to fix it. I did have some spare time in June, but since Belgian citizens have to file their tax reports in that month, and since you may need the software in these packages to be able to do so, I did not think it was proper to indeed do an upload before the deadline of June 30th; I did not want to have to scramble to fix a botched upload at exactly the wrong time.

On the 11th of this month, a patch was sent to the bugreport that would fix it. This was the weekend before I would leave for DebCamp/DebConf, so I postponed working on it until I would arrive here in Cáceres.

Secondly, what I'm most upset about is the fact that the upload wasn't tested. It couldn't have been; the only way you can test this package is by using a smartcard reader and a Belgian ID card; since the uploader does not have the Belgian nationality, I doubt he has such a card. I don't personally do an NMU that often; but when I do, I consider it my duty to perform extensive testing, even more so than with my own packages. In this particular case, doing such tests is quite important, as I've had issues with the software in the past, where new builds wouldn't work properly for reasons that I haven't fully been able to pin down.

Of course the uploader couldn't know all that, but this is exactly why he should have talked to me before doing the upload. Of course, in theory, I could have documented all my knowledge about the package; but in practice, it's pretty hard to do that well (i.e., many of the things I know about my packages involve 'feeling', 'instinct', and 'experience', which does not easily translate to 'words').

For clarity: I'm not upset about the fact that I've been NMU'ed. That's happened in the past, even for this particular package, and that's been perfectly fine. What I am unhappy about, is that the NMU happened to DELAYED/1 (which gives me very little time to intervene) and that it happened without prior coordination with me.

So, was I really wrong in preventing the upload?

note: since the new upstream release that was available was eventually uploaded to experimental rather than unstable, I just uploaded 2.6.0-7 which also fixes the bug in question.

/en/computer/debian

PermaLink

Fri, 24 Jul 2009

Bison

The Bison parser expects to report the error by calling an error reporting function named "yyerror", which you must supply. It is called by "yyparse" whenever a syntax error is found, and it receives one argument. For a syntax error, the string is normally "syntax error".
If you invoke the directive "%error-verbose" in the Bison declarations section, then Bison provides a more verbose and specific error message string instead of just plain "syntax error".

Sounds good, right?

Well, no, not entirely.

syntax error, unexpected $undefined

Well, goody. Now I know what's going on.

note: yes, I do know that there are other ways to debug a Bison parser than just to use the parser error string. It's just that this could have been more useful, like, say, provide the line on which the error is found? The file I'm trying to parse here is pretty large, thank you very much.

/en/computer/code

PermaLink

DebCamp 9: stuff

This has, by far, been the most productive DebConf ever, for me.

Not that this means all that much—mostly that previous DebCamps haven't been productive at all—but I still got a few things done.

There've basically been three things that I worked on: d-i support for the Intel SS4000-E; a belpic/beid upstream update; and a minor incremental NBD update.

The latter was simple, and was basically the first thing I did. I had a chat with Vagrant Cascadian, who does a lot of LTSP stuff in Debian, and added some stuff to the package that would make his life a bit easier. Not a lot of work; as I'm also upstream for NBD, and as it's been one of those packages that I've maintained since an eternity, I know the code pretty well. Half a day later, all the code was there.

D-i support for the SS4000-E wasn't that hard (most of the hard parts had already been done by Martin Michlmayr), but unfortunately some bits are not yet completely in order—mostly having to do with the fact that the original firmware has a kernel command line embedded in the kernel. As such, for now, you'll have to connect to the serial line in order to fix the redboot config; maybe we'll come up with a sane way to fix that in the future, but as long as we don't, that does mean you need a serial null modem cable.

Not that you need to solder anything (the main board has a connector for a regular serial port; you just need to plug the right cable to the right connector, so it's not that bad.

The final thing was horrible. A piece of software that presumably works well, but initially wouldn't even compile on my laptop because of pointer/int confusion; a build system made on shell scripts and qmake; and other similar things.

Eventually, I just gave up and uploaded what I had to experimental. It works, to some extent, but should be improved over the next few weeks. That's not for today, however.

/en/computer/debian

PermaLink

Microsoft and the GPL

For some strange reason, people all over the net are oohing and awing over Microsoft releasing some drivers to use Linux on their proprietary virtualization software. I'm oohing too; not because of the drivers, but because of all the buzz that goes around it.

Ten years ago, I would have oohed and awed, too. At that time, Microsoft was fighting open source and free software like a cancer. Today, they're not; they provide open source software for Windows themselves (such as an installer framework which they provide through sourceforge), and actively cooperate with many open source and free software projects through their open source labs. They even have a section of their website dedicated to open source software

A large company like Microsoft can't survive if it tries to actively work against what the marketplace wants. The fact that they were indeed so actively fighting against open source software is quite likely why the first decade of the 21st century has seen such a huge loss of market share for them; like any large company, they needed to adapt or lose. They've chosen the first; good for them, and that might be good for us too.

Over the past decade, I've seen Microsoft warming up to open source, to some extent. This is why I don't understand much of the 'Boycott Novell' lunatics; sure, I don't trust Microsoft enough yet to be willing to say that they don't have any plans that will negatively affect us; however, that doesn't mean I will assume that evil things are their plans; unless proven otherwise, I will assume they have the best interest of their customers and/or shareholders as their main goal.

Which is why I was totally not surprised in seeing a GPL patch from Microsoft at this point in time. Rather, I find it normal and expected behaviour, a continuation of an evolution that has been going on for the better part of a decade now.

/en/computer

PermaLink

Wed, 22 Jul 2009

Buildd maintenance

In Debian, I've been a buildd maintainer since 2001; most of that time was for the m68k port (I still am active there, though not as much as I used to be), but there's also been a short stint with armeb, and since a while I'm now also a PowerPC buildd maintainer. I used to do just one powerpc host at first, but now I maintain both malo and voltaire, with Philipp Kern doing praetorius.

This probably makes me one of the more experienced buildd maintainers in Debian today, together with the likes of LaMont Jones and Ryan Murray. I did do a talk about how this is supposed to work at FOSDEM 2004, but that's now five years ago, and some things have changed since. Also, a not-videotaped talk isn't very helpful if you weren't there.

So I'd thought I'd write up what it means to be a buildd maintainer. There's of course the documentation on the Debian.org website, but that only explains how the system works in theory; it does not explain what us buildd maintainers tend to do on a daily basis.

So let's have a look at that, shall we?

Basically, the work of a buildd maintainer is pretty monotonuous, and an experienced buildd maintainer will usually have a set of scripts to help them. Their work can be categorized into three main categories. In order of frequency, these are:

1. Log handling;
2. State handling;
3. Host and chroot maintenance

Log handling

The first is the most obvious one. Every time the buildd builds a package, it will send a full log of the build to buildd.debian.org and to myself. The successful ones are signed with a simple script:

#!/bin/bash
tmpfile=$(mktemp)
sed -i -e '1,/\.changes:$/d;/^[[:space:]]*$/,$d' $1 | tr -d "\200-\377" > $tmpfile
cat $tmpfile > $1
rm $tmpfile

Easy: use a sed command to fish out the embedded .changes file, and write that to the original file. I use a folder-hook to set this script as my 'editor' in mutt when I'm in my buildd mail directory; thus the result is thereby mailed off to the buildd. In that same folder-hook, mutt is also configured to send the reply gpg-signed in the 'traditional' format, without confirmation, and with just one keystroke, so that (after I have entered my GPG key passphrase) I can send off all the signed changes files in one go. A possible improvement could be to change the macro so that it would work with mutt's 'tag' feature (it doesn't, currently), but that's not a big issue (currently, doing 100 mails takes a few seconds and some careful counting).

Note the 'tr'; this is to avoid 8bit characters from appearing in the mail, which might otherwise be converted to their quoted-printable version in transit to the buildd; and since buildd-mail (the part that receives that mail) does not understand MIME, this would corrupt the GPG signature. This way, we do lose a few characters from the changelog, but that doesn't really matter -- the source still contains the unmodified changelog entry.

With this script, I often handle my 'success' folder several times a day. It's no effort, anyway.

The somewhat harder but much more fun part of log handling, is the handling of failure mails. Since there are loads and loads of possible failures, the scripts to handle these are somewhat more involved. I did receive a script from LaMont at some point, a few years ago, which I then built on so as to improve it. It's not perfect, but it does handle a few common cases with no extra input from me. Some of the others are not so easy, however.

One of the more common cases that cannot easily be automated is the case of the buildd failing to install a certain package, because 'foo depends on bar, but it will not be installed'. This is apt's way of telling you that bar depends on foobar which depends on quux (>= 1:2.3.4-5) which depends on libfrobnitz2, but that has now been replaced by libfrobnitz3. Or some such. The only way to figure out what the hell the problem is, is to walk the dependency tree and figure out stuff from there.

There is an 'edos-debcheck' that reportedly can help with this; personally, I wrote a set of perl scripts that will cache a Packages file into DBM files, and then allow you to walk over them to help you figure out what's wrong. They're not perfect, but if you use the '-v' option to check-dep-waits and verify the output when it tells you about missing libraries, it should be able to figure out the whole dependency tree I described above, and will allow me to write a proper dep-wait response, allowing the buildd host to automatically retry the package when the missing dependency is available.

Also somewhat common and routine are things like transient network failures (in which case we use either 'retry' or 'give-back' if the buildd hasn't figured that out by itself and done the latter), the maintainer uploading a new version of the package while the previous version is building (resulting in wanna-build firing off an email to the buildd host, which in turn results in buildd killing the build by removing the build directory; this is not always easily distinguishable from a regular failure, so I commonly respond to that mail with a failure message; if it did indeed fail because of a newer version, then buildd will notice that and ignore my mail), the incoming.d.o Packages file (which is only available to buildd hosts, so don't ask) being out of sync with reality (which happens 4 times a day for about an hour. In this case build-deps will fail to install, requiring a retry or give-back), and similar things.

Other things are less common; but because of that, they are not routine and require an in-depth investigation. Sometimes the fix is to just file a bug report and/or to mark the package as 'failed' (and let the maintainer or a porter handle the problem); sometimes the failures are due a maintainer script in a package being utterly broken, resulting in either some build-deps being uninstallable or (worse) the buildd chroot being fucked up. Sometimes a build is interrupted halfway through, leaving the chroot in an unclean state (sbuild is not pbuilder, and does not remove and recreate its chroot between builds). This would push us to category 3 of our work.

Basically, however, figuring out which is which takes some experience. Not all compilers are based on gcc (there are some really weird languages in Debian), and thus not all of their error output is the same; learning their different error modes can help quite a lot. Additionally, by continually compiling 10G worth of software, you'll be stress-testing your toolchain. If you've never seen an 'Internal Compiler Error' before, you will once you become a buildd maintainer, and it helps if you know what they are and how to deal with them (even if there isn't much one can do beyond filing bugs).

Obviously, handling failures takes some more time than does handling success mails, and it's not something I do quite as often. The exact time between both varies, but it's usually somewhere between a few days and one or two weeks—unless I suddenly stop receiving success mails from one of my buildd hosts, in which case I know something is utterly wrong and will usually investigate immediately.

State handling

With 'state handling', I mean managing the state of a package in the wanna-build database. There's help about this from the people on the debian-wb-team mailinglist; call me oldfashioned, but I still do consider this to be the final responsibility of the buildd maintainers. After all, the routine state changes are a result of decisions that I make; as such, if I fuck up, it should be me who fixes the fuckup. Also, if I mark a package as 'failed' because I believe the maintainer fucked up, then the debian-wb-team people may not know about my reasoning there, and might give the package back to another failure (although I would consider the latter pretty rare).

These requests are pretty common. Quite often, they're unnecessary—many maintainers are unaware of the intricacies of the wanna-build system, and may misunderstand that when a build is in dep-wait state, it will automatically migrate to needs-build once dependencies are available. About as often, however, they are very much necessary, and, since regular Debian package maintainers do not have access to the wanna-build database, require someone who does have access to said database to update it for them.

Having said that, there are cases where I will preemptively edit the wanna-build database. Usually this is to do something useful with packages in 'Building' state that have been in that state for far too long; either upload the package if its signature mail got lost (which happens once in a blue moon), or give the package back if its build was not attempted even though it is marked as such (this should not happen, but the system is not perfect and it does). Sometimes this is because I figured out that some common build-dependency (say, the GTK or Qt libraries) are in a transitional state and currently not installable; and rather than having a build daemon try a bunch of packages and failing them all, I may want to note in the wanna-build database that they should not bother attempting these 75 packages before the GTK package was done. This isn't done as often on the official Debian machines (since the release managers will do it for me there), but in m68k we do need to do this ourselves.

These kind of requests happen once every few days up to once every few weeks, and take little time to deal with.

Host and chroot maintenance

This is the hardest and least fun part of buildd maintenance, but it is just as necessary. Luckily, it is not as often needed.

Because Debian Unstable is a system that's in a constant state of flux, often things will break. This is even more of a problem on a buildd chroot, since it builds out of incoming; a maintainer may upload a package with a fucked postinst script, have its build succeed, but then fail spectacularly to install. This maintainer may notice that, and may upload a new package half an hour later. As such, the broken package will not end up on the system of a user or Debian Unstable, but between the time of the upload of the broken package and that of the new package, the old package will be available to buildd hosts, who may use it to completely and utterly destroy their build chroot. The joys of having a high turnaround time.

Luckily, Debian package maintainers are not stupid, and this kind of fuckup does not happen every other day. It does happen, however, and when it does, this often means manual work for the buildd maintainer. In the best case, it's a matter of syntactically fixing a postinst script and calling 'apt-get -f install' or 'dpkg --configure -a'. In the worst case (which is almost, but not quite entirely, totally unheard of), it's a matter of rebuilding the buildd chroot. In addition to that, a machine which runs 24/7 for the sole purpose of building packages tends to generate quite a lot of disk activity, which in turn tends to be detrimental to the disk in the long turn. If not looked after properly, disks will die, taking the entire buildd chroot with them. That requires rebuilding them. Obviously, this last issue is dealt with by the Debian System Administration team in the case of official Debian hosts, but the same is not true for the m68k port.

A somewhat more common thing that needs to be taken care of is the fact that buildd does not in all cases clean up after itself. For instance, when a new version of a package is uploaded to the archive between the time that the buildd host built it and the time the buildd maintainer sent the signed .changes file back, then buildd will say "I haven't got that package taken as Building" and refuse to upload it. This makes sense (you can't upload an old version of a package, since there wouldn't be any source for it, and dak would refuse the upload), but it does mean that the packages aren't cleaned out. Arguably a bug in buildd-mail, over time it will result in the disk filling up with outdated packages, and those require manual work from the admin. I recently (as in, a few hours ago now) finished a script to check each .changes file in the "build result" directory against the wanna-build database, and list those that are no longer necessary. I already had a script that, given a list of .changes files, would remove every .deb file listed in the given .changes files, and then proceed to remove the .changes files themselves. Combined, these do make that kind of work somewhat less of a burden.

As said, this kind of work does not need to be done all that often; for instance, I just cleaned the build result directory on voltaire and malo, my two powerpc buildd hosts, and found old files from late 2008...

And that's it, I guess. It may seem to be quite much, but in reality it isn't; the thing I've always liked about buildd maintenance is the fact that you do something little for Debian every day, but that it ends up being something big and helpful after a while.

Of course, the little things are the cherry on the cake. By looking at a lot of build logs, one eventually learns a thing or two about build systems, which is valuable knowledge. Getting build logs from the whole of Debian allows one to learn things about the archive that many people don't know about—for instance, did you know that we had a package called trousers? I didn't, until I signed the buildd log...

Update: changed the URL of this post to be under the buildd/ directory, rather than having it conflict with that and thus killing its permalink and making it impossible to comment on this post. Oops.

/en/computer/debian/buildd

PermaLink

Tue, 21 Jul 2009

Security modules in firefox

One bug I have open against the belpic packages is that upon installation, the security module should be enabled so that the user doesn't have to do this anymore. While I tend to agree that this might conditionally make sense, the problem, of course, is making it actually work that way.

The SuSE folks have written a mono utility to easily enable and disable modules in firefox, thunderbird, and OpenOffice.org. While that's fine, I'd like to be able to enable or disable this security module on a system-wide basis.

Unfortunately, unless I'm missing something, that doesn't seem possible. That is, I can use modutil from the libnss3-tools package to create a security module database in /etc/iceweasel/profile (which gets symlinked to from /usr/share/iceweasel/defaults/profile), but firefox then conveniently ignores it.

I'm guessing the reason for that is the fact that usually, such security modules want to have certificates to go with them. In that case, you wouldn't necessarily want to share them with other users.

But in this case—a security module that enables the use of a smartcard reader to read a key from a smartcard—enabling the module system-wide should not be a security risk.

/en/computer

PermaLink

I love vista.

No, I don't use it, and I wouldn't want to. However, its hardware requirements are so ridiculously high, that hardware manufacturers these days have no option but to seriously bump the performance of their systems. As a result, they ship with ridiculous amounts of RAM and CPU power, while still keeping prices reasonable.

Of course performance still sucks your pants off if you indeed do use Vista, but not if you use an actual operating system.

I love vista.

/en/computer/hardware

PermaLink

Fri, 17 Jul 2009

Death

GNOME-KEYRING MUST DIE

that is all.

/en/computer/cluebat

PermaLink

Wed, 15 Jul 2009

RepRap: followup.

I received some comments on my previous blog post; one of them I guess was from one of the people who helps develop reprap, and who pointed out that the Java software is not the only thing; there are apparently alternatives in python, and a C-based one in development. Goodie.

Another managed to put out this gibberish:

I'd just like to point out that C and POSIX isn't very easily portable to the majority of computer systems that users use (ie. Windows), not to mention that C doesn't have a whole lot of abstractions away from the OS (eg. building up path names is not easily portable in C).

My friend, that is why they have these things called 'libraries' and 'cygwin', or even 'MingW'. There are many programs who were written in C and who are portable across Windows and POSIX; Apache, OpenSC, to name but a few. I'll grant you that it is more work to develop applications that are cross-platform using C, but that doesn't mean it can't be done. At least, however, they'll be easier to install.

/en/computer/hardware

PermaLink