WEBlog -- Wouter's Eclectic Blog

Sat, 14 Apr 2012

LOAD: tutorial on Debian Packaging

Two weeks ago, I was at LOAD, where I did a tutorial on Debian Packaging. Unfortunately, I wasn't doing very well physically, and as a result my preparation wasn't what I had hoped for. It ended up being not so much a tutorial as a walking the audience through packaging a piece of software—in casu NBD with debhelper in dh mode. Not very difficult, but since there's nothing for people to fall back on afterwards, it may help if they have a writedown of what I said during the session; and I promised to put it on my blog. So, here goes. However, note that the canonical written tutorial on Debian Packaging is here.

With apologies to readers of Planet Debian, most of whom for which all this is probably old hat.

Unlike RPM packaging, in Debian, the data that tells the debian packaging system how to package a piece of software is functionally split among a number of files. All these files go in a toplevel directory in the source package, (unsurprisingly) called 'debian'.

There are four files that are required for every Debian package: the control, changelog, copyright, and rules files. Without these files, dpkg-dev will fail to produce a package—any package.

The first, the control file, contains metadata on packages: the name of the package, its description, the dependencies, etc. Basically, almost all the information you see when you run 'apt-cache show <package>' (with one exception) is in the control file. If you have a source package that builds multiple binary packages, then you should have multiple 'Package' stanzas in the control file.

In my opinion, creating a control file is easiest done by copying a control file from another (similar?) package, and modifying it to suit the software you're dealing with.

The one exception, the one bit of metadata that is not found in the control file is the version number: this data is contained in the changelog file. This file looks like a free-form file for the most part, but it really is a machine-parsable format; as such, it's best to edit it with specialized tools, such as the debian-changelog-mode in the emacs editor, or the debchange script, also available as dch, from the devscripts package. In the changelog, you should document any changes you make to the package, making sure the version number (top line), distribution (unstable, experimental, stable, ...; top line), urgency (top line; used mainly when testing needs to be updated urgently for things like security updates), author (bottom line) and date and time (also bottom line) are correct.

The copyright file, unsurprisingly, should contain the copyrights statements of the original software, and the license (or a reference to a copy of the same license text under /usr/share/common-licenses). This file is still free-format in most packages today, although a machine-readable format for this file has recently been defined. Of the four required files, this is probably the most boring one, but hey, we can't like everything we do.

The last file, the rules file, is where all the action happens. This file is defined as a Makefile, of which the targets are called for various parts of the build system.

The rules file has a number of required targets (such as build, clean, and others), but if you're using debhelper in dh mode, you don't need to worry about those; instead, you can use the following simple rules file:

#!/usr/bin/make -f

%:
	dh $@

With the first line, the shebang, we make clear that this is a makefile, and that it should be called by make. The next is a generic target (the % is a wildcard for make), with just one command: 'call dh with the name of the target being called'. Since dh implements all required rules targets, that immediately gets you a working package. Go ahead, try! Run 'dpkg-buildpackage -rfakeroot', and see what happens.

Did that work? Maybe, maybe not. First, you'll have seen loads of warnings about something involving a "compatibility level". This is from debhelper; this compatibility level, or 'API level' if you wish, allows debhelper to move forward and make incompatible changes without breaking all the existing packages out there; whenever the compat level is bumped, some new functionality will only be made available if you also raise the compat level in your source package (and then you'll know you may have some changes to do in your package). Since the original debhelper, over a decade ago, did not have a compat level, the absense of a compat level signals compat level 1, which is far outdated now, and about to be unsupported. Hence the warning. The fix is simple: create a file debian/compat containing a single number: the compatibility level you're working with. It's best to use the most recent level which debhelper supports when you create your package, which as of this writing is level 9.

Ignoring the compatibility level, if you're building a software package which uses a well-established build system, and you only build one binary package out of that, chances are pretty high that everything worked as expected. If not, you'll have more work.

Building multiple packages requires that you tell debhelper somehow which file goes in which package. You can do this with a file debian/package.install, where package is the name of the binary package you're building.

The install file is read by dh_install, one of many tools in the debhelper suite. You see, in the old days (before debhelper 7), debhelper was just a suite of tools, which required that you wrote a debian/rules file containing all the individual tools to be called in the correct order. This mode is still supported; in fact, even if you do use dh, you do, since all that tool really does is call the right tools in the right order; the real work is done by the invidual tools. They all have their own man page; to understand how dh_install chooses which file to put in which package, go read man dh_install. Go ahead, do it now; I'll wait.

Back? Good. You've now learned that dh_install can install files either from the source directory (useful for packages containing only scripts or so), or from a directory debian/tmp. If you use an autotools-based software package, this is what dh will do for you.

When you called dpkg-builpackage above, you may also have noticed that the output contained many lines starting with dh_, one of which said dh_install. As you may have guessed, dh echoes every debhelper command just before it will execute them. This allows you to look at the output, and see what's happening. For more detail, set the environment variable DH_VERBOSE to a non-zero value.

I'm sure you'll have seen one or two debhelper tools that could make your package better. Go ahead, go and read their man pages to see what they do, and how they do it. Most of these will require you to create a file debian/package.toolname to specify details.

In some cases, you may need to specify command-line arguments to the tool to get it to do what you want. In yet other cases, the tool won't support what you need it to do, and you'll have to do something manual. What to do now? The dh command line doesn't support adding extra commands easily. Does this mean you'd have to revert to old-style long, non-dh debian/rules files?

Luckily, no. You can create an override target. If dh detects that you have such a target for a particular tool, it will call that target instead of the tool. This target's rules can then call the tool in question (or not), and add any command line arguments, or extra commands, as needed.

An override target is a normal Makefile target with a name of the form override_dh_something, where dh_something is the name of the tool you wish to override.

At this point, I'd reached almost the end of my two allotted hours, and a member of the audience asked how to make dpkg deal with configuration files.

The answer is, you don't need to do anything! Not if you use debhelper, anyway; all you need to do is install the file in /etc. Since debian policy specifies that all files in /etc need to be configuration files and that no conffiles may be placed anywhere but in /etc, debhelper will automatically mark any file installed in /etc as a conffile, which will cause dpkg to ask the user what to do with changes to such files.

Note that in Debian parlance, a conffile is not the same thing as a configuration file: a conffile is a configuration file that is part of the binary package (i.e., if you call 'dpkg --contents' on the .deb file, you'll see it in the output), whereas a configuration file is a file (any file, including files generated during or after installation of the .deb file on a system).

Occasionally, this is also why some differences to configuration files are managed through debconf, while others aren't: the ones managed through debconf are non-conffile configuration files managed through ucf, whereas the others are conffiles.

So in the simplest of cases, if you want to install a configuration file and you want to make sure it's protected against accidental overwrites by package upgrades, all you need to do is make sure it's installed to /etc; debhelper will do the rest.

And that concludes this introduction. Please note it's only an introduction, not a full-blown tutorial; while this will allow you to get started, you may have to learn a bit more if you wish to eventually upload a package to Debian.

/en/computer/debian

PermaLink

Tue, 27 Mar 2012

Bjorn Monnens on Debian

If this were twitter (and if I had a twitter account, which I don't), this post would start with "RT". As it is, you'll have to make do with this link: Bjorn Monnens on why he chose Debian testing over Fedora.

This weekend I switched again to Debian 6 testing as this is also Gnome 3. I had to do the same stuff as for Fedora but I noticed already that it’s much more stable and it is also much faster than Fedora. Boot is really a difference of +30 seconds.

Go team!

(with apologies to those who read this blog through Planet Grep, as Bjorn is there too)

/en/computer/debian

PermaLink

Wed, 07 Sep 2011

No more MySQL on my machine

I've said it before: I think MySQL is a toy, and I want as little to do with it as possible.

Unfortunately, since the KDE PIM suite depends on akonadi, which depends on a database, which was initially implemented against MySQL, not having MySQL installed on a Debian machine if you want to use the KDE PIM suite, for the longest time, wasn't even possible. Today it is, but it requires some manual steps:

sudo apt-get --purge install akonadi-backend-postgresql akonadi-backend-mysql-
akonadictl stop
rm -rf .config/akonadi
rm -rf .local/share/akonadi
cat > .config/akonadi/akonadiserverrc <<EOF
[%General]
Driver=QPSQL

[QPSQL]
Name=akonadi-wouter
Host=
Options=
StartServer=false
EOF
createdb akonadi-wouter

Change "akonadi-wouter" for a suitable database name.

And now it should work. Note that this requires you run PostgreSQL system-wide and create a system-wide database in that server. It should, in theory, be possible to have akonadi run its own private PostgreSQL instance, much like how the MySQL backend does things, but a) I couldn't get that to work, and b) I don't think having each user run their own database server instance is a good idea to start with, so honestly I didn't try very hard.

And there, pronto! I now no longer have toy databases on my laptop. Word.

/en/computer/debian

PermaLink

Wed, 31 Aug 2011

Debian-installer support for NBD

Folkert asks, not without reason, whether my debian-installer support for NBD is available for testing. The answer is: yes, it is!

At DebConf11 in Banja Luka, I sat down with Otavio, who helped me get the necessary modules uploaded in the correct way. As such, you can now install debian to an NBD device with a regular debian-installer build—although for now, you'll probably still need a daily build to get it to work[1].

To test, you should do the following:

• Download a daily image for your architecture.
• Either boot d-i with the extra boot-time argument modules=partman-nbd, or boot it in expert mode and select the partman-nbd module in the the 'download installer components' menu.

And that's it! Everything else should Just Work(tm).

Well, almost. There's some loose ends that I need to fix, which I haven't had the time to look into yet. Hopefully that won't take too long.

[1] It's possible that cdimage builds will work too, but I haven't tested that.

/en/computer/debian

PermaLink

Thu, 18 Aug 2011

Happy Birthday, Debian!

Okay, I'm a few days late, but still.

The 16th of August is Debian's birthday. It was on the 16th of August, 1993, that Ian Murdock announced the 'imminent' release of "the Debian Linux Release".

As such, this year Debian celebrates its 18th birthday. It's incredible, but this means that if Debian was a person, it could now legally drink hard liquor. At least in some jurisdictions.

What's more incredible is that I just realized that, since I've been involved with Debian for over 10 years, I've actually got first-hand experience for more than half Debian's lifetime.

Time sure flies.

/en/computer/debian

PermaLink

Tue, 26 Jul 2011

The things I find myself doing today.

Building debian-installer for m68k

...which requires localechooser

...which build-depends on python-lxml (amongst other things)

...which is uninstallable

...which contains some pythonesque source files that are converted into 6.7M C files (by something called "Cython")

...which is then compiled against python2.6. That takes somewhere between half an hour and one hour on this ARAnyM emulated box (I haven't timed, actually).

...a step which is then repeated for python2.7, and possibly more (I haven't gotten that far yet).

...and this file is located in a directory with dozens more of those pythonesque source files

Looks like this is going to take a while. I should probably look into getting distcc set up.

/en/computer/debian/m68k

PermaLink

Mon, 25 Jul 2011

m68k revival

If you thought Debian/m68k was dead and buried and where happy about it, I'm sorry to disappoint you.

The reason m68k had died out slightly was not that we'd lost interest, or that we'd been kicked out of the archive, or anything of the sorts, as I've seen people believe. Instead, the reasons were fully technical: glibc past some particular version needs TLS (thread-local storage) support, which hadn't been implemented for m68k yet, and nobody within the Debian/m68k team had the expertise to work on this. That was all.

Not so long ago, some people within RedHat had started doing a TLS implementation for m68k, however. This was paid for by one of their customers who needed the ColdFire to work, and therefore it was focused mostly on ColdFire. However, ColdFire is close enough to plain old m68k that it was possible to port this TLS support to plain old without too much effort. Unfortunately, most of our machines were pretty much nonfunctional by that time; so in order to get things to work, we'd have to

• Reinstall some of our machines, fix the others, without a functional installer,
• Do the necessary porting and package integration work,
• Rebootstrap the architecture,
• do other things necessary (such as providing a working installer, etc).

Not quite impossible, but still a lot of work that would need someone to put in quite an amount of time. This wasn't something that people in the m68k port could invest; and I'm also not sure we had the necessary skills amongst the team to do the necessary porting.

So the work lay there, and it wasn't being done. Until one Thorsten Glaser, author of a shell called 'mksh' that I hadn't heard of until then, came along, and noted that his mksh package wasn't being built for m68k. Upon investigating, he discovered, as expected, that the port was almost dead. Now most people would leave it at that and decide that they couldn't build for an architecture that wasn't being maintained anymore—and that would be perfectly fine— but not Thorsten. On the brink of death, he almost single-handedly revived the m68k port, and got stuff to work to such an extent that it is now again possible to build a working and fairly up-to-date Linux/m68k ARAnyM virtual machine.

As I'm typing this, my ARAnyM instance is checking out the d-i repository (so I can start building d-i), and building sudo at the same time.

Let's see what comes from this.

/en/computer/debian/m68k

PermaLink

Wed, 29 Jun 2011

NBD support in debian-installer

... is now working.

It isn't finished yet; there are still some loose ends. But at least it works; if you combine this kernel with this debian-installer image, you'll be able to install to an NBD target.

Known bugs:

• if you install your root device on an NBD device, the installer will attempt to install a boot loader to the local hard disk. Make sure to bring it into expert mode before the tasksel part to avoid overwriting your MBR -- or do what I did, and install to a (virtual) diskless computer.
• Also, due to an issue with the initscripts package, halt will make the kernel panic rather than power down the system—but at that point it's safe to switch off the power, anyway.
• The installer currently fails to add a nbdroot= parameter to the kernel command line when installing root to NBD. When you're rebooting into the installed system, you should add something like nbdroot=<server>,<port> to your kernel command line, otherwise the installed system will not find the root device.

Other than that, it should work fine; testing is welcome (but note standard development code caveats apply).

/en/computer/debian

PermaLink

Wed, 20 Apr 2011

beid software version 4

No, it hasn't been released yet, but they're working on it.

We've not been doing much about beid since squeeze was released, mostly because we understood that version 4 of the software was quite close, and that working on some 3.x version in that light would not make much sense anymore. But that doesn't mean I haven't done anything about it, at all; a while back, the FedICT people contracted me to help them build Debian packages they could put an 'official' stamp on, and provide through their website or some such. Some pre-release versions of these packages are now available through their google code project, and it would be welcome if people could try them out and give feedback.

Links:

• The middleware package contains the browser plugin, which doubles as the PKCS#11 library. It should even auto-install, which the packages currently in the archive don't (due to technical reasons that would take us too far).
• The viewer contains an application that allows you to view the data on your eID card, print out a page with identity information, etc. It's not required for everyone, but can be interesting none the less.

In both cases, look under 'featured downloads'. These contain snapshot builds that should be fairly stable, but are not officially supported yet. Alternatively, you can track the head of the code (packages that are built automatically upon commit) by going to some alternative pages for the middleware and the viewer. Feedback is welcome, preferably through the relevant Google Code bugtrackers.

/en/computer/debian/belpic

PermaLink

Wed, 09 Mar 2011

NBD in d-i

Success!

It only took, erm,

wouter@celtic:~/debian/debian-installer-wouter/partman-nbd/debian$ cat changelog 
partman-nbd (0.1) UNRELEASED; urgency=low

  * Initial release.

 -- Wouter Verhelst   Fri, 28 Dec 2007 11:45:43 +0100

just over three years, but I've finally gotten it to work. More or less.

That is to say, I have a VM currently running which is finishing up an installation. It hasn't completed yet, but it's gotten past the critical point now and I'm hopeful that a reboot test will be successful. If it is, I'll need to make sure that the updates that I've done on the live installer to massage it into running are committed to packages too, and then I'm all set.

Looks like you'll be able to install wheezy on a system without hard disk. How's that for 'Universal operating system'?

/en/computer/debian/d-i

PermaLink