Rock. Or not.

A week or two ago, one of the hard disks in rock utterly broke down.

In itself, this wasn't a problem, since the disk was part of a 3x40G RAID5 set. Moreover, I knew it was dying, and was preparing to buy me a new disk anyway; the fact that it broke down only sped up matters. No worries.

However, there's one slight problem with rock: it's a pretty old SMP box, and because of some bug somewhere (which I still need to hunt down), it can't run anything later than 2.6.12 in SMP mode.

The new disk was a 80G disk, to replace a 40G disk. I wasn't very happy about the RAID5 array, but since I needed more disk space than what a RAID1 or RAID10 array on three 40G partitions would give me, I didn't have much choice. One of the disks in the RAID5 set was a 80G, too, however, with a 40G partition dedicated to the RAID5 set; so rather than doing ugly things with two disks only halfway committed to the RAID1, I decided to convert the RAID5 set to a RAID1 one. When you use LVM-on-RAID, as I do, that's pretty easy: you just create the new array (in degraded mode, on the new disk); add it to the volume group; use pvmove to move all data to the new array; use vgreduce to remove the old array from the volume group; and finally, repurpose the freed-up space to make sure the RAID1 set is no longer degraded.

Which is all nice, except that on 2.6.12, when pvmove locks the entire volume group, you can't write to it anymore, which is quite painful if things like /var and / are on that volume group; gives you a nice deadlock.

So I compiled me a 2.6.19.2 kernel with SMP support explicitly disabled, in order to be able to do the pvmove dance without much issues. That worked. I only forgot to do one rather important thing: to regenerate the initrd after all data had been moved; mainly because it took quite a while for the RAID1 resync to finish, and I had other things to do by then.

Yesterday, we had a power failure, and obviously rock failed to come up again.

By doing some utterly ugly things with the sarge d-i and forcing the yaird-generated initrd into a debug shell, I was able to boot the box properly again, and had it running 2.6.12. Which took all of three hours, or so. By then, it was kinda late, so I went to bed.

This morning, my brother woke me to say he couldn't reach the Internet. Since rock is my gateway, I got suspicious. Running dmesg showed me a number of warnings from the kernel, where it said that it didn't properly understand some of the hashing stuff in some directory inodes. As a result, several files had been corrupted—including /var/lib/dpkg/status and /var/lib/dpkg/status-old. The file with 'extra' information that aptitude maintains, however, /var/lib/aptitude/pkgstates, still existed and was readable. So I downloaded me a one-week-old Packages file from snapshot.debian.net, and wrote this perl script:

#!/usr/bin/perl

open(APTITUDE,"</var/lib/aptitude/pkgstates");
open(STATUS,">status");
open(PKGS,"<Packages");

my $pkgent;
my $aptent;
my %aptitude;
my $i;
my $key;
my $val;

while(<APTITUDE>) {
	if(/^$/) {
		$aptitude{"Package:" . $aptent->{"Package"} . "\n"}=$aptent;
		$aptent = {};
	} else {
		chomp;
		($key,$val) = split /:/;
		$aptent->{$key} = $val;
	}
}

close APTITUDE;

while(<PKGS>) {
	if(/^$/) {
		$aptent = $aptitude{$pkgent->[0]};
		if($aptent->{"State"} == " 1") {
			print STATUS $pkgent->[0];
			print STATUS "Status: install ok installed\n";
			for($i=1;$i<$#{$pkgent};$i++) {
				if(!($pkgent->[$i] =~ /^(Filename|Size|MD5sum)/)) {
					print STATUS $pkgent->[$i];
				}
			}
			print STATUS "\n";
		}
		$pkgent = [];
	} else {
		push @{$pkgent}, $_;
	}
}

close PKGS;
close STATUS;

... which generated a somewhat valid status file, that I copied to its right place. Yeah, that's an ugly hack, but at least I don't have to reinstall rock now. Phew.