Using NBD to get rid of an LVM installation.

LVM can be a virtue. However, it can also be a pain at times; for instance, there's loss of diskspace due to LVM's metadata that needs to be stored as well. This amounts to approximately 5% of the diskspace (if I'm not mistaken), which can be quite a bit. Also, this is an extra abstraction layer, which requires memory and CPU time to be processed. This will be negligable on most modern computers, but sometimes it isn't.

I had a system that had quite a few things on LVM, and wanted to get rid of them. Unfortunately, to remove the LVM physical volumes would mean that I'd have to get rid of the data, which would mean storing it somewhere else, which in turn would probably mean that I wouldn't be able to use the system for quite a long time. That wasn't acceptable.

So, instead, what I did was to export a whole hard disk (one with enough space to hold the logical volumes on that machine) using NBD, and migrate stuff that way:

  • First, set up the NBD export. The config file should look something like this:
    • [generic]
	user = nbd
	group = disk
[export]
	exportname = /dev/hde
	port = 1234

    This will export all of /dev/hde on port 1234. Obviously you wouldn't want to do this on an untrusted network, but that's always true with NBD (in my case, the link was a 1M crossed cable -- quite safe).

  • Next, we'll connect the client:
    • nbd-client <server> 1234 /dev/nbd0

    ... assuming /dev/nbd0 is not in use yet. Provided the client machine is running kernel 2.6.26 or newer and nbd-client 2.9.10 or later (e.g., Debian Lenny or later), and you're running udev, this will create a number of /dev/nbd0pX device nodes, with X starting at 1, one for each partition on the disk. Of course, one can also export a specific partition rather than the whole disk, or just one file on a filesystem, or create a physical volume of the entire disk rather than just a partition -- your choice.

  • Now we need to add the networked block device to the LVM volume group. Assuming the second partition is the one we want to use:
    • pvcreate /dev/nbd0p2
vgextend /dev/<vg> /dev/nbd0p2

    If you now run 'vgdisplay', you'll see that the size of the LVM volume has increased considerably.

  • Next, we'll want to move data over the network. Assuming the 'local' LVM volume is on /dev/sda7:
    • pvmove -v /dev/sda7

    This will show progress information every 15 seconds while the data is being moved from the local hard disk to the NBD device. You'll be able to continue using the system as usual--although of course the system will gradually get slower and slower as more and more data is only available through a comparatively slow network connection.

  • Once everything's moved to the other sytem, you're all set to change stuff locally:
    • vgreduce -a /dev/<vg>
cfdisk /dev/sda
...
  • Finally, you'll need to move the data from the NBD device back to the local hard disk. If you're still using LVM, that can be done using another pvmove operation. If not, you'll have to do something else, like cp -a. Obviously in that case, there will be some time in which you won't be able to use the system properly.

This final step might not be as simple as it sounds; e.g., if your /home filesystem is on LVM currently and you want to move that out of LVM, you'll have to make sure only root is logged on; If /var is on LVM, too, you may have to go to single-user; etc. Figuring out the details for this one is left as an exercise to the reader.

However, I will note that since somewhere between etch and lenny, doing root-on-NBD is possible; and if you're running unstable, then doing root-on-LVM-on-NBD should be possible, too, although I would suggest a little caution before trying that.

In any case, using NBD in this way just saved me a whole lot of no-computer-time here. Which is great.