LVM Rescue Procedure

Rule number one: Don't Panic!. Think twice before you do anything. Rule number two: If you did panic, there is still hope.

Issue

Add a bunch of new disks your box while it is running. In order to add them to your LVM volume group, edit /etc/lvm.conf to allow the new devices to be picked up. All goes well. Create two new Volumes.

Now for some reason the old version of /etc/lvm.conf gets restored. Nothing happens until the next reboot, when LVM can not re-assemble all the disks for the volume group.

Don't try this on a real system, unless things look really bad! Don't come whining when you destroy your gear.

Recovery instructions

If you realize that /etc/lvm.conf is not picking up all the disks, you are already on the winning side. Just edit the file, and run lvmdiskscan to make lvm check again. If everything is back where it should be, run lvscan to see if any of the volumes are inactive. Reactivate them with:

lvchange -a y /dev/volume_group/lv_name

If you have already destroyed some config by stroking pvcreate the wrong way, there is still hope. Go to /etc/lvm/archive/ and find the most recent backup of lvm.conf. The backups are plain text! Before you do anything else though, make sure that all disks are visible now! With this established, read man vgcfgrestore to restore the vg configuration to its former glory.

pvcreate  --restorefile /etc/lvm/archive/filename --uuid uuid
vgcfgrestore --file /etc/lvm/archive/filename

It may be, that you are still missing one logical volume. The LVM commands create an automatic backup prior to any configuration changes. This means that there is no backup of the configuration AFTER the last change. So how to get that last volume back. If you look at the last backup file, you will see the command that was used to create the last volume. The good news is, that you can issue this same command with the -Zn option to get your volume back (YES READ THE MANUAL PAGE!). LVM seem to be pretty predictable in the way it creates new volumes. Obviously this will only work, when nothing else has been changed.

lvcreate --size 20G -Zn --name lv_name volume_group
fsck -f /dev/volume_group/lv_name

And yes, this is not just an academic example. My adrenalin is still pumping.

NOTE: The content of this website is accessible with any browser. The graphical design though relies completely on CSS2 styles. If you see this text, this means that your browser does not support CSS2. Consider upgrading to a standard conformant browser like Mozilla Firefox, Opera, Safari or Konqueror for example.