How I recovered from a wrong lvremove

Table of Contents

Have you ever hit Enter on your keyboard and immediately felt the greatest anguish of your life?

It was late in the afternoon, around 17:00. My team and I were doing some routine maintenance on a production environment. We needed to clean up a corrupt VHD, it was supposed to be a standard short five minute job… Instead, it turned into the most stressful hours of my life during which we struggled to recover an entire Storage Repository (SR) after I accidentally deleted all Logical Volumes (LV) in a Volume Group (VG).

Note

In the aforementioned production environment, each VM disk (in the VHD format) is stored as a LV and is exposed to the VM as a block device. A SR is the meta-name for a group of VHD managed by a specific driver, here LVM. One SR is represented by a single VG.

Here’s the story of how a simple syntax error caused a massive outage and how reading the man (and some bash scripts) saved the day by transforming a data loss into a 2 hours service interruption without a single byte of data lost.

The Mistake

The job was simple: remove a single, corrupted Logical Volume.

I executed the command: lvremove VG_NAME LV_NAME. Any reader that once manipulated LVM could spot where the error lies… Usually a lvremove command is written in one of these ways:

lvremove VG_NAME/LV_NAME
lvremove /dev/VG_NAME/LV_NAME

Because I forgot a slash / didn’t specify the whole path, the command’s scope was evaluated much broader than intended, essentially targeting all LVs within that VG.

To make matters worse, I’ve been asked asked for confirmation to delete the management (MGT) LV. I hit n (no) to abort that specific deletion, but instead of exiting, LVM happily proceeded to delete all the other logical volumes in the group without further confirmation.

As I witnessed the terminal print line after line Logical volume "..." successfully removed, my colleagues and I immediately realised what had happened. Production VMs suddenly lost their underlying disks. All VM I/Os halted.

The Panic (and the missing backups)

After a small and contained panic attack, I managed to get control over myself pretty quickly considering what was at stake and that I couldn’t let my team take the stress for me.

My first thought was,

“No problem, LVM automatically backs up metadata.”

Normally, when LVM configuration changes, it saves a backup in /etc/lvm/archive. You can just run a vgcfgrestore and be back in business in minutes.

We checked the archive directory. It was empty. Due to an edge-case in how this specific environment was configured, the standard LVM filesystem archives weren’t there. We needed to find another way.

The Solution: The LVM Metadata Ring Buffer

Since we couldn’t rely on the standard LVM backups, I decided to look at the raw block device. Upon reading the man (yes, the man) I learnt that LVM maintains a circular ring buffer of recent configuration changes in the first ~1MB of the Physical Volume (PV) header. Allowing to essentially revert to any state immediately preceding an issue, let’s say a catastrophic typo.

Note

I wasn’t left handling this alone. In the meantime my colleagues were searching for alternative solutions.

Here is the step-by-step process I used to recover the volumes.

1. Identify the Physical Volume

First, I needed the exact disk backing the PV for the lost VG.

# Identify the device ID backing the SR
pvs | grep VG_NAME
# Result: /dev/disk/by-id/DISK_SERIAL

2. Extracting the Metadata Records

Instead of blindly dumping the entire 1MB header with dd, I used pvck -v to pinpoint the exact offset and size of every metadata record stored in the ring buffer.

Note

I got really lucky here, depending on the version of LVM and some factors, it is possible that pvck doesn’t find the metadata entries!

I made this little bash script to extract each historical state into its own raw file, and simultaneously parsed out the VHDs present in that state into a text file for easy comparison.

pvck -v /dev/disk/by-id/DISK_SERIAL 2>&1 | \
grep 'Found LVM2 metadata record' | \
sed 's/.*offset=\([0-9]*\), size=\([0-9]*\).*/\1 \2/'| \
while read off sz; do
    echo "Offset: $off | Size: $sz | "

    # Dump the specific metadata block
    dd if=/dev/disk/by-id/DISK_SERIAL bs=1 skip=$off count=$sz > metadata_$off.raw

    # Parse the VHDs inside that block for comparison
    strings metadata_$off.raw | grep "VHD-" | sed 's/ {.*//' | \
    sort > vhds_$off.txt
done

Since I needed the entry to be precise to the byte, I used a block size of 1. This was not optimal but I wasn’t in the mood to do some bash arithmetic.

Each LV had a VHD- prefix. I used that to extract the names and count of LVs of each entry quickly.

3. Differential Analysis

Now I had a bunch of text files representing different points in time. We knew from the console output of my disastrous lvremove command exactly how many LVs were deleted (46) and the one that remained (1). Therefore, the pre-incident state should have 47 VHDs.

# 1. Sort by VHD count to find the candidate states.
wc -l vhds_*.txt | sort -rn | grep '47 '

# 2. Verify the difference between our candidate (offset 5775360) and the current broken state
comm -13 vhds_deleted.txt vhds_candidate.txt

# 3. Get the seqno of each candidate state and compare against the very next transaction
strings metadata_candidate.raw | grep seqno # output: 347
strings metadata_*.raw | grep 348

# Check that the difference between the desired state (347) and the first deletion step (348) is exactly 1 VHD
comm -13 vhds_candidate.txt vhds_348.txt

This confirmed we had the exact offset (5775360) right before the deletion started.

Note

In the case of this lvremove command, each LV got deleted one by one. This meant one metadata entry per LV. Luckily there wasn’t enough LV to overflow the ring buffer.

4. The Restoration

With the correct raw metadata file isolated, we converted it into a standard LVM config file and did a dry-run test.

# Convert raw strings to config file
strings metadata_5775360.raw > recovery.vg

# ALWAYS DO A DRY RUN FIRST
vgcfgrestore VG_NAME --test --file recovery.vg

# The real deal
vgcfgrestore VG_NAME --file recovery.vg

5. Service Rescan

Once LVM was aware of the structure again, we rescanned the VG to make sure our storage driver could find each VHD.

/usr/bin/vhd-util scan -f -m VHD-* -l VG_NAME

YES!!! All LVs were restored and 100% of the data was intact. We rebooted the affected VMs and normal operations resumed roughly two hours after the incident started.

Lessons Learned

Prod outages, besides being absolutely terrible are also incredible learning opportunities. Here is what I took away from this one:

  1. RTFM: The solution came from the technical specs and the man. I later found another blog post about this recovery method
  2. Do dry-runs before destructive commands: Doing a dry-run before anything could have prevented us from running the command with the wrong syntax.
  3. Beware of fatigue: It was the end of the day. A simple peer review of the command syntax, or a dry-run flag, would have prevented this entirely. I know I get lenient with commands I use all the time.
  4. Always backup your data: If the LVM archive/backups were properly configured we could have restored to the previous state quickly and make this a matter of minutes.

In the end, we saved the production environment without data loss. It was a stressful two hours, but the custom scripts and expertise gained made it a valuable learning experience.

Handling stress

Watching the command delete each volume one by one triggered an intense wave of stress and a minor panic attack. I somehow managed to overcome it and became strangely calm and hyper-focused during the actual recovery process.

It was only after the call ended that I truly understood the phrase “the silence is deafening.” Sitting alone, drowning in that silence, was the worst feeling of the entire afternoon. Fortunately, I already had dinner plans with some friends, so I rushed outside to get some fresh air and hit the road. I called one of my friends on the way because I really needed to talk it out. I managed to pull myself together later on and actually had a wonderful evening.

Unsurprisingly, I am now the target of constant jokes whenever the topic of removing LVs or deleting data comes up. I honestly deserve it

Go to Top