ZFS zpool Expansion… – The Stuff We Use

Having moved my entire RAIDZ2 cluster to a bigger drives quite a while ago, I never had the need to expand the entire volume size (yet). But since I was to embark on creating a new RAIDZ3 for some other reason, I decided to first expand the part1s across all six disks (where by RAIDZ2 was sitting astride):

Export the zpool:

zfs export <POOL NAME>

Backup the GPT tables:

mkdir -p /root/gpt-backup

for d in /dev/sd[a-f]; do
    sgdisk --backup="/root/gpt-backup/$(basename "$d").gpt" "$d"
done

Then resize part1 across all 6 disks, based on your expanded ending sector:

for d in /dev/sd[a-f]; do
    sgdisk --delete=1 "$d"
    sgdisk --new=1:2048:20835092399 "$d"
    sgdisk --typecode=1:BF01 "$d"
    sgdisk --change-name=1:<POOL NAME> "$d"
done

Don’t worry: sgdisk re-creating the partition 1 information with the same starting sector and later ending sector simply means your data is still there, but the partition is expanded.

You can now check the “new” boundaries:

sfdisk -l /dev/sd[a-f]

And you can also check that the ZFS labels still exist:

for d in /dev/sd[a-f]1; do
    echo "===== $d ====="
    zdb -l "$d" | grep -E 'name:|pool_guid:|top_guid:|guid:|txg:'
done

You can then proceed to re-import the pool, either with a blanket command below, or using specific IDs:

zpool import -d /dev/disk/by-id <POOL NAME>

Then check:

zpool status -P <POOL NAME>

Ensure the pool is set to auto-expand:

zpool set autoexpand=on <POOL NAME>

And then explicitly set each member vdev to expand:

for d in /dev/sd[a-f]1; do
    zpool online -e <POOL NAME> "$d"
done

And this where all the trouble started…

No matter how many times I did that last two commands, or even exported and reimported, ZFS refused to expand the individual vdevs, despite the “extra space” being acknowledged (by the EXDPANDSZ parameter showing as such)…

# zpool list <POOL>
NAME      SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
<POOL>   32.5T  30.7T  1.85T        -     25.2T    18%    94%  1.00x    ONLINE  -
# zpool list -v <POOL>
NAME                                                  SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
<POOL>                                               32.5T  30.7T  1.85T        -     25.2T    18%    94%  1.00x    ONLINE  -
  raidz2-0                                           32.5T  30.7T  1.85T        -     25.2T    18%  94.3%      -    ONLINE
    ata-<redacted>-part1                             9.70T      -      -        -         -      -      -      -    ONLINE
    ata-<redacted>-part1                             9.70T      -      -        -         -      -      -      -    ONLINE
    ata-<redacted>-part1                             9.70T      -      -        -         -      -      -      -    ONLINE
    ata-<redacted>-part1                             9.70T      -      -        -         -      -      -      -    ONLINE
    ata-<redacted>-part1                             9.70T      -      -        -         -      -      -      -    ONLINE
    ata-<redacted>-part1                             9.70T      -      -        -         -      -      -      -    ONLINE
logs                                                     -      -      -        -         -      -      -      -         -
  nvme-<redacted>-part5                                 4G   536K  3.97G        -         -     0%  0.01%      -    ONLINE

Eventually, this worked i.e. spelling out the individual vdev IDs:

for d in /dev/disk/by-id/ata-<redacted>-part1 \
         /dev/disk/by-id/ata-<redacted>-part1 \
         /dev/disk/by-id/ata-<redacted>-part1 \
         /dev/disk/by-id/ata-<redacted>-part1 \
         /dev/disk/by-id/ata-<redacted>-part1 \
         /dev/disk/by-id/ata-<redacted>-part1; \
         do zpool online -e <POOL NAME> "$d"; done

Seems like just using the lazy /dev/sd[a-f]1 method doesn’t work, may be because it fails to match the registered devices as spelt out by zpool list -v <POOL NAME>.

Go figure.

Leave a Reply Cancel reply