I’m a ZFS on Root guy, so I get to be my own support on most days. Today, I wanted to see how hard common tasks that I once did on #FreeBSD with ZFS tools would be. Namely, I wanted to pluck out a 250G drive from my Root Pool and replace it with a 1TB drive, live, with a minimal of fuss.
I think I was successful, and it wasn’t that hard. I did have to teach myself a bit about how sgdisk(8) works and what not to do, but in the end I came up with the following bit of notes that seemed to work for me. A lot of my previous ZFS experience was applicable here, but I have spent decades being oblivious to the magic of Grub and boot loaders on x86 Linux ports.
Thank you to the fine folks over at OpenZFS for having a VERY easy to follow 22.04 install guide, which I definitely stole things from.
The following is just my notes and if you see some things that will bite me, let me know on Mastodon, you can find me over at infosec.exchange
# sgdisk -p /dev/sda
Disk /dev/sda: 781422768 sectors, 372.6 GiB
Model: MK0400GCTZA
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): DFE9708A-6076-4E24-B683-64A286A1E751
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 781422734
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)
Number Start (sector) End (sector) Size Code Name
1 2048 1050623 512.0 MiB EF00 EFI system partition
2 1050624 5244927 2.0 GiB 8200 Linux swap
3 5244928 9439231 2.0 GiB BE00 Solaris boot
4 9439232 781422734 368.1 GiB BF00 Solaris root
# sgdisk -p /dev/sdb
Disk /dev/sdb: 537234768 sectors, 256.2 GiB
Model: Crucial_CT275MX3
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): AC04D23F-E1FD-4B88-9880-B9679DCF4BBD
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 537234734
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)
Number Start (sector) End (sector) Size Code Name
1 2048 1050623 512.0 MiB EF00 EFI System Partition
2 1050624 5244927 2.0 GiB 8200
3 5244928 9439231 2.0 GiB BE00
4 9439232 537234734 251.7 GiB BF00
---------------------------------------------------------------------------------
# sgdisk -d 1 /dev/nvme0n1
# sgdisk -n 1:2048:1050623 -t 1:ef00 -c1:"EFI system partition" /dev/nvme0n1
# sgdisk -n 2:1050624:5244927 -t 2:8200 -c2:"Linux swap" /dev/nvme0n1
# sgdisk -n 3:5244928:9439231 -t 3:be00 -c3:"Solaris boot" /dev/nvme0n1
# sgdisk -N 4 -t 4:bf00 -c4:"Solaris root" /dev/nvme0n1
# sgdisk -p /dev/nvme0n1
Disk /dev/nvme0n1: 500118192 sectors, 238.5 GiB
Model: WDC WDS256G1X0C-00ENX0
Sector size (logical/physical): 512/512 bytes
Disk identifier (GUID): F2E3AFF0-5C0C-4A03-BE8C-83EC2DEDB914
Partition table holds up to 128 entries
Main partition table begins at sector 2 and ends at sector 33
First usable sector is 34, last usable sector is 500118158
Partitions will be aligned on 2048-sector boundaries
Total free space is 2014 sectors (1007.0 KiB)
Number Start (sector) End (sector) Size Code Name
1 2048 1050623 512.0 MiB EF00 EFI system partition
2 1050624 5244927 2.0 GiB 8200 Linux swap
3 5244928 9439231 2.0 GiB BE00 Solaris boot
4 9439232 500118158 234.0 GiB BF00 Solaris root
--------------------------------------------------------------------------------
# cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md127 : active raid1 sdb2[0] sda2[1]
2094080 blocks super 1.2 [2/2] [UU]
unused devices: <none>
root@hattie:~# mdadm --manage /dev/md127 --fail /dev/sda2
mdadm: set /dev/sda2 faulty in /dev/md127
root@hattie:~# mdadm --manage /dev/md127 --remove /dev/sda2
mdadm: hot removed /dev/sda2 from /dev/md127
root@hattie:~# cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md127 : active raid1 sdb2[0]
ex 2094080 blocks super 1.2 [2/1] [U_]
unused devices: <none>
# mdadm --manage /dev/md127 --add /dev/nvme0n1p2
mdadm: added /dev/nvme0n1p2
root@hattie:~# cat /proc/mdstat
Personalities : [raid1] [linear] [multipath] [raid0] [raid6] [raid5] [raid4] [raid10]
md127 : active raid1 nvme0n1p2[2] sdb2[0]
2094080 blocks super 1.2 [2/1] [U_]
[===>.................] recovery = 18.7% (392832/2094080) finish=0.2min speed=130944K/sec
unused devices: <none>
-------------------------------------------------------------------------------------
# zpool status bpool
pool: bpool
state: ONLINE
scan: scrub repaired 0B in 00:00:01 with 0 errors on Sun Jul 2 00:00:02 2023
config:
NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
2ae2089b-40b0-1944-9c6d-a436c8ba3295 ONLINE 0 0 0
b2bbd1e6-7b68-4d90-bf9c-f6d6bb13daac ONLINE 0 0 0
# ls -l /dev/disk/by-partuuid/
total 0
lrwxrwxrwx 1 root root 10 Jul 4 12:59 0240b02b-5a83-45f7-b209-de850d66cc48 -> ../../sda4
lrwxrwxrwx 1 root root 10 Jul 4 13:01 26050a01-81fa-df4c-bb64-b825072a4cb5 -> ../../sdb2
lrwxrwxrwx 1 root root 15 Jul 4 13:04 277066fd-6087-45d3-bc21-36e2be379a7a -> ../../nvme0n1p2
lrwxrwxrwx 1 root root 10 Jul 4 13:01 2ae2089b-40b0-1944-9c6d-a436c8ba3295 -> ../../sdb3
lrwxrwxrwx 1 root root 15 Jul 4 13:00 76e8a1b2-3877-4732-8b6b-2c4f47615733 -> ../../nvme0n1p4
lrwxrwxrwx 1 root root 10 Jul 4 12:59 83437572-9020-4945-8354-47d99f2de483 -> ../../sda2
lrwxrwxrwx 1 root root 15 Jul 4 13:00 86c776d9-9aa6-47de-8d45-26dc8833cea8 -> ../../nvme0n1p3
lrwxrwxrwx 1 root root 10 Jul 4 13:01 8a0ac60b-20ed-4bde-a27d-dd4f4160fa29 -> ../../sdb1
lrwxrwxrwx 1 root root 15 Jul 4 13:00 a963a8f4-3765-477c-bc08-6ea53f71b22b -> ../../nvme0n1p1
lrwxrwxrwx 1 root root 10 Jul 4 12:59 b2bbd1e6-7b68-4d90-bf9c-f6d6bb13daac -> ../../sda3
lrwxrwxrwx 1 root root 10 Jul 4 12:59 d8132cf7-b5f7-4721-8340-3ce42d4b918e -> ../../sda1
lrwxrwxrwx 1 root root 10 Jul 4 13:01 e70d10dd-f1bd-f345-b5a9-e285acd26d8a -> ../../sdb4
# zpool replace bpool /dev/disk/by-partuuid/b2bbd1e6-7b68-4d90-bf9c-f6d6bb13daac /dev/disk/by-partuuid/86c776d9-9aa6-47de-8d45-26dc8833cea8
# zpool status
pool: bpool
state: ONLINE
status: One or more devices is currently being resilvered. The pool will
continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
scan: resilver in progress since Tue Jul 4 13:08:52 2023
305M scanned at 102M/s, 125M issued at 41.6M/s, 305M total
121M resilvered, 40.89% done, 00:00:04 to go
config:
NAME STATE READ WRITE CKSUM
bpool ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
2ae2089b-40b0-1944-9c6d-a436c8ba3295 ONLINE 0 0 0
replacing-1 ONLINE 0 0 0
b2bbd1e6-7b68-4d90-bf9c-f6d6bb13daac ONLINE 0 0 0
86c776d9-9aa6-47de-8d45-26dc8833cea8 ONLINE 0 0 0 (resilvering)
errors: No known data errors
-------------------------------------------------------------------------------------------------
~# update-initramfs -c -k all
update-initramfs: Generating /boot/initrd.img-5.19.0-45-generic
cp: cannot stat '/lib/firmware/updates/amdgpu': No such file or directory
W: Possible missing firmware /lib/firmware/amdgpu/ip_discovery.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/vega10_cap.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/sienna_cichlid_cap.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/navi12_cap.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/psp_13_0_10_sos.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/aldebaran_cap.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/aldebaran_sjt_mec2.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/aldebaran_sjt_mec.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_3_imu.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_3_rlc.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_3_mec.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_3_me.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_3_pfp.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_0_toc.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/sdma_6_0_3.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/sienna_cichlid_mes1.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/sienna_cichlid_mes.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/navi10_mes.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_3_mes1.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_3_mes.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/smu_13_0_10.bin for module amdgpu
I: The initramfs will attempt to resume from /dev/md127
I: (UUID=514a7d66-c1d7-41c2-86e1-b072f30e2dd6)
I: Set the RESUME variable to override this.
update-initramfs: Generating /boot/initrd.img-5.19.0-46-generic
cp: cannot stat '/lib/firmware/updates/amdgpu': No such file or directory
W: Possible missing firmware /lib/firmware/amdgpu/ip_discovery.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/vega10_cap.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/sienna_cichlid_cap.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/navi12_cap.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/psp_13_0_10_sos.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/aldebaran_cap.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/aldebaran_sjt_mec2.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/aldebaran_sjt_mec.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_3_imu.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_3_rlc.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_3_mec.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_3_me.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_3_pfp.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_0_toc.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/sdma_6_0_3.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/sienna_cichlid_mes1.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/sienna_cichlid_mes.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/navi10_mes.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_3_mes1.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/gc_11_0_3_mes.bin for module amdgpu
W: Possible missing firmware /lib/firmware/amdgpu/smu_13_0_10.bin for module amdgpu
I: The initramfs will attempt to resume from /dev/md127
I: (UUID=514a7d66-c1d7-41c2-86e1-b072f30e2dd6)
I: Set the RESUME variable to override this.
--------------------------------------------------------------------------------------
# grub-install --target=x86_64-efi --efi-directory=/boot/efi \
--bootloader-id=ubuntu --recheck --no-floppy
Installing for x86_64-efi platform.
Installation finished. No error reported.