A little difficult: real ZFS raid
by camerabambai from LinuxQuestions.org on (#5BE6B)
On Solaris 11 is really easy to obtain a real raid ZFS: you can loose one hard disk and the system boot.
This is what I did on Solaris 11
a)Check the partitions
Code:zpool status rpool
formatb)I can copy the partitions
Code:prtvtoc /dev/rdsk/c0t0d0s2 | fmthard -s - /dev/rdsk/c0t1d0s2c)I attach the second disk to the first for mirroring
Code:zpool attach -f rpool c0t0d0s0 c0t1d0s0d)Wait until resilvering is complete
Code:zpool statuse)Grub configure
Code:#on x86 with Solaris =< 11.3
installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c0t1d0s0
#On Sparc
installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/c1t1d0s0
#on x86 with Solaris => 11.4
bootadm install-bootloader /dev/rdsk/c1t1d0s0h)On reboot is OK, works fine.
If you remove one of the disk the system boot without problems.
Now return on Linux, is easy(but long) to install zfs on Debian starting with a livecd, I have successfully boot from a raidz(!!) but with only one problem: the bootpool and rootpool are on raidz so are redundant.
But the efi partition was single! So if you lost the disc you have to boot from a livecd, recreate the efi partition, recreate the grub structures, etc...very boring and not immediate as Solaris did.
So I had an idea: try to booting from a raid efi. I don't know if is possible here is what i did
a)The system is a virtual machine, Linux debian host, Linux Debian guest.
I start with the live-cd with xfce, and two qcow2 disks. Ram is 4gb for guest.
b)I start the livecd and install ssh to control it from a remote
Code:sudo su -
apt -y update
apt -y install openssh-server vim
systemctl start ssh
echo -e 'password\npassword\n' userc)from my shell..
Code:ssh -l user debianzfs
sudo su -d)on livecd we install git python3-pip and the requirements for zfs
Code:apt -y install git python3-pip
git clone https://github.com/openzfs/openzfs-docs
cd openzfs-docs
pip3 install -r docs/requirements.txt
PATH=$HOME/.local/bin:$PATHe)I edit apt
Code:cat > /etc/apt/sources.list <<EOF
deb http://deb.debian.org/debian buster main contrib
deb http://deb.debian.org/debian buster-backports main contrib
EOFf)install zfs on livecd
Code:apt -y update
apt install -y cryptsetup
apt install -y debootstrap gdisk dkms dpkg-dev linux-headers-$(uname -r)
apt install -y -t buster-backports --no-install-recommends zfs-dkms mdadm gdisk
modprobe zfs
apt install -y -t buster-backports zfsutils-linuxg)setup the disks
Code:apt -y install mdadm
mdadm --stop /dev/md*
mdadm --zero-superblock /dev/vda*... etcCode:wipefs -a /dev/vda
wipefs -a /dev/vdb
h)partitioning(I use FD00, but also using EF00 the system fail to boot)
Code:sgdisk -n 1:2048:+512M -t1:FD00 -c1:EFI /dev/vda
sgdisk -n 2:`sgdisk -f`:+1G -t2:BF01 -c2:BOOTPOOL /dev/vda
sgdisk -n 3:`sgdisk -f`:0 -t3:BF00 -c3:ROOTPOOL /dev/vda
sgdisk -n 1:2048:+512M -t1:FD00 -c1:EFI2 /dev/vdb
sgdisk -n 2:`sgdisk -f`:+1G -t2:BF01 -c2:BOOTPOOL2 /dev/vdb
sgdisk -n 3:`sgdisk -f`:0 -t3:BF00 -c3:ROOTPOOL2 /dev/vdbi)creating the bootpool
Code:zpool create -f -o ashift=12 -d -o feature@async_destroy=enabled -o feature@bookmarks=enabled -o feature@embedded_data=enabled -o feature@empty_bpobj=enabled -o feature@enabled_txg=enabled -o feature@extensible_dataset=enabled -o feature@filesystem_limits=enabled -o feature@hole_birth=enabled -o feature@large_blocks=enabled -o feature@lz4_compress=enabled -o feature@spacemap_histogram=enabled -o feature@zpool_checkpoint=enabled -O acltype=posixacl -O canmount=off -O compression=lz4 -O devices=off -O normalization=formD -O relatime=on -O xattr=sa -O mountpoint=/boot -R /mnt bpool mirror /dev/disk/by-partlabel/BOOTPOOL /dev/disk/by-partlabel/BOOTPOOL2m)status control
Code:zpool list
zpool statuso)now I create the rootpool
Code:zpool create \
-o ashift=12 \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R /mnt rpool mirror /dev/disk/by-partlabel/ROOTPOOL /dev/disk/by-partlabel/ROOTPOOL2p)creating zfs filesets
Code:zfs create -o canmount=off -o mountpoint=none rpool/ROOT
zfs create -o canmount=off -o mountpoint=none bpool/BOOT
zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/debian
zfs mount rpool/ROOT/debian
zfs create -o mountpoint=/boot bpool/BOOT/debian
zfs create rpool/home
zfs create -o mountpoint=/root rpool/home/root
zfs create -o canmount=off rpool/var
zfs create -o canmount=off rpool/var/lib
zfs create rpool/var/log
zfs create rpool/var/spoolCode:zfs create -o com.sun:auto-snapshot=false rpool/var/cache
zfs create -o com.sun:auto-snapshot=false rpool/var/tmp
chmod 1777 /mnt/var/tmpCode:zfs create rpool/opt
zfs create -o canmount=off rpool/usr
zfs create rpool/usr/local
zfs create rpool/var/snap
zfs create -o com.sun:auto-snapshot=false rpool/var/lib/nfs
zfs create -o com.sun:auto-snapshot=false rpool/tmp
chmod 1777 /mnt/tmps)installing Debian on mnt
Code:debootstrap buster /mntt)I have a working dns, so no need to edit etc/hosts, etc...
u)configure apt/sources on mnt
Code:cat > /mnt/etc/apt/sources.list<<EOF
deb http://deb.debian.org/debian buster main contrib
deb-src http://deb.debian.org/debian buster main contrib
deb http://security.debian.org/debian-security buster/updates main contrib
deb-src http://security.debian.org/debian-security buster/updates main contrib
deb http://deb.debian.org/debian buster-updates main contrib
deb-src http://deb.debian.org/debian buster-updates main contrib
EOF
cat > /mnt/etc/apt/sources.list.d/buster-backports.list <<EOF
deb http://deb.debian.org/debian buster-backports main contrib
deb-src http://deb.debian.org/debian buster-backports main contrib
EOF
cat > /mnt/etc/apt/preferences.d/90_zfs <<EOF
Package: libnvpair1linux libuutil1linux libzfs2linux libzfslinux-dev libzpool2linux python3-pyzfs pyzfs-doc spl spl-dkms zfs-dkms zfs-dracut zfs-initramfs zfs-test zfsutils-linux zfsutils-linux-dev zfs-zed
Pin: release n=buster-backports
Pin-Priority: 990
EOFv)chroot!
Code:mount --rbind /dev /mnt/dev
mount --rbind /proc /mnt/proc
mount --rbind /sys /mnt/sys
chroot /mnt bash --loginz)on chroot I install some packages
Code:ln -s /proc/self/mounts /etc/mtab
apt -y update
apt -y install console-setup locales vim bash-completion network-manager apt-filez1)I reconfigure some of them..
Code:dpkg-reconfigure locales tzdata keyboard-configuration console-setupz2)I install zfs on chroot
Code:apt install --yes dpkg-dev linux-headers-amd64 linux-image-amd64 mdadm
apt install --yes zfs-initramfs
echo REMAKE_INITRD=yes > /etc/dkms/zfs.confz3)installing grub, mdadm, configure efi for fstab, etc
Code:apt -y install dosfstools mdadm
mdadm --stop /dev/md127
mdadm --create /dev/md0 --level=1 --name debianzfs:0 --raid-devices=2 /dev/disk/by-partlabel/EFI /dev/disk/by-partlabel/EFI2
mkdosfs -F 32 -s 1 -n EFI /dev/md0
mkdir /boot/efi
echo /dev/disk/by-label/EFI \
/boot/efi vfat nofail,x-systemd.device-timeout=1 0 1 >> /etc/fstab
mount /boot/efi
apt install --yes grub-efi-amd64 shim-signedz4)I remove osprober
Code:dpkg --purge os-proberz5)I set the root password on chroot
Code:echo -e 'yourpass\nyourpass\n'|passwd rootz6)I create the zfs-import service for systemd
Code:cat >/etc/systemd/system/zfs-import-bpool.service <<EOF
[Unit]
DefaultDependencies=no
Before=zfs-import-scan.service
Before=zfs-import-cache.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/zpool import -N -o cachefile=none bpool
[Install]
WantedBy=zfs-import.target
EOF
systemctl enable zfs-import-bpool.service
z8)we need some modules for raid and efi
Code:vim /etc/initramfs-tools/modules
raid1
fat
vfatz9)reconfigure grub and initramfs, and mdadm.conf
grub-probe /boot
Code:vim /etc/default/grub
GRUB_CMDLINE_LINUX="rd.md.uuid=4e9080bf:37029179:6f75e264:95e69e28 root=ZFS=rpool/ROOT/debian mds=full,nosmt mitigations=auto console=tty1 console=ttyS0,115200 quiet"
GRUB_TERMINAL="console serial"Code:vim /etc/mdadm/mdadm.conf
ARRAY /dev/md/0 metadata=1.2 UUID=4e9080bf:37029179:6f75e264:95e69e28 name=debianzfs:0
update-initramfs -c -k all
update-grubz9)now install grub on both disks of raid1
Code:for i in vda vdb ;do grub-install --removable --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=debian --recheck --no-floppy /dev/$i;donez10)configure zed
Code:mkdir /etc/zfs/zfs-list.cache
touch /etc/zfs/zfs-list.cache/bpool
touch /etc/zfs/zfs-list.cache/rpool
ln -s /usr/lib/zfs-linux/zed.d/history_event-zfs-list-cacher.sh /etc/zfs/zed.d
zed -F &Code:zfs set canmount=on bpool/BOOT/debian
zfs set canmount=noauto rpool/ROOT/debian
cat /etc/zfs/zfs-list.cache/bpool
cat /etc/zfs/zfs-list.cache/rpoolz12)ONLY if the cat bpool and cat rpool on z11 return something(not empty lines) terminate zed
Code:%
CTRL+Cz13)remove /mnt from cache files
Code:sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/*z14)check if mnt is removed
Code:cat /etc/zfs/zfs-list.cache/bpool
cat /etc/zfs/zfs-list.cache/rpoolz14)installing openssh on chroot
Code:apt -y install openssh-serverz15)create the startup.nsh..
Code:cat > /boot/efi/startup.nsh <<EOF
\EFI\debian\grubx64.efi
EOFz16)finally we can reboot..
Code:exit
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \
xargs -i{} umount -lf {}
zpool export -a
rebootbut the result is a little bad..
https://images2.imgbox.com/cb/f2/SR7SBZCV_o.png
What in your opinion doesn't work? The firmware efi cannot read the md0?
This is what I did on Solaris 11
a)Check the partitions
Code:zpool status rpool
formatb)I can copy the partitions
Code:prtvtoc /dev/rdsk/c0t0d0s2 | fmthard -s - /dev/rdsk/c0t1d0s2c)I attach the second disk to the first for mirroring
Code:zpool attach -f rpool c0t0d0s0 c0t1d0s0d)Wait until resilvering is complete
Code:zpool statuse)Grub configure
Code:#on x86 with Solaris =< 11.3
installgrub /boot/grub/stage1 /boot/grub/stage2 /dev/rdsk/c0t1d0s0
#On Sparc
installboot -F zfs /usr/platform/`uname -i`/lib/fs/zfs/bootblk /dev/rdsk/c1t1d0s0
#on x86 with Solaris => 11.4
bootadm install-bootloader /dev/rdsk/c1t1d0s0h)On reboot is OK, works fine.
If you remove one of the disk the system boot without problems.
Now return on Linux, is easy(but long) to install zfs on Debian starting with a livecd, I have successfully boot from a raidz(!!) but with only one problem: the bootpool and rootpool are on raidz so are redundant.
But the efi partition was single! So if you lost the disc you have to boot from a livecd, recreate the efi partition, recreate the grub structures, etc...very boring and not immediate as Solaris did.
So I had an idea: try to booting from a raid efi. I don't know if is possible here is what i did
a)The system is a virtual machine, Linux debian host, Linux Debian guest.
I start with the live-cd with xfce, and two qcow2 disks. Ram is 4gb for guest.
b)I start the livecd and install ssh to control it from a remote
Code:sudo su -
apt -y update
apt -y install openssh-server vim
systemctl start ssh
echo -e 'password\npassword\n' userc)from my shell..
Code:ssh -l user debianzfs
sudo su -d)on livecd we install git python3-pip and the requirements for zfs
Code:apt -y install git python3-pip
git clone https://github.com/openzfs/openzfs-docs
cd openzfs-docs
pip3 install -r docs/requirements.txt
PATH=$HOME/.local/bin:$PATHe)I edit apt
Code:cat > /etc/apt/sources.list <<EOF
deb http://deb.debian.org/debian buster main contrib
deb http://deb.debian.org/debian buster-backports main contrib
EOFf)install zfs on livecd
Code:apt -y update
apt install -y cryptsetup
apt install -y debootstrap gdisk dkms dpkg-dev linux-headers-$(uname -r)
apt install -y -t buster-backports --no-install-recommends zfs-dkms mdadm gdisk
modprobe zfs
apt install -y -t buster-backports zfsutils-linuxg)setup the disks
Code:apt -y install mdadm
mdadm --stop /dev/md*
mdadm --zero-superblock /dev/vda*... etcCode:wipefs -a /dev/vda
wipefs -a /dev/vdb
h)partitioning(I use FD00, but also using EF00 the system fail to boot)
Code:sgdisk -n 1:2048:+512M -t1:FD00 -c1:EFI /dev/vda
sgdisk -n 2:`sgdisk -f`:+1G -t2:BF01 -c2:BOOTPOOL /dev/vda
sgdisk -n 3:`sgdisk -f`:0 -t3:BF00 -c3:ROOTPOOL /dev/vda
sgdisk -n 1:2048:+512M -t1:FD00 -c1:EFI2 /dev/vdb
sgdisk -n 2:`sgdisk -f`:+1G -t2:BF01 -c2:BOOTPOOL2 /dev/vdb
sgdisk -n 3:`sgdisk -f`:0 -t3:BF00 -c3:ROOTPOOL2 /dev/vdbi)creating the bootpool
Code:zpool create -f -o ashift=12 -d -o feature@async_destroy=enabled -o feature@bookmarks=enabled -o feature@embedded_data=enabled -o feature@empty_bpobj=enabled -o feature@enabled_txg=enabled -o feature@extensible_dataset=enabled -o feature@filesystem_limits=enabled -o feature@hole_birth=enabled -o feature@large_blocks=enabled -o feature@lz4_compress=enabled -o feature@spacemap_histogram=enabled -o feature@zpool_checkpoint=enabled -O acltype=posixacl -O canmount=off -O compression=lz4 -O devices=off -O normalization=formD -O relatime=on -O xattr=sa -O mountpoint=/boot -R /mnt bpool mirror /dev/disk/by-partlabel/BOOTPOOL /dev/disk/by-partlabel/BOOTPOOL2m)status control
Code:zpool list
zpool statuso)now I create the rootpool
Code:zpool create \
-o ashift=12 \
-O acltype=posixacl -O canmount=off -O compression=lz4 \
-O dnodesize=auto -O normalization=formD -O relatime=on \
-O xattr=sa -O mountpoint=/ -R /mnt rpool mirror /dev/disk/by-partlabel/ROOTPOOL /dev/disk/by-partlabel/ROOTPOOL2p)creating zfs filesets
Code:zfs create -o canmount=off -o mountpoint=none rpool/ROOT
zfs create -o canmount=off -o mountpoint=none bpool/BOOT
zfs create -o canmount=noauto -o mountpoint=/ rpool/ROOT/debian
zfs mount rpool/ROOT/debian
zfs create -o mountpoint=/boot bpool/BOOT/debian
zfs create rpool/home
zfs create -o mountpoint=/root rpool/home/root
zfs create -o canmount=off rpool/var
zfs create -o canmount=off rpool/var/lib
zfs create rpool/var/log
zfs create rpool/var/spoolCode:zfs create -o com.sun:auto-snapshot=false rpool/var/cache
zfs create -o com.sun:auto-snapshot=false rpool/var/tmp
chmod 1777 /mnt/var/tmpCode:zfs create rpool/opt
zfs create -o canmount=off rpool/usr
zfs create rpool/usr/local
zfs create rpool/var/snap
zfs create -o com.sun:auto-snapshot=false rpool/var/lib/nfs
zfs create -o com.sun:auto-snapshot=false rpool/tmp
chmod 1777 /mnt/tmps)installing Debian on mnt
Code:debootstrap buster /mntt)I have a working dns, so no need to edit etc/hosts, etc...
u)configure apt/sources on mnt
Code:cat > /mnt/etc/apt/sources.list<<EOF
deb http://deb.debian.org/debian buster main contrib
deb-src http://deb.debian.org/debian buster main contrib
deb http://security.debian.org/debian-security buster/updates main contrib
deb-src http://security.debian.org/debian-security buster/updates main contrib
deb http://deb.debian.org/debian buster-updates main contrib
deb-src http://deb.debian.org/debian buster-updates main contrib
EOF
cat > /mnt/etc/apt/sources.list.d/buster-backports.list <<EOF
deb http://deb.debian.org/debian buster-backports main contrib
deb-src http://deb.debian.org/debian buster-backports main contrib
EOF
cat > /mnt/etc/apt/preferences.d/90_zfs <<EOF
Package: libnvpair1linux libuutil1linux libzfs2linux libzfslinux-dev libzpool2linux python3-pyzfs pyzfs-doc spl spl-dkms zfs-dkms zfs-dracut zfs-initramfs zfs-test zfsutils-linux zfsutils-linux-dev zfs-zed
Pin: release n=buster-backports
Pin-Priority: 990
EOFv)chroot!
Code:mount --rbind /dev /mnt/dev
mount --rbind /proc /mnt/proc
mount --rbind /sys /mnt/sys
chroot /mnt bash --loginz)on chroot I install some packages
Code:ln -s /proc/self/mounts /etc/mtab
apt -y update
apt -y install console-setup locales vim bash-completion network-manager apt-filez1)I reconfigure some of them..
Code:dpkg-reconfigure locales tzdata keyboard-configuration console-setupz2)I install zfs on chroot
Code:apt install --yes dpkg-dev linux-headers-amd64 linux-image-amd64 mdadm
apt install --yes zfs-initramfs
echo REMAKE_INITRD=yes > /etc/dkms/zfs.confz3)installing grub, mdadm, configure efi for fstab, etc
Code:apt -y install dosfstools mdadm
mdadm --stop /dev/md127
mdadm --create /dev/md0 --level=1 --name debianzfs:0 --raid-devices=2 /dev/disk/by-partlabel/EFI /dev/disk/by-partlabel/EFI2
mkdosfs -F 32 -s 1 -n EFI /dev/md0
mkdir /boot/efi
echo /dev/disk/by-label/EFI \
/boot/efi vfat nofail,x-systemd.device-timeout=1 0 1 >> /etc/fstab
mount /boot/efi
apt install --yes grub-efi-amd64 shim-signedz4)I remove osprober
Code:dpkg --purge os-proberz5)I set the root password on chroot
Code:echo -e 'yourpass\nyourpass\n'|passwd rootz6)I create the zfs-import service for systemd
Code:cat >/etc/systemd/system/zfs-import-bpool.service <<EOF
[Unit]
DefaultDependencies=no
Before=zfs-import-scan.service
Before=zfs-import-cache.service
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/sbin/zpool import -N -o cachefile=none bpool
[Install]
WantedBy=zfs-import.target
EOF
systemctl enable zfs-import-bpool.service
z8)we need some modules for raid and efi
Code:vim /etc/initramfs-tools/modules
raid1
fat
vfatz9)reconfigure grub and initramfs, and mdadm.conf
grub-probe /boot
Code:vim /etc/default/grub
GRUB_CMDLINE_LINUX="rd.md.uuid=4e9080bf:37029179:6f75e264:95e69e28 root=ZFS=rpool/ROOT/debian mds=full,nosmt mitigations=auto console=tty1 console=ttyS0,115200 quiet"
GRUB_TERMINAL="console serial"Code:vim /etc/mdadm/mdadm.conf
ARRAY /dev/md/0 metadata=1.2 UUID=4e9080bf:37029179:6f75e264:95e69e28 name=debianzfs:0
update-initramfs -c -k all
update-grubz9)now install grub on both disks of raid1
Code:for i in vda vdb ;do grub-install --removable --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=debian --recheck --no-floppy /dev/$i;donez10)configure zed
Code:mkdir /etc/zfs/zfs-list.cache
touch /etc/zfs/zfs-list.cache/bpool
touch /etc/zfs/zfs-list.cache/rpool
ln -s /usr/lib/zfs-linux/zed.d/history_event-zfs-list-cacher.sh /etc/zfs/zed.d
zed -F &Code:zfs set canmount=on bpool/BOOT/debian
zfs set canmount=noauto rpool/ROOT/debian
cat /etc/zfs/zfs-list.cache/bpool
cat /etc/zfs/zfs-list.cache/rpoolz12)ONLY if the cat bpool and cat rpool on z11 return something(not empty lines) terminate zed
Code:%
CTRL+Cz13)remove /mnt from cache files
Code:sed -Ei "s|/mnt/?|/|" /etc/zfs/zfs-list.cache/*z14)check if mnt is removed
Code:cat /etc/zfs/zfs-list.cache/bpool
cat /etc/zfs/zfs-list.cache/rpoolz14)installing openssh on chroot
Code:apt -y install openssh-serverz15)create the startup.nsh..
Code:cat > /boot/efi/startup.nsh <<EOF
\EFI\debian\grubx64.efi
EOFz16)finally we can reboot..
Code:exit
mount | grep -v zfs | tac | awk '/\/mnt/ {print $3}' | \
xargs -i{} umount -lf {}
zpool export -a
rebootbut the result is a little bad..
https://images2.imgbox.com/cb/f2/SR7SBZCV_o.png
What in your opinion doesn't work? The firmware efi cannot read the md0?