Btrfs: Mount failures at boot

I have a system running Tumbleweed that recently received a ton of updates (2800+ ... "Back that truck up over here!"). Part of the disk space on this system is configured as btrfs with subvolumes used to organize the space. Btrfs has been part of this system since it was built a couple of years ago and has been trouble-free up until this round of updates. I should note that there were updates to btrfs-related packages in this recent batch.

Anyway, following these updates and the mandatory reboot, I had the following error messages appear in the "journalctl -xb" output:
Code:Sep 04 14:14:36 klaatu kernel: Btrfs loaded, crc32c=crc32c-intel, assert=on
Sep 04 14:14:36 klaatu kernel: BTRFS: device label localdata devid 1 transid 1257061 /dev/sdb scanned by mount (611)
Sep 04 14:14:36 klaatu kernel: BTRFS info (device sdb): disk space caching is enabled
Sep 04 14:14:36 klaatu kernel: BTRFS info (device sdb): has skinny extents
Sep 04 14:14:36 klaatu kernel: BTRFS error (device sdb): devid 2 uuid aa68d2e2-d78a-4bde-8fbe-858d48c26f8a is missing
Sep 04 14:14:36 klaatu kernel: BTRFS error (device sdb): failed to read the system array: -2
Sep 04 14:14:36 klaatu kernel: BTRFS error (device sdb): open_ctree failed
Sep 04 14:14:36 klaatu kernel: BTRFS info (device sdb): disk space caching is enabled
Sep 04 14:14:36 klaatu kernel: BTRFS info (device sdb): has skinny extents
Sep 04 14:14:36 klaatu kernel: BTRFS error (device sdb): devid 2 uuid aa68d2e2-d78a-4bde-8fbe-858d48c26f8a is missing
Sep 04 14:14:36 klaatu kernel: BTRFS error (device sdb): failed to read the system array: -2
Sep 04 14:14:36 klaatu kernel: BTRFS error (device sdb): open_ctree failed
Sep 04 14:14:36 klaatu kernel: BTRFS info (device sdb): disk space caching is enabled
Sep 04 14:14:36 klaatu kernel: BTRFS info (device sdb): has skinny extents
Sep 04 14:14:36 klaatu kernel: BTRFS error (device sdb): devid 2 uuid aa68d2e2-d78a-4bde-8fbe-858d48c26f8a is missing
Sep 04 14:14:36 klaatu kernel: BTRFS error (device sdb): failed to read the system array: -2
Sep 04 14:14:36 klaatu kernel: BTRFS error (device sdb): open_ctree failed
Sep 04 14:14:36 klaatu kernel: BTRFS info (device sdb): disk space caching is enabled
Sep 04 14:14:36 klaatu kernel: BTRFS info (device sdb): has skinny extents
Sep 04 14:14:36 klaatu kernel: BTRFS error (device sdb): devid 2 uuid aa68d2e2-d78a-4bde-8fbe-858d48c26f8a is missing
Sep 04 14:14:36 klaatu kernel: BTRFS error (device sdb): failed to read the system array: -2
Sep 04 14:14:36 klaatu kernel: BTRFS error (device sdb): open_ctree failed
Sep 04 14:14:36 klaatu kernel: BTRFS info (device sdb): disk space caching is enabled
Sep 04 14:14:36 klaatu kernel: BTRFS info (device sdb): has skinny extents
Sep 04 14:14:36 klaatu kernel: BTRFS error (device sdb): devid 2 uuid aa68d2e2-d78a-4bde-8fbe-858d48c26f8a is missing
Sep 04 14:14:36 klaatu kernel: BTRFS error (device sdb): failed to read the system array: -2
Sep 04 14:14:36 klaatu kernel: BTRFS error (device sdb): open_ctree failedand the system was dumped into maintenance mode. Note: the boldface messages are repeated for each btrfs mount that appears in /etc/fstab. The btrfs devices are "/dev/sda" and "/dev/sdb". Using "btrfs inspect-internal dump-super /dev/sda" I can verify that "/dev/sda" is the "missing" device. ("blkid" tells me the same thing.)

One bit of oddness:
Code:# btrfs filesystem show
Label: 'localdata' uuid: 210f1faf-f5ad-4adb-8cb4-851a7b8ed149
Total devices 2 FS bytes used 697.76GiB
devid 1 size 931.51GiB used 703.03GiB path /dev/sdb
devid 2 size 931.51GiB used 703.03GiB path /dev/sdaSo I'm now confused as to why the system doesn't want to mount the btrfs filesystem. Is /dev/sda there? Or not there?

I was able to mount the btrfs filesystem but it appears to be inaccessible---one of the subvolumes is exported and clients are unable to access anything under an NFS client's mount point. In fact, issuing "ls -l mount-point" returns:
Code:d????????? ? ? ? ? ? mount-pointEven for root. I.e., probably the generic result for an NFS mount that's not working correctly on the server. (Can't say for certain---I've never seen this result before.)

In order to make a backup of the contents of the btrfs filesystem/subvolumes (across the network to another system), I mounted everything btrfs-related read-only. That went OK but, following the backups, I unmounted everything btrfs-related until I had a fix in hand. (Sort of sucks in the meantime as one of the subvolumes is "/home" where everyone's Maildirs are located. Luckily, our domain registrar buffers our incoming mail.)

BTW, while it's not a rigorous test, the disk /dev/sda appears to be OK. I was able to issue:
Code:# dd if=/dev/sda | od -c | lessand see the names of familiar items -- subdirectory names, etc. -- in the output. So, as far as I can tell at this point, /dev/sda has not actually failed---at least not seriously enough for the SMART self tests to generate alerts.

Questions:

Q1: Any ideas why this would occur? The system had been up for about 2-3 weeks (since the last update session requiring a reboot) and no obvious errors like this one after that update.

Q2: How to correct whatever problem is preventing /dev/sda from being automatically becoming part of the btrfs filesystem? Remove-then-add sda and re-add it? Or "btrfs device replace /dev/sda"? Well, I ran across a btrfs tutorial that suggested that you can mount btrfs filesystems by manually specifying the devices manually in /etc/fstab. So I changed:
Code:/dev/sdb /localdata btrfs defaults 0 0
/dev/sdb /home btrfs subvol=home 0 0to
Code:/dev/sdb /localdata btrfs device=/dev/sda,device=/dev/sdb,defaults 0 0
/dev/sdb /home btrfs device=/dev/sda,device=/dev/sdb,subvol=home 0 0and everything mounts following a reboot. I made the mount option change to one btrfs entry at a time and rebooted after each change. The first modified entry mounted followed by an error for the next btrfs entry. Changing each entry would allow that one to mount followed by an error for the next entry. Oddly, the last btrfs entry mounted without my having to change the mount options to "device=/dev...". I changed it anyway.

While I'm glad that everything btrfs seems to be working again, I not enthusiastic about having to use a mount option that most everyone says is a Bad Idea(tm) because device names can change at boot time. (Something I frequently encountered until I switched to using LABEL= in /etc/fstab---even for swap partitions.)

Something for a btrfs guru:

I've labeled the btrfs filesystem but using "LABEL=labelname" in /etc/fstab doesn't appear to work for btrfs entries. I tried using the label after I discovered that the "device=/dev/..." options were working and found that the boot time errors returned and I reverted back to using "/dev/sdb" in the "fs_spec" field. So...

Q3: Are btrfs labels supported in /etc/fstab entries?

Q4: Can one label subvolumes? Or is the label for the overall btrfs filesystem plus the "subvol=" option all that's needed?

I was about to embark on an expansion of the btrfs configuration (by adding new and removing old disks) but would like to know that /etc/fstab and the btrfs configuration is solid and I won't begin having weird failures because of something amiss in /etc/fstab.

Yes... I know I can specify UUIDs in /etc/fstab. and I will if that's the only way to avoid future problems. (But only under duress. :^D )

Any suggestions are welcomed.

As always... TIA