Urgent. RAID 5 recovery woes. Unable to add fresh disc to array to begin recovery. Help!
by LukeR1886 from LinuxQuestions.org on (#6J8RW)
So I had a cabling issue on my OMV 6 server, which caused the drives to go offline and lose sync. I already got started on the OMV forum, and got some very good help at first, but this may be above their heads/ experience, or they just have day jobs and haven't responded to the latest yet.
I've lost track of everything I've done, and hope i haven't caused any harm to the array. Let's dig in. I'll do my best to make a few replies to separated the different steps taken and the progress.
The array failed from a cabling or HBA issue. I replaced the cable with new and replaced the HBA with a known-good unit. To attempt to reassemble the array, I ran mdadm --assemble --force --verbose /dev/md127 /dev/sdb /dev/sdc /dev/sdd /dev/sde
It assembled with 3 of 4 discs, showing one disc was possibly out of date
PHP Code:root@openmediavault:~#mdadm--assemble--force--verbose/dev/md127/dev/sdb/dev/sdc/dev/sdd/dev/sde
mdadm:lookingfordevicesfor/dev/md127
mdadm:/dev/sdbisidentifiedasamemberof/dev/md127,slot1.
mdadm:/dev/sdcisidentifiedasamemberof/dev/md127,slot0.
mdadm:/dev/sddisidentifiedasamemberof/dev/md127,slot2.
mdadm:/dev/sdeisidentifiedasamemberof/dev/md127,slot3.
mdadm:added/dev/sdcto/dev/md127as0(possiblyoutofdate)
mdadm:added/dev/sddto/dev/md127as2
mdadm:added/dev/sdeto/dev/md127as3
mdadm:added/dev/sdbto/dev/md127as1
mdadm:/dev/md127hasbeenstartedwith3drives(outof4).
And here were the cat /proc/mdstat and blkid at that time
PHP Code:root@openmediavault:~#cat/proc/mdstat
Personalities:[raid6][raid5][raid4][linear][multipath][raid0][raid1][raid10]
md127:active(auto-read-only)raid5sde[3]sdb[1]sdd[2]
17581171200blockssuper1.2level5,512kchunk,algorithm2[4/3][_UUU]
bitmap:0/44pages[0KB],65536KBchunk
unuseddevices:<none>
root@openmediavault:~#blkid
/dev/sda1:UUID="43b1e19a-3a73-477f-b758-db5b833bf8af"BLOCK_SIZE="4096"TYPE="ext4"PARTUUID="71f4a275-01"
/dev/sda5:UUID="02abfba3-4cd8-40a1-b479-75598af04eb7"TYPE="swap"PARTUUID="71f4a275-05"
/dev/sdc:UUID="0d9284e2-af7d-455e-721e-c1482f768fe4"UUID_SUB="7df74adc-8d0d-5800-3a4f-2f243b67fe1a"LABEL="Bern:Storage"TYPE="linux_raid_member"
/dev/sdb:UUID="0d9284e2-af7d-455e-721e-c1482f768fe4"UUID_SUB="697882ba-3d2c-24a1-242a-33f8c80d2f48"LABEL="Bern:Storage"TYPE="linux_raid_member"
/dev/sdd:UUID="0d9284e2-af7d-455e-721e-c1482f768fe4"UUID_SUB="f4f3838b-f381-01d0-ef9c-0eb96ee49e1d"LABEL="Bern:Storage"TYPE="linux_raid_member"
/dev/sde:UUID="0d9284e2-af7d-455e-721e-c1482f768fe4"UUID_SUB="392758f0-7e5b-d0b6-fe65-ab743d6268b8"LABEL="Bern:Storage"TYPE="linux_raid_member"
/dev/md127:LABEL="Storage"UUID="fc4c074a-e2d7-46ee-82bd-2983d057c941"BLOCK_SIZE="4096"TYPE="ext4"
/dev/sr0:BLOCK_SIZE="2048"UUID="2022-04-26-18-09-57-00"LABEL="openmediavault20220426-20:09"TYPE="iso9660"PTUUID="a3602feb"PTTYPE="dos"
I've lost track of everything I've done, and hope i haven't caused any harm to the array. Let's dig in. I'll do my best to make a few replies to separated the different steps taken and the progress.
The array failed from a cabling or HBA issue. I replaced the cable with new and replaced the HBA with a known-good unit. To attempt to reassemble the array, I ran mdadm --assemble --force --verbose /dev/md127 /dev/sdb /dev/sdc /dev/sdd /dev/sde
It assembled with 3 of 4 discs, showing one disc was possibly out of date
PHP Code:root@openmediavault:~#mdadm--assemble--force--verbose/dev/md127/dev/sdb/dev/sdc/dev/sdd/dev/sde
mdadm:lookingfordevicesfor/dev/md127
mdadm:/dev/sdbisidentifiedasamemberof/dev/md127,slot1.
mdadm:/dev/sdcisidentifiedasamemberof/dev/md127,slot0.
mdadm:/dev/sddisidentifiedasamemberof/dev/md127,slot2.
mdadm:/dev/sdeisidentifiedasamemberof/dev/md127,slot3.
mdadm:added/dev/sdcto/dev/md127as0(possiblyoutofdate)
mdadm:added/dev/sddto/dev/md127as2
mdadm:added/dev/sdeto/dev/md127as3
mdadm:added/dev/sdbto/dev/md127as1
mdadm:/dev/md127hasbeenstartedwith3drives(outof4).
And here were the cat /proc/mdstat and blkid at that time
PHP Code:root@openmediavault:~#cat/proc/mdstat
Personalities:[raid6][raid5][raid4][linear][multipath][raid0][raid1][raid10]
md127:active(auto-read-only)raid5sde[3]sdb[1]sdd[2]
17581171200blockssuper1.2level5,512kchunk,algorithm2[4/3][_UUU]
bitmap:0/44pages[0KB],65536KBchunk
unuseddevices:<none>
root@openmediavault:~#blkid
/dev/sda1:UUID="43b1e19a-3a73-477f-b758-db5b833bf8af"BLOCK_SIZE="4096"TYPE="ext4"PARTUUID="71f4a275-01"
/dev/sda5:UUID="02abfba3-4cd8-40a1-b479-75598af04eb7"TYPE="swap"PARTUUID="71f4a275-05"
/dev/sdc:UUID="0d9284e2-af7d-455e-721e-c1482f768fe4"UUID_SUB="7df74adc-8d0d-5800-3a4f-2f243b67fe1a"LABEL="Bern:Storage"TYPE="linux_raid_member"
/dev/sdb:UUID="0d9284e2-af7d-455e-721e-c1482f768fe4"UUID_SUB="697882ba-3d2c-24a1-242a-33f8c80d2f48"LABEL="Bern:Storage"TYPE="linux_raid_member"
/dev/sdd:UUID="0d9284e2-af7d-455e-721e-c1482f768fe4"UUID_SUB="f4f3838b-f381-01d0-ef9c-0eb96ee49e1d"LABEL="Bern:Storage"TYPE="linux_raid_member"
/dev/sde:UUID="0d9284e2-af7d-455e-721e-c1482f768fe4"UUID_SUB="392758f0-7e5b-d0b6-fe65-ab743d6268b8"LABEL="Bern:Storage"TYPE="linux_raid_member"
/dev/md127:LABEL="Storage"UUID="fc4c074a-e2d7-46ee-82bd-2983d057c941"BLOCK_SIZE="4096"TYPE="ext4"
/dev/sr0:BLOCK_SIZE="2048"UUID="2022-04-26-18-09-57-00"LABEL="openmediavault20220426-20:09"TYPE="iso9660"PTUUID="a3602feb"PTTYPE="dos"