How to recover data from corrupted NVMe disk
by cacoo from LinuxQuestions.org on (#51FZP)
I have Fedora 31 installed on my laptop (NVMe + LVM partitions) . It failed to boot today and I have tried several methods to recover on live CD but all failed. Is there anything I could do to recover my data?
Firstly I tried fsck, it failed quickly:
Code:fsck -fy /dev/mapper/fedora-root
fsck from util-linux 2.34
e2fsck 1.45.3 (14-Jul-2019)
/dev/mapper/fedora-root: recovering journal
Superblock needs_recovery flag is clear, but journal has data.
Run journal anyway? yes
fsck.ext4: Input/output error while recovering journal of /dev/mapper/fedora-root
fsck.ext4: unable to set superblock flags on /dev/mapper/fedora-root
/dev/mapper/fedora-root: ********** WARNING: Filesystem still has errors **********
Then I tried to recover with backup superblock by e2fsck, it failed, too:
Code:e2fsck -b 32768 /dev/mapper/fedora-root
e2fsck 1.45.3 (14-Jul-2019)
Superblock needs_recovery flag is clear, but journal has data.
Recovery flag not set in backup superblock, so running journal anyway.
/dev/mapper/fedora-root: recovering journal
e2fsck: Input/output error while recovering journal of /dev/mapper/fedora-root
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences: +(819200--820231) +(884736--885767) +(1605632--1606663) +(4096000--4097031) +(7962624--7963655) +(11239424--11240455)
Fix<y>? yes to all
Free blocks count wrong for group #0 (23506, counted=23500).
Fix? yes
...
Padding at end of inode bitmap is not set. Fix? yes
Block bitmap differences: Group 0 block bitmap does not match checksum.
FIXED.
Error writing file system info: Input/output error
/dev/mapper/fedora-root: ***** FILE SYSTEM WAS MODIFIED *****output from dmesg:
Code:[ 3323.929740] blk_update_request: I/O error, dev nvme0n1, sector 2101248 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3323.929751] buffer_io_error: 190 callbacks suppressed
[ 3323.929754] Buffer I/O error on dev dm-2, logical block 0, lost async page write
[ 3323.934180] blk_update_request: I/O error, dev nvme0n1, sector 2101264 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3323.934192] Buffer I/O error on dev dm-2, logical block 2, lost async page write
[ 3323.938619] blk_update_request: I/O error, dev nvme0n1, sector 2101288 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3323.938637] Buffer I/O error on dev dm-2, logical block 5, lost async page write
[ 3323.943054] blk_update_request: I/O error, dev nvme0n1, sector 2109760 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3323.943066] Buffer I/O error on dev dm-2, logical block 1064, lost async page write
[ 3323.947484] blk_update_request: I/O error, dev nvme0n1, sector 6295808 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3323.947496] Buffer I/O error on dev dm-2, logical block 524320, lost async page write
[ 3323.951929] blk_update_request: I/O error, dev nvme0n1, sector 18878720 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3323.951942] Buffer I/O error on dev dm-2, logical block 2097184, lost async page write
[ 3323.956369] blk_update_request: I/O error, dev nvme0n1, sector 18878776 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3323.956386] Buffer I/O error on dev dm-2, logical block 2097191, lost async page write
[ 3323.960801] blk_update_request: I/O error, dev nvme0n1, sector 18878824 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3323.960819] Buffer I/O error on dev dm-2, logical block 2097197, lost async page write
[ 3323.965308] blk_update_request: I/O error, dev nvme0n1, sector 18879336 op 0x1:(WRITE) flags 0x800 phys_seg 3 prio class 0
[ 3323.965320] Buffer I/O error on dev dm-2, logical block 2097261, lost async page write
[ 3323.965332] Buffer I/O error on dev dm-2, logical block 2097262, lost async page write
[ 3323.969779] blk_update_request: I/O error, dev nvme0n1, sector 23073432 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0Other information may help:
Code:smartctl -a /dev/nvme0n1
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
- available spare has fallen below threshold
- media has been placed in read only mode
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x09
Temperature: 0 Celsius
Available Spare: 100%
Available Spare Threshold: 3%
Percentage Used: 37%
Data Units Read: 2,815,372 [1.44 TB]
Data Units Written: 3,781,234 [1.93 TB]
Host Read Commands: 160,034,805
Host Write Commands: 218,461,099
Controller Busy Time: 735
Power Cycles: 558
Power On Hours: 351
Unsafe Shutdowns: 92
Media and Data Integrity Errors: 4
Error Information Log Entries: 37
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Temperature Sensor 1: 0 Celsius
Temperature Sensor 2: 0 Celsius
Error Information (NVMe Log 0x01, max 4 entries)
Num ErrCount SQId CmdId Status PELoc LBA NSID VS
0 37 0 0x002b 0xc004 0x000 0 0 -
1 36 0 0x0023 0xc004 0x000 0 0 -
2 35 0 0x0021 0xc005 0x000 0 0 -
3 34 0 0x001e 0xc004 0x000 0 0 -
Firstly I tried fsck, it failed quickly:
Code:fsck -fy /dev/mapper/fedora-root
fsck from util-linux 2.34
e2fsck 1.45.3 (14-Jul-2019)
/dev/mapper/fedora-root: recovering journal
Superblock needs_recovery flag is clear, but journal has data.
Run journal anyway? yes
fsck.ext4: Input/output error while recovering journal of /dev/mapper/fedora-root
fsck.ext4: unable to set superblock flags on /dev/mapper/fedora-root
/dev/mapper/fedora-root: ********** WARNING: Filesystem still has errors **********
Then I tried to recover with backup superblock by e2fsck, it failed, too:
Code:e2fsck -b 32768 /dev/mapper/fedora-root
e2fsck 1.45.3 (14-Jul-2019)
Superblock needs_recovery flag is clear, but journal has data.
Recovery flag not set in backup superblock, so running journal anyway.
/dev/mapper/fedora-root: recovering journal
e2fsck: Input/output error while recovering journal of /dev/mapper/fedora-root
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
Block bitmap differences: +(819200--820231) +(884736--885767) +(1605632--1606663) +(4096000--4097031) +(7962624--7963655) +(11239424--11240455)
Fix<y>? yes to all
Free blocks count wrong for group #0 (23506, counted=23500).
Fix? yes
...
Padding at end of inode bitmap is not set. Fix? yes
Block bitmap differences: Group 0 block bitmap does not match checksum.
FIXED.
Error writing file system info: Input/output error
/dev/mapper/fedora-root: ***** FILE SYSTEM WAS MODIFIED *****output from dmesg:
Code:[ 3323.929740] blk_update_request: I/O error, dev nvme0n1, sector 2101248 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3323.929751] buffer_io_error: 190 callbacks suppressed
[ 3323.929754] Buffer I/O error on dev dm-2, logical block 0, lost async page write
[ 3323.934180] blk_update_request: I/O error, dev nvme0n1, sector 2101264 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3323.934192] Buffer I/O error on dev dm-2, logical block 2, lost async page write
[ 3323.938619] blk_update_request: I/O error, dev nvme0n1, sector 2101288 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3323.938637] Buffer I/O error on dev dm-2, logical block 5, lost async page write
[ 3323.943054] blk_update_request: I/O error, dev nvme0n1, sector 2109760 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3323.943066] Buffer I/O error on dev dm-2, logical block 1064, lost async page write
[ 3323.947484] blk_update_request: I/O error, dev nvme0n1, sector 6295808 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3323.947496] Buffer I/O error on dev dm-2, logical block 524320, lost async page write
[ 3323.951929] blk_update_request: I/O error, dev nvme0n1, sector 18878720 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3323.951942] Buffer I/O error on dev dm-2, logical block 2097184, lost async page write
[ 3323.956369] blk_update_request: I/O error, dev nvme0n1, sector 18878776 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3323.956386] Buffer I/O error on dev dm-2, logical block 2097191, lost async page write
[ 3323.960801] blk_update_request: I/O error, dev nvme0n1, sector 18878824 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
[ 3323.960819] Buffer I/O error on dev dm-2, logical block 2097197, lost async page write
[ 3323.965308] blk_update_request: I/O error, dev nvme0n1, sector 18879336 op 0x1:(WRITE) flags 0x800 phys_seg 3 prio class 0
[ 3323.965320] Buffer I/O error on dev dm-2, logical block 2097261, lost async page write
[ 3323.965332] Buffer I/O error on dev dm-2, logical block 2097262, lost async page write
[ 3323.969779] blk_update_request: I/O error, dev nvme0n1, sector 23073432 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0Other information may help:
Code:smartctl -a /dev/nvme0n1
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
- available spare has fallen below threshold
- media has been placed in read only mode
SMART/Health Information (NVMe Log 0x02)
Critical Warning: 0x09
Temperature: 0 Celsius
Available Spare: 100%
Available Spare Threshold: 3%
Percentage Used: 37%
Data Units Read: 2,815,372 [1.44 TB]
Data Units Written: 3,781,234 [1.93 TB]
Host Read Commands: 160,034,805
Host Write Commands: 218,461,099
Controller Busy Time: 735
Power Cycles: 558
Power On Hours: 351
Unsafe Shutdowns: 92
Media and Data Integrity Errors: 4
Error Information Log Entries: 37
Warning Comp. Temperature Time: 0
Critical Comp. Temperature Time: 0
Temperature Sensor 1: 0 Celsius
Temperature Sensor 2: 0 Celsius
Error Information (NVMe Log 0x01, max 4 entries)
Num ErrCount SQId CmdId Status PELoc LBA NSID VS
0 37 0 0x002b 0xc004 0x000 0 0 -
1 36 0 0x0023 0xc004 0x000 0 0 -
2 35 0 0x0021 0xc005 0x000 0 0 -
3 34 0 0x001e 0xc004 0x000 0 0 -