'du' on XFS crashes my kernel
by plznobug from LinuxQuestions.org on (#6GBTF)
Hi all,
Recently my server has been experiencing crashes, and the system logs indicate that it was caused by XFS when running du command.
The log content is as follows.
[341534.127141] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[341534.127191] IPv6: ADDRCONF(NETDEV_CHANGE): vethdd5557d6: link becomes ready
[341534.127251] cni0: port 4(vethdd5557d6) entered blocking state
[341534.127252] cni0: port 4(vethdd5557d6) entered forwarding state
[341546.280806] cni0: port 4(vethdd5557d6) entered disabled state
[341546.338512] device vethdd5557d6 left promiscuous mode
[341546.338521] cni0: port 4(vethdd5557d6) entered disabled state
[341558.138857] usercopy: Kernel memory exposure attempt detected from SLUB object 'kmalloc-16' (offset 13, size 35)!
[341558.149235] ------------[ cut here ]------------
[341558.149237] kernel BUG at mm/usercopy.c:102!
[341558.153617] invalid opcode: 0000 [#1] SMP NOPTI
[341558.158246] CPU: 53 PID: 50423 Comm: du Kdump: loaded Tainted: G W OE --------- -t - 4.18.0 #1
[341558.169650] Hardware name: ZTE R5300 G4X/R53G4X, BIOS 03.06.0400 09/01/2022
[341558.176711] RIP: 0010:usercopy_abort+0x74/0x76
[341558.181253] Code: 0f 45 c6 51 48 89 f9 48 c7 c2 3d 9c ea 85 41 52 48 c7 c6 c5 7b e9 85 48 c7 c7 08 9d ea 85 48 0f 45 f2 48 89 c2 e8 b9 33 e6 ff <0f> 0b 49 89 e8 31 c9 44 89 e2 31 f6 48 c7 c7 71 9c ea 85 e8 74 ff
[341558.200144] RSP: 0018:ff48e500225a3cf8 EFLAGS: 00010246
[341558.205468] RAX: 0000000000000065 RBX: ff3bb03416f1c4fd RCX: 0000000000000000
[341558.212703] RDX: 0000000000000000 RSI: ff3bb07f7f156a08 RDI: ff3bb07f7f156a08
[341558.219937] RBP: 0000000000000023 R08: 0000000000037b2b R09: 0000000000aaaaaa
[341558.227171] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000001
[341558.234398] R13: ff3bb03416f1c520 R14: 0000000000000023 R15: ff3bb03416f1c4fd
[341558.241627] FS: 00007f1045872b88(0000) GS:ff3bb07f7f140000(0000) knlGS:0000000000000000
[341558.249820] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[341558.255664] CR2: 0000558aae40aff0 CR3: 0000007ec20de005 CR4: 0000000000761ee0
[341558.262896] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[341558.270132] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[341558.277367] PKRU: 55555554
[341558.280165] Call Trace:
[341558.282708] __check_heap_object+0xda/0x110
[341558.286991] __check_object_size+0xfa/0x181
[341558.291272] filldir64+0xbe/0x130
[341558.294727] xfs_dir2_sf_getdents.isra.8+0x130/0x230 [xfs]
[341558.300331] xfs_readdir+0x15b/0x190 [xfs]
[341558.304522] iterate_dir+0x13c/0x190
[341558.308194] ksys_getdents64+0x9c/0x130
[341558.312131] ? iterate_dir+0x190/0x190
[341558.315977] __x64_sys_getdents64+0x16/0x20
[341558.320260] do_syscall_64+0x5b/0x1b0
[341558.324020] entry_SYSCALL_64_after_hwframe+0x65/0xca
[341558.329170] RIP: 0033:0x7f1045604401
[341558.332840] Code: 0f 05 eb 02 89 18 48 89 d0 5b c3 53 8b 47 14 49 89 f8 39 47 10 48 8d 77 20 7c 37 48 63 3f ba 00 08 00 00 b8 d9 00 00 00 0f 05 <85> c0 48 89 c3 7f 15 c1 e8 1f 74 3a 83 fb fe 74 35 f7 db e8 89 09
[341558.351717] RSP: 002b:00007fffa21af410 EFLAGS: 00000246 ORIG_RAX: 00000000000000d9
[341558.359388] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1045604401
[341558.366622] RDX: 0000000000000800 RSI: 0000558aae40a0c0 RDI: 0000000000000005
[341558.373858] RBP: 0000558aad5bfc40 R08: 0000558aae40a0a0 R09: 0000000000000000
[341558.381095] R10: 0000000000000000 R11: 0000000000000246 R12: 0000558aae40a0a0
[341558.388334] R13: 0000558aad5bfc40 R14: 0000000000000000 R15: 0000000000000000
[341558.395568] Modules linked in: ceph(OE) udp_diag tcp_diag inet_diag xt_multiport veth vxlan ip6_udp_tunnel udp_tunnel nbd(OE) rbd(OE) libceph(OE) dns_resolver xt_statistic xt_nat nf_tables ip_set ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs ip6table_mangle ip6t_MASQUERADE ip6table_filter ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables iptable_mangle xt_comment xt_mark xt_conntrack ipt_MASQUERADE nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack binfmt_misc bonding esp6_offload esp6 esp4_offload esp4 mlx5_fpga_tools(OE) mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel xfs kvm nvme irqbypass crct10dif_pclmul crc32_pclmul libcrc32c
[341558.470406] ghash_clmulni_intel pcspkr nvme_core mei_me i2c_i801 joydev mei sg wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad br_netfilter bridge stp llc overlay ip_tables ext4 mbcache jbd2 sd_mod ast mlx5_core(OE) i2c_algo_bit mlxfw(OE) ttm tls(t) vfio_mdev(OE) drm_kms_helper vfio_iommu_type1 syscopyarea vfio sysfillrect ahci sysimgblt fb_sys_fops mdev(OE) libahci drm crc32c_intel libata mlx_compat(OE)
I tried to use crash tool for analysis, and here are some information I have analyzed. It looks like XFS encountered a UAF problem causing a system bug.
crash-8.0.2> bt -FFsx
PID: 50423 TASK: ff3bb06b17418000 CPU: 53 COMMAND: "du"
#0 [ff48e500225a3a80] machine_kexec+0x1be at ffffffff84e57f3e
ff48e500225a3a88: 00002adfb820a900 ff3bb00000000000
ff48e500225a3a98: 0000000015001000 ff3bb00015001000
ff48e500225a3aa8: 0000000015000000 7ffefbffaa800800
ff48e500225a3ab8: dddc2adfb820a900 ff48e500225a3c48
ff48e500225a3ac8: ff48e500225a3ae0 ff48e500225a3c48
ff48e500225a3ad8: __crash_kexec+109
#1 [ff48e500225a3ad8] __crash_kexec+0x6d at ffffffff84f56bed
ff48e500225a3ae0: [ff3bb03416f1c4fd:kmalloc-16] 0000000000000023
ff48e500225a3af0: [ff3bb03416f1c520:kmalloc-16] 0000000000000001
ff48e500225a3b00: 0000000000000023 [ff3bb03416f1c4fd:kmalloc-16]
ff48e500225a3b10: 0000000000000001 0000000000000000
ff48e500225a3b20: 0000000000aaaaaa 0000000000037b2b
ff48e500225a3b30: 0000000000000065 0000000000000000
ff48e500225a3b40: 0000000000000000 ff3bb07f7f156a08
ff48e500225a3b50: ff3bb07f7f156a08 ffffffffffffffff
ff48e500225a3b60: usercopy_abort+116 0000000000000010
ff48e500225a3b70: 0000000000010246 ff48e500225a3cf8
ff48e500225a3b80: 0000000000000018 dddc2adfb820a900
ff48e500225a3b90: ff48e500225a3c48 000000000000000b
ff48e500225a3ba0: crash_kexec+61
#2 [ff48e500225a3ba0] crash_kexec+0x3d at ffffffff84f57acd
ff48e500225a3ba8: .LC0+142 0000000000000246
ff48e500225a3bb8: oops_end+189
#3 [ff48e500225a3bb8] oops_end+0xbd at ffffffff84e20e9d
ff48e500225a3bc0: 0000000000000006 [ff3bb06b17418000:task_struct(3831:cri-containerd-1415d8f4b03b128efdfb1d94a9b92eab05336df4419a5b5e060bb67cb3cbcc66.scope)]
ff48e500225a3bd0: 0000000000000000 do_trap+124
#4 [ff48e500225a3bd8] do_trap+0x7c at ffffffff84e1d6fc
ff48e500225a3be0: .LC0+142 usercopy_abort+116
ff48e500225a3bf0: ff48e500225a3c48 0000000000000000
ff48e500225a3c00: 0000000000000000 0000000000000000
ff48e500225a3c10: 0000000000000000 0000000000000000
ff48e500225a3c20: do_invalid_op+54
#5 [ff48e500225a3c20] do_invalid_op+0x36 at ffffffff84e1dfc6
ff48e500225a3c28: usercopy_abort+116 0000000000000000
ff48e500225a3c38: 0000000000000000 invalid_op+20
#6 [ff48e500225a3c40] invalid_op+0x14 at ffffffff85800cc4
[exception RIP: usercopy_abort+116]
RIP: ffffffff850b49b5 RSP: ff48e500225a3cf8 RFLAGS: 00010246
RAX: 0000000000000065 RBX: ff3bb03416f1c4fd RCX: 0000000000000000
RDX: 0000000000000000 RSI: ff3bb07f7f156a08 RDI: ff3bb07f7f156a08
RBP: 0000000000000023 R8: 0000000000037b2b R9: 0000000000aaaaaa
R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000001
R13: ff3bb03416f1c520 R14: 0000000000000023 R15: ff3bb03416f1c4fd
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
ff48e500225a3c48: [ff3bb03416f1c4fd:kmalloc-16] 0000000000000023
ff48e500225a3c58: [ff3bb03416f1c520:kmalloc-16] 0000000000000001
ff48e500225a3c68: 0000000000000023 [ff3bb03416f1c4fd:kmalloc-16]
ff48e500225a3c78: 0000000000000001 0000000000000000
ff48e500225a3c88: 0000000000aaaaaa 0000000000037b2b
ff48e500225a3c98: 0000000000000065 0000000000000000
ff48e500225a3ca8: 0000000000000000 ff3bb07f7f156a08
ff48e500225a3cb8: ff3bb07f7f156a08 ffffffffffffffff
ff48e500225a3cc8: usercopy_abort+116 0000000000000010
ff48e500225a3cd8: 0000000000010246 ff48e500225a3cf8
ff48e500225a3ce8: 0000000000000018 usercopy_abort+116
ff48e500225a3cf8: .LC0+19 000000000000000d
ff48e500225a3d08: 0000000000000023 __check_heap_object+218
#7 [ff48e500225a3d10] __check_heap_object+0xda at ffffffff8508d0ea
ff48e500225a3d18: __check_object_size+250
#8 [ff48e500225a3d18] __check_object_size+0xfa at ffffffff850b48ba
ff48e500225a3d20: 0000000000000000 ff48e500225a3ed0
ff48e500225a3d30: 0000558aae40a0f0 0000000000000038
ff48e500225a3d40: filldir64+190
#9 [ff48e500225a3d40] filldir64+0xbe at ffffffff850d149e
ff48e500225a3d48: 0000000000000daf ff48e500225a3ed0
ff48e500225a3d58: [ff3bb04952f3b800:xfs_inode(2566374:cri-containerd-d5d70e471b01937fad2b22127712f10fb2d2b0c7aabadc805e82a7d53cd587f8.scope)] [ff3bb03416f1c4f0:kmalloc-16]
ff48e500225a3d68: [ff3bb03416f1c4fa:kmalloc-16] 0000000000000000
ff48e500225a3d78: xfs_dir2_sf_getdents+304
#10 [ff48e500225a3d78] xfs_dir2_sf_getdents+0x130 at ffffffffc6e056e0 [xfs]
ff48e500225a3d80: 00f11634b03bff10 [ff3bb040814dc1a0:kmalloc-32]
ff48e500225a3d90: ff48e500225a3dd0 0000000000000000
ff48e500225a3da0: ff48e500225a3ed0 000000000000000a
ff48e500225a3db0: [ff3bb04952f3b800:xfs_inode(2566374:cri-containerd-d5d70e471b01937fad2b22127712f10fb2d2b0c7aabadc805e82a7d53cd587f8.scope)] 0000000000000001
ff48e500225a3dc0: xfs_readdir+347
#11 [ff48e500225a3dc0] xfs_readdir+0x15b at ffffffffc6e05e7b [xfs]
ff48e500225a3dc8: 0000000000000000 [ff3bb040814dc1a0:kmalloc-32]
ff48e500225a3dd8: 0000000000000000 0000000000000000
ff48e500225a3de8: 0000000000000000 0000000000000000
ff48e500225a3df8: 0000000000000000 0000000000000000
ff48e500225a3e08: [ff3bb04952f3b800:xfs_inode(2566374:cri-containerd-d5d70e471b01937fad2b22127712f10fb2d2b0c7aabadc805e82a7d53cd587f8.scope)] 0000000000000000
ff48e500225a3e18: 0000000000000000 0000000000000000
ff48e500225a3e28: 0000000000000000 0000000000000000
ff48e500225a3e38: 0000000000000000 0000000000000000
ff48e500225a3e48: 0000000000000000 dddc2adfb820a900
ff48e500225a3e58: [ff3bb05261a06e00:filp(3831:cri-containerd-1415d8f4b03b128efdfb1d94a9b92eab05336df4419a5b5e060bb67cb3cbcc66.scope)] 00000000fffffffe
ff48e500225a3e68: [ff3bb04952f3b938:xfs_inode(2566374:cri-containerd-d5d70e471b01937fad2b22127712f10fb2d2b0c7aabadc805e82a7d53cd587f8.scope)] [ff3bb04952f3b9e0:xfs_inode(2566374:cri-containerd-d5d70e471b01937fad2b22127712f10fb2d2b0c7aabadc805e82a7d53cd587f8.scope)]
ff48e500225a3e78: ff48e500225a3ed0 0000000000000001
ff48e500225a3e88: iterate_dir+316
#12 [ff48e500225a3e88] iterate_dir+0x13c at ffffffff850d138c
ff48e500225a3e90: 00000000653b4e62 00000000000000d9
ff48e500225a3ea0: ff48e500225a3f28 [ff3bb05261a06e00:filp(3831:cri-containerd-1415d8f4b03b128efdfb1d94a9b92eab05336df4419a5b5e060bb67cb3cbcc66.scope)]
ff48e500225a3eb0: 0000000000000800 [ff3bb05261a06e00:filp(3831:cri-containerd-1415d8f4b03b128efdfb1d94a9b92eab05336df4419a5b5e060bb67cb3cbcc66.scope)]
ff48e500225a3ec0: 0000000000000000 ksys_getdents64+156
#13 [ff48e500225a3ec8] ksys_getdents64+0x9c at ffffffff850d200c
ff48e500225a3ed0: filldir64 0000000000000daf
ff48e500225a3ee0: 0000558aae40a0f0 0000558aae40a0d8
ff48e500225a3ef0: ffffffea000007d0 dddc2adfb820a900
ff48e500225a3f00: 0000000000000000 00000000000000d9
ff48e500225a3f10: 0000000000000000 0000000000000000
ff48e500225a3f20: 0000000000000000 ff48e500225a3f58
ff48e500225a3f30: __x64_sys_getdents64+22
#14 [ff48e500225a3f30] __x64_sys_getdents64+0x16 at ffffffff850d20b6
ff48e500225a3f38: do_syscall_64+91
#15 [ff48e500225a3f38] do_syscall_64+0x5b at ffffffff84e041db
ff48e500225a3f40: 0000000000000000 0000000000000000
ff48e500225a3f50: entry_SYSCALL_64_after_hwframe+101
#16 [ff48e500225a3f50] entry_SYSCALL_64_after_hwframe+0x65 at ffffffff858000ad
RIP: 00007f1045604401 RSP: 00007fffa21af410 RFLAGS: 00000246
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1045604401
RDX: 0000000000000800 RSI: 0000558aae40a0c0 RDI: 0000000000000005
RBP: 0000558aad5bfc40 R8: 0000558aae40a0a0 R9: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000558aae40a0a0
R13: 0000558aad5bfc40 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: 00000000000000d9 CS: 0033 SS: 002b
crash-8.0.2> dis -rx xfs_dir2_sf_getdents+304
0xffffffffc6e055b0 <xfs_dir2_sf_getdents>: nopl 0x0(%rax,%rax,1) [FTRACE NOP]
0xffffffffc6e055b5 <xfs_dir2_sf_getdents+0x5>: mov 0x8(%rdx),%r9
0xffffffffc6e055b9 <xfs_dir2_sf_getdents+0x9>: movzbl 0x9(%rdi),%ecx
0xffffffffc6e055bd <xfs_dir2_sf_getdents+0xd>: movabs $0x7fffffff8,%r10
0xffffffffc6e055c7 <xfs_dir2_sf_getdents+0x17>: mov 0x14(%rdi),%r8d
0xffffffffc6e055cb <xfs_dir2_sf_getdents+0x1b>: lea 0x0(,%r9,8),%rax
0xffffffffc6e055d3 <xfs_dir2_sf_getdents+0x23>: and %r10,%rax
0xffffffffc6e055d6 <xfs_dir2_sf_getdents+0x26>: sar %cl,%rax
0xffffffffc6e055d9 <xfs_dir2_sf_getdents+0x29>: cmp %eax,%r8d
0xffffffffc6e055dc <xfs_dir2_sf_getdents+0x2c>: jb 0xffffffffc6e057d9 <xfs_dir2_sf_getdents+0x229>
0xffffffffc6e055e2 <xfs_dir2_sf_getdents+0x32>: push %r15
0xffffffffc6e055e4 <xfs_dir2_sf_getdents+0x34>: shl %cl,%r8
0xffffffffc6e055e7 <xfs_dir2_sf_getdents+0x37>: push %r14
0xffffffffc6e055e9 <xfs_dir2_sf_getdents+0x39>: push %r13
0xffffffffc6e055eb <xfs_dir2_sf_getdents+0x3b>: push %r12
0xffffffffc6e055ed <xfs_dir2_sf_getdents+0x3d>: mov %rsi,%r12
0xffffffffc6e055f0 <xfs_dir2_sf_getdents+0x40>: push %rbp
0xffffffffc6e055f1 <xfs_dir2_sf_getdents+0x41>: mov %rdx,%rbp
0xffffffffc6e055f4 <xfs_dir2_sf_getdents+0x44>: push %rbx
0xffffffffc6e055f5 <xfs_dir2_sf_getdents+0x45>: sub $0x10,%rsp
0xffffffffc6e055f9 <xfs_dir2_sf_getdents+0x49>: mov 0x68(%rsi),%rax
0xffffffffc6e055fd <xfs_dir2_sf_getdents+0x4d>: mov 0x60(%rsi),%r13
0xffffffffc6e05601 <xfs_dir2_sf_getdents+0x51>: mov %rdi,0x8(%rsp)
0xffffffffc6e05606 <xfs_dir2_sf_getdents+0x56>: mov 0x68(%rax),%ecx
0xffffffffc6e05609 <xfs_dir2_sf_getdents+0x59>: mov 0x6c(%rax),%ebx
0xffffffffc6e0560c <xfs_dir2_sf_getdents+0x5c>: add %r8,%rcx
0xffffffffc6e0560f <xfs_dir2_sf_getdents+0x5f>: add %r8,%rbx
0xffffffffc6e05612 <xfs_dir2_sf_getdents+0x62>: sar $0x3,%rcx
0xffffffffc6e05616 <xfs_dir2_sf_getdents+0x66>: sar $0x3,%rbx
0xffffffffc6e0561a <xfs_dir2_sf_getdents+0x6a>: mov %ecx,%eax
0xffffffffc6e0561c <xfs_dir2_sf_getdents+0x6c>: cmp %rax,%r9
0xffffffffc6e0561f <xfs_dir2_sf_getdents+0x6f>: jg 0xffffffffc6e05658 <xfs_dir2_sf_getdents+0xa8>
0xffffffffc6e05621 <xfs_dir2_sf_getdents+0x71>: and $0x7fffffff,%ecx
0xffffffffc6e05627 <xfs_dir2_sf_getdents+0x77>: mov (%rdx),%rax
0xffffffffc6e0562a <xfs_dir2_sf_getdents+0x7a>: mov $0x4,%r9d
0xffffffffc6e05630 <xfs_dir2_sf_getdents+0x80>: mov %rbp,%rdi
0xffffffffc6e05633 <xfs_dir2_sf_getdents+0x83>: mov %rcx,0x8(%rdx)
0xffffffffc6e05637 <xfs_dir2_sf_getdents+0x87>: mov $0x1,%edx
0xffffffffc6e0563c <xfs_dir2_sf_getdents+0x8c>: mov 0x20(%rsi),%r8
0xffffffffc6e05640 <xfs_dir2_sf_getdents+0x90>: mov $0xffffffffc6e49a70,%rsi
0xffffffffc6e05647 <xfs_dir2_sf_getdents+0x97>: call 0xffffffff85a011f0 <__x86_indirect_thunk_rax>
0xffffffffc6e0564c <xfs_dir2_sf_getdents+0x9c>: test %eax,%eax
0xffffffffc6e0564e <xfs_dir2_sf_getdents+0x9e>: jne 0xffffffffc6e0577f <xfs_dir2_sf_getdents+0x1cf>
0xffffffffc6e05654 <xfs_dir2_sf_getdents+0xa4>: mov 0x8(%rbp),%r9
0xffffffffc6e05658 <xfs_dir2_sf_getdents+0xa8>: mov %ebx,%eax
0xffffffffc6e0565a <xfs_dir2_sf_getdents+0xaa>: cmp %r9,%rax
0xffffffffc6e0565d <xfs_dir2_sf_getdents+0xad>: jge 0xffffffffc6e05790 <xfs_dir2_sf_getdents+0x1e0>
0xffffffffc6e05663 <xfs_dir2_sf_getdents+0xb3>: cmpb $0x1,0x1(%r13)
0xffffffffc6e05668 <xfs_dir2_sf_getdents+0xb8>: sbb %rax,%rax
0xffffffffc6e0566b <xfs_dir2_sf_getdents+0xbb>: xor %r15d,%r15d
0xffffffffc6e0566e <xfs_dir2_sf_getdents+0xbe>: and $0xfffffffffffffffc,%rax
0xffffffffc6e05672 <xfs_dir2_sf_getdents+0xc2>: cmpb $0x0,0x0(%r13)
0xffffffffc6e05677 <xfs_dir2_sf_getdents+0xc7>: lea 0xa(%r13,%rax,1),%r14
0xffffffffc6e0567c <xfs_dir2_sf_getdents+0xcc>: jne 0xffffffffc6e0570d <xfs_dir2_sf_getdents+0x15d>
0xffffffffc6e05682 <xfs_dir2_sf_getdents+0xd2>: jmp 0xffffffffc6e05760 <xfs_dir2_sf_getdents+0x1b0>
0xffffffffc6e05687 <xfs_dir2_sf_getdents+0xd7>: mov 0x20(%rdx),%rax
0xffffffffc6e0568b <xfs_dir2_sf_getdents+0xdb>: mov %r14,%rsi
0xffffffffc6e0568e <xfs_dir2_sf_getdents+0xde>: mov %r13,%rdi
0xffffffffc6e05691 <xfs_dir2_sf_getdents+0xe1>: and $0x7fffffff,%ebx
0xffffffffc6e05697 <xfs_dir2_sf_getdents+0xe7>: call 0xffffffff85a011f0 <__x86_indirect_thunk_rax>
0xffffffffc6e0569c <xfs_dir2_sf_getdents+0xec>: mov %r14,%rdi
0xffffffffc6e0569f <xfs_dir2_sf_getdents+0xef>: mov %rax,(%rsp)
0xffffffffc6e056a3 <xfs_dir2_sf_getdents+0xf3>: mov 0x68(%r12),%rax
0xffffffffc6e056a8 <xfs_dir2_sf_getdents+0xf8>: mov 0x10(%rax),%rax
0xffffffffc6e056ac <xfs_dir2_sf_getdents+0xfc>: call 0xffffffff85a011f0 <__x86_indirect_thunk_rax>
0xffffffffc6e056b1 <xfs_dir2_sf_getdents+0x101>: mov %rbx,0x8(%rbp)
0xffffffffc6e056b5 <xfs_dir2_sf_getdents+0x105>: mov (%r12),%rdi
0xffffffffc6e056b9 <xfs_dir2_sf_getdents+0x109>: movzbl %al,%esi
0xffffffffc6e056bc <xfs_dir2_sf_getdents+0x10c>: call 0xffffffffc6e05560 <xfs_dir3_get_dtype>
0xffffffffc6e056c1 <xfs_dir2_sf_getdents+0x111>: movzbl (%r14),%edx
0xffffffffc6e056c5 <xfs_dir2_sf_getdents+0x115>: lea 0x3(%r14),%rsi ii> 0x3( r14) is the second parameter, so r14 stores the isfepi
0xffffffffc6e056c9 <xfs_dir2_sf_getdents+0x119>: mov %rbx,%rcx
0xffffffffc6e056cc <xfs_dir2_sf_getdents+0x11c>: mov 0x0(%rbp),%r11
0xffffffffc6e056d0 <xfs_dir2_sf_getdents+0x120>: movzbl %al,%r9d
0xffffffffc6e056d4 <xfs_dir2_sf_getdents+0x124>: mov (%rsp),%r8
0xffffffffc6e056d8 <xfs_dir2_sf_getdents+0x128>: mov %rbp,%rdi
0xffffffffc6e056db <xfs_dir2_sf_getdents+0x12b>: call 0xffffffff85a01330 <__x86_indirect_thunk_r11>
0xffffffffc6e056e0 <xfs_dir2_sf_getdents+0x130>: test %eax,%eax
crash-8.0.2> dis -rx filldir64+190
0xffffffff850d13e0 <filldir64>: nopl 0x0(%rax,%rax,1) [FTRACE NOP]
0xffffffff850d13e5 <filldir64+0x5>: push %r15
0xffffffff850d13e7 <filldir64+0x7>: push %r14. iii> the second pushes up from the stack frame base is the isfepi
Up to this point, I think the second element in the stack of filldir64 is sfep.
0xffffffff850d13e9 <filldir64+0x9>: push %r13
0xffffffff850d13eb <filldir64+0xb>: lea 0x1b(%rdx),%r13d
0xffffffff850d13ef <filldir64+0xf>: push %r12
0xffffffff850d13f1 <filldir64+0x11>: and $0xfffffff8,%r13d
0xffffffff850d13f5 <filldir64+0x15>: push %rbp
0xffffffff850d13f6 <filldir64+0x16>: push %rbx
0xffffffff850d13f7 <filldir64+0x17>: movl $0xffffffea,0x24(%rdi)
0xffffffff850d13fe <filldir64+0x1e>: cmp %r13d,0x20(%rdi)
0xffffffff850d1402 <filldir64+0x22>: jl 0xffffffff850d14f9 <filldir64+0x119>
0xffffffff850d1408 <filldir64+0x28>: mov %rsi,%r15
0xffffffff850d140b <filldir64+0x2b>: mov 0x18(%rdi),%rsi
0xffffffff850d140f <filldir64+0x2f>: mov %rdi,%rbp
0xffffffff850d1412 <filldir64+0x32>: test %rsi,%rsi
0xffffffff850d1415 <filldir64+0x35>: je 0xffffffff850d143f <filldir64+0x5f>
0xffffffff850d1417 <filldir64+0x37>: mov %gs:0x15c80,%rax
0xffffffff850d1420 <filldir64+0x40>: mov (%rax),%rax
0xffffffff850d1423 <filldir64+0x43>: test $0x4,%al
0xffffffff850d1425 <filldir64+0x45>: jne 0xffffffff850d1500 <filldir64+0x120>
0xffffffff850d142b <filldir64+0x4b>: stac
0xffffffff850d142e <filldir64+0x4e>: xor %eax,%eax
0xffffffff850d1430 <filldir64+0x50>: mov %rcx,0x8(%rsi)
0xffffffff850d1434 <filldir64+0x54>: clac
0xffffffff850d1437 <filldir64+0x57>: test %eax,%eax
0xffffffff850d1439 <filldir64+0x59>: jne 0xffffffff850d14e2 <filldir64+0x102>
0xffffffff850d143f <filldir64+0x5f>: mov 0x10(%rbp),%r12
0xffffffff850d1443 <filldir64+0x63>: stac
0xffffffff850d1446 <filldir64+0x66>: xor %eax,%eax
0xffffffff850d1448 <filldir64+0x68>: mov %r8,(%r12)
0xffffffff850d144c <filldir64+0x6c>: clac
0xffffffff850d144f <filldir64+0x6f>: test %eax,%eax
0xffffffff850d1451 <filldir64+0x71>: jne 0xffffffff850d14e2 <filldir64+0x102>
0xffffffff850d1457 <filldir64+0x77>: stac
0xffffffff850d145a <filldir64+0x7a>: movq $0x0,0x8(%r12)
0xffffffff850d1463 <filldir64+0x83>: clac
0xffffffff850d1466 <filldir64+0x86>: test %eax,%eax
0xffffffff850d1468 <filldir64+0x88>: jne 0xffffffff850d14e2 <filldir64+0x102>
0xffffffff850d146a <filldir64+0x8a>: stac
0xffffffff850d146d <filldir64+0x8d>: mov %r13w,0x10(%r12)
0xffffffff850d1473 <filldir64+0x93>: clac
0xffffffff850d1476 <filldir64+0x96>: test %eax,%eax
0xffffffff850d1478 <filldir64+0x98>: jne 0xffffffff850d14e2 <filldir64+0x102>
0xffffffff850d147a <filldir64+0x9a>: stac
0xffffffff850d147d <filldir64+0x9d>: mov %eax,%ebx
0xffffffff850d147f <filldir64+0x9f>: mov %r9b,0x12(%r12)
0xffffffff850d1484 <filldir64+0xa4>: clac
0xffffffff850d1487 <filldir64+0xa7>: test %ebx,%ebx
0xffffffff850d1489 <filldir64+0xa9>: jne 0xffffffff850d14e2 <filldir64+0x102>
0xffffffff850d148b <filldir64+0xab>: movslq %edx,%r14
0xffffffff850d148e <filldir64+0xae>: mov %r15,%rdi. ii> r15 is the first parameter when calling __check_object_size
0xffffffff850d1491 <filldir64+0xb1>: mov $0x1,%edx
0xffffffff850d1496 <filldir64+0xb6>: mov %r14,%rsi
0xffffffff850d1499 <filldir64+0xb9>: call 0xffffffff850b47c0 <__check_object_size>
0xffffffff850d149e <filldir64+0xbe>: mov %r14,%rdx
Up to this point, r15 is the given object for __check_object_size to validate.
And no one change the r15 until the end, so we can get the given object from stack of usercopy_abort i.e., R15: ff3bb03416f1c4fd.
I check isfepi and the give object by using ikmemi
crash-8.0.2> kmem ff3bb03416f1c4fa
CACHE OBJSIZE ALLOCATED TOTAL SLABS SSIZE NAME
ff3bb00107c0fa00 16 2074327 2860288 11173 4k kmalloc-16
SLAB MEMORY NODE TOTAL ALLOCATED FREE
ff8faa00d05bc700 ff3bb03416f1c000 0 256 76 180
FREE / [ALLOCATED]
ff3bb03416f1c4f0 (cpu 4 cache)
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
ff8faa00d05bc700 3416f1c000 ff3bb00107c0fa00 ff3bb03416f1ca00 1 17ffffc0000100 slab
crash-8.0.2> kmem ff3bb03416f1c4fd
CACHE OBJSIZE ALLOCATED TOTAL SLABS SSIZE NAME
ff3bb00107c0fa00 16 2074327 2860288 11173 4k kmalloc-16
SLAB MEMORY NODE TOTAL ALLOCATED FREE
ff8faa00d05bc700 ff3bb03416f1c000 0 256 76 180
FREE / [ALLOCATED]
ff3bb03416f1c4f0 (cpu 4 cache)
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
ff8faa00d05bc700 3416f1c000 ff3bb00107c0fa00 ff3bb03416f1ca00 1 17ffffc0000100 slab
It seemed the object have been freed.
I have no idea what is going on here.
Sincerely hope that you lot give me some help.
Below is some information about my server.
# xfs_db -r /dev/nvme0n1
xfs_db> version
versionnum [0xbcb5+0x18a] = V5,NLINK,DIRV2,ATTR,ALIGN,LOGV2,EXTFLG,SECTOR,MOREBITS,ATTR2,LAZYSBCOUNT,PROJID32BIT,CRC,FTYPE
Linux kernel version is v4.18 and the OS is Centos7.
Best regards.
Recently my server has been experiencing crashes, and the system logs indicate that it was caused by XFS when running du command.
The log content is as follows.
[341534.127141] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
[341534.127191] IPv6: ADDRCONF(NETDEV_CHANGE): vethdd5557d6: link becomes ready
[341534.127251] cni0: port 4(vethdd5557d6) entered blocking state
[341534.127252] cni0: port 4(vethdd5557d6) entered forwarding state
[341546.280806] cni0: port 4(vethdd5557d6) entered disabled state
[341546.338512] device vethdd5557d6 left promiscuous mode
[341546.338521] cni0: port 4(vethdd5557d6) entered disabled state
[341558.138857] usercopy: Kernel memory exposure attempt detected from SLUB object 'kmalloc-16' (offset 13, size 35)!
[341558.149235] ------------[ cut here ]------------
[341558.149237] kernel BUG at mm/usercopy.c:102!
[341558.153617] invalid opcode: 0000 [#1] SMP NOPTI
[341558.158246] CPU: 53 PID: 50423 Comm: du Kdump: loaded Tainted: G W OE --------- -t - 4.18.0 #1
[341558.169650] Hardware name: ZTE R5300 G4X/R53G4X, BIOS 03.06.0400 09/01/2022
[341558.176711] RIP: 0010:usercopy_abort+0x74/0x76
[341558.181253] Code: 0f 45 c6 51 48 89 f9 48 c7 c2 3d 9c ea 85 41 52 48 c7 c6 c5 7b e9 85 48 c7 c7 08 9d ea 85 48 0f 45 f2 48 89 c2 e8 b9 33 e6 ff <0f> 0b 49 89 e8 31 c9 44 89 e2 31 f6 48 c7 c7 71 9c ea 85 e8 74 ff
[341558.200144] RSP: 0018:ff48e500225a3cf8 EFLAGS: 00010246
[341558.205468] RAX: 0000000000000065 RBX: ff3bb03416f1c4fd RCX: 0000000000000000
[341558.212703] RDX: 0000000000000000 RSI: ff3bb07f7f156a08 RDI: ff3bb07f7f156a08
[341558.219937] RBP: 0000000000000023 R08: 0000000000037b2b R09: 0000000000aaaaaa
[341558.227171] R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000001
[341558.234398] R13: ff3bb03416f1c520 R14: 0000000000000023 R15: ff3bb03416f1c4fd
[341558.241627] FS: 00007f1045872b88(0000) GS:ff3bb07f7f140000(0000) knlGS:0000000000000000
[341558.249820] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[341558.255664] CR2: 0000558aae40aff0 CR3: 0000007ec20de005 CR4: 0000000000761ee0
[341558.262896] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[341558.270132] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[341558.277367] PKRU: 55555554
[341558.280165] Call Trace:
[341558.282708] __check_heap_object+0xda/0x110
[341558.286991] __check_object_size+0xfa/0x181
[341558.291272] filldir64+0xbe/0x130
[341558.294727] xfs_dir2_sf_getdents.isra.8+0x130/0x230 [xfs]
[341558.300331] xfs_readdir+0x15b/0x190 [xfs]
[341558.304522] iterate_dir+0x13c/0x190
[341558.308194] ksys_getdents64+0x9c/0x130
[341558.312131] ? iterate_dir+0x190/0x190
[341558.315977] __x64_sys_getdents64+0x16/0x20
[341558.320260] do_syscall_64+0x5b/0x1b0
[341558.324020] entry_SYSCALL_64_after_hwframe+0x65/0xca
[341558.329170] RIP: 0033:0x7f1045604401
[341558.332840] Code: 0f 05 eb 02 89 18 48 89 d0 5b c3 53 8b 47 14 49 89 f8 39 47 10 48 8d 77 20 7c 37 48 63 3f ba 00 08 00 00 b8 d9 00 00 00 0f 05 <85> c0 48 89 c3 7f 15 c1 e8 1f 74 3a 83 fb fe 74 35 f7 db e8 89 09
[341558.351717] RSP: 002b:00007fffa21af410 EFLAGS: 00000246 ORIG_RAX: 00000000000000d9
[341558.359388] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1045604401
[341558.366622] RDX: 0000000000000800 RSI: 0000558aae40a0c0 RDI: 0000000000000005
[341558.373858] RBP: 0000558aad5bfc40 R08: 0000558aae40a0a0 R09: 0000000000000000
[341558.381095] R10: 0000000000000000 R11: 0000000000000246 R12: 0000558aae40a0a0
[341558.388334] R13: 0000558aad5bfc40 R14: 0000000000000000 R15: 0000000000000000
[341558.395568] Modules linked in: ceph(OE) udp_diag tcp_diag inet_diag xt_multiport veth vxlan ip6_udp_tunnel udp_tunnel nbd(OE) rbd(OE) libceph(OE) dns_resolver xt_statistic xt_nat nf_tables ip_set ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs ip6table_mangle ip6t_MASQUERADE ip6table_filter ip6table_nat nf_conntrack_ipv6 nf_defrag_ipv6 nf_nat_ipv6 ip6_tables iptable_mangle xt_comment xt_mark xt_conntrack ipt_MASQUERADE nf_conntrack_netlink nfnetlink xt_addrtype iptable_filter iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 nf_nat nf_conntrack binfmt_misc bonding esp6_offload esp6 esp4_offload esp4 mlx5_fpga_tools(OE) mlx5_ib(OE) ib_uverbs(OE) ib_core(OE) x86_pkg_temp_thermal intel_powerclamp coretemp iTCO_wdt iTCO_vendor_support kvm_intel xfs kvm nvme irqbypass crct10dif_pclmul crc32_pclmul libcrc32c
[341558.470406] ghash_clmulni_intel pcspkr nvme_core mei_me i2c_i801 joydev mei sg wmi ipmi_si ipmi_devintf ipmi_msghandler acpi_power_meter acpi_pad br_netfilter bridge stp llc overlay ip_tables ext4 mbcache jbd2 sd_mod ast mlx5_core(OE) i2c_algo_bit mlxfw(OE) ttm tls(t) vfio_mdev(OE) drm_kms_helper vfio_iommu_type1 syscopyarea vfio sysfillrect ahci sysimgblt fb_sys_fops mdev(OE) libahci drm crc32c_intel libata mlx_compat(OE)
I tried to use crash tool for analysis, and here are some information I have analyzed. It looks like XFS encountered a UAF problem causing a system bug.
crash-8.0.2> bt -FFsx
PID: 50423 TASK: ff3bb06b17418000 CPU: 53 COMMAND: "du"
#0 [ff48e500225a3a80] machine_kexec+0x1be at ffffffff84e57f3e
ff48e500225a3a88: 00002adfb820a900 ff3bb00000000000
ff48e500225a3a98: 0000000015001000 ff3bb00015001000
ff48e500225a3aa8: 0000000015000000 7ffefbffaa800800
ff48e500225a3ab8: dddc2adfb820a900 ff48e500225a3c48
ff48e500225a3ac8: ff48e500225a3ae0 ff48e500225a3c48
ff48e500225a3ad8: __crash_kexec+109
#1 [ff48e500225a3ad8] __crash_kexec+0x6d at ffffffff84f56bed
ff48e500225a3ae0: [ff3bb03416f1c4fd:kmalloc-16] 0000000000000023
ff48e500225a3af0: [ff3bb03416f1c520:kmalloc-16] 0000000000000001
ff48e500225a3b00: 0000000000000023 [ff3bb03416f1c4fd:kmalloc-16]
ff48e500225a3b10: 0000000000000001 0000000000000000
ff48e500225a3b20: 0000000000aaaaaa 0000000000037b2b
ff48e500225a3b30: 0000000000000065 0000000000000000
ff48e500225a3b40: 0000000000000000 ff3bb07f7f156a08
ff48e500225a3b50: ff3bb07f7f156a08 ffffffffffffffff
ff48e500225a3b60: usercopy_abort+116 0000000000000010
ff48e500225a3b70: 0000000000010246 ff48e500225a3cf8
ff48e500225a3b80: 0000000000000018 dddc2adfb820a900
ff48e500225a3b90: ff48e500225a3c48 000000000000000b
ff48e500225a3ba0: crash_kexec+61
#2 [ff48e500225a3ba0] crash_kexec+0x3d at ffffffff84f57acd
ff48e500225a3ba8: .LC0+142 0000000000000246
ff48e500225a3bb8: oops_end+189
#3 [ff48e500225a3bb8] oops_end+0xbd at ffffffff84e20e9d
ff48e500225a3bc0: 0000000000000006 [ff3bb06b17418000:task_struct(3831:cri-containerd-1415d8f4b03b128efdfb1d94a9b92eab05336df4419a5b5e060bb67cb3cbcc66.scope)]
ff48e500225a3bd0: 0000000000000000 do_trap+124
#4 [ff48e500225a3bd8] do_trap+0x7c at ffffffff84e1d6fc
ff48e500225a3be0: .LC0+142 usercopy_abort+116
ff48e500225a3bf0: ff48e500225a3c48 0000000000000000
ff48e500225a3c00: 0000000000000000 0000000000000000
ff48e500225a3c10: 0000000000000000 0000000000000000
ff48e500225a3c20: do_invalid_op+54
#5 [ff48e500225a3c20] do_invalid_op+0x36 at ffffffff84e1dfc6
ff48e500225a3c28: usercopy_abort+116 0000000000000000
ff48e500225a3c38: 0000000000000000 invalid_op+20
#6 [ff48e500225a3c40] invalid_op+0x14 at ffffffff85800cc4
[exception RIP: usercopy_abort+116]
RIP: ffffffff850b49b5 RSP: ff48e500225a3cf8 RFLAGS: 00010246
RAX: 0000000000000065 RBX: ff3bb03416f1c4fd RCX: 0000000000000000
RDX: 0000000000000000 RSI: ff3bb07f7f156a08 RDI: ff3bb07f7f156a08
RBP: 0000000000000023 R8: 0000000000037b2b R9: 0000000000aaaaaa
R10: 0000000000000000 R11: 0000000000000001 R12: 0000000000000001
R13: ff3bb03416f1c520 R14: 0000000000000023 R15: ff3bb03416f1c4fd
ORIG_RAX: ffffffffffffffff CS: 0010 SS: 0018
ff48e500225a3c48: [ff3bb03416f1c4fd:kmalloc-16] 0000000000000023
ff48e500225a3c58: [ff3bb03416f1c520:kmalloc-16] 0000000000000001
ff48e500225a3c68: 0000000000000023 [ff3bb03416f1c4fd:kmalloc-16]
ff48e500225a3c78: 0000000000000001 0000000000000000
ff48e500225a3c88: 0000000000aaaaaa 0000000000037b2b
ff48e500225a3c98: 0000000000000065 0000000000000000
ff48e500225a3ca8: 0000000000000000 ff3bb07f7f156a08
ff48e500225a3cb8: ff3bb07f7f156a08 ffffffffffffffff
ff48e500225a3cc8: usercopy_abort+116 0000000000000010
ff48e500225a3cd8: 0000000000010246 ff48e500225a3cf8
ff48e500225a3ce8: 0000000000000018 usercopy_abort+116
ff48e500225a3cf8: .LC0+19 000000000000000d
ff48e500225a3d08: 0000000000000023 __check_heap_object+218
#7 [ff48e500225a3d10] __check_heap_object+0xda at ffffffff8508d0ea
ff48e500225a3d18: __check_object_size+250
#8 [ff48e500225a3d18] __check_object_size+0xfa at ffffffff850b48ba
ff48e500225a3d20: 0000000000000000 ff48e500225a3ed0
ff48e500225a3d30: 0000558aae40a0f0 0000000000000038
ff48e500225a3d40: filldir64+190
#9 [ff48e500225a3d40] filldir64+0xbe at ffffffff850d149e
ff48e500225a3d48: 0000000000000daf ff48e500225a3ed0
ff48e500225a3d58: [ff3bb04952f3b800:xfs_inode(2566374:cri-containerd-d5d70e471b01937fad2b22127712f10fb2d2b0c7aabadc805e82a7d53cd587f8.scope)] [ff3bb03416f1c4f0:kmalloc-16]
ff48e500225a3d68: [ff3bb03416f1c4fa:kmalloc-16] 0000000000000000
ff48e500225a3d78: xfs_dir2_sf_getdents+304
#10 [ff48e500225a3d78] xfs_dir2_sf_getdents+0x130 at ffffffffc6e056e0 [xfs]
ff48e500225a3d80: 00f11634b03bff10 [ff3bb040814dc1a0:kmalloc-32]
ff48e500225a3d90: ff48e500225a3dd0 0000000000000000
ff48e500225a3da0: ff48e500225a3ed0 000000000000000a
ff48e500225a3db0: [ff3bb04952f3b800:xfs_inode(2566374:cri-containerd-d5d70e471b01937fad2b22127712f10fb2d2b0c7aabadc805e82a7d53cd587f8.scope)] 0000000000000001
ff48e500225a3dc0: xfs_readdir+347
#11 [ff48e500225a3dc0] xfs_readdir+0x15b at ffffffffc6e05e7b [xfs]
ff48e500225a3dc8: 0000000000000000 [ff3bb040814dc1a0:kmalloc-32]
ff48e500225a3dd8: 0000000000000000 0000000000000000
ff48e500225a3de8: 0000000000000000 0000000000000000
ff48e500225a3df8: 0000000000000000 0000000000000000
ff48e500225a3e08: [ff3bb04952f3b800:xfs_inode(2566374:cri-containerd-d5d70e471b01937fad2b22127712f10fb2d2b0c7aabadc805e82a7d53cd587f8.scope)] 0000000000000000
ff48e500225a3e18: 0000000000000000 0000000000000000
ff48e500225a3e28: 0000000000000000 0000000000000000
ff48e500225a3e38: 0000000000000000 0000000000000000
ff48e500225a3e48: 0000000000000000 dddc2adfb820a900
ff48e500225a3e58: [ff3bb05261a06e00:filp(3831:cri-containerd-1415d8f4b03b128efdfb1d94a9b92eab05336df4419a5b5e060bb67cb3cbcc66.scope)] 00000000fffffffe
ff48e500225a3e68: [ff3bb04952f3b938:xfs_inode(2566374:cri-containerd-d5d70e471b01937fad2b22127712f10fb2d2b0c7aabadc805e82a7d53cd587f8.scope)] [ff3bb04952f3b9e0:xfs_inode(2566374:cri-containerd-d5d70e471b01937fad2b22127712f10fb2d2b0c7aabadc805e82a7d53cd587f8.scope)]
ff48e500225a3e78: ff48e500225a3ed0 0000000000000001
ff48e500225a3e88: iterate_dir+316
#12 [ff48e500225a3e88] iterate_dir+0x13c at ffffffff850d138c
ff48e500225a3e90: 00000000653b4e62 00000000000000d9
ff48e500225a3ea0: ff48e500225a3f28 [ff3bb05261a06e00:filp(3831:cri-containerd-1415d8f4b03b128efdfb1d94a9b92eab05336df4419a5b5e060bb67cb3cbcc66.scope)]
ff48e500225a3eb0: 0000000000000800 [ff3bb05261a06e00:filp(3831:cri-containerd-1415d8f4b03b128efdfb1d94a9b92eab05336df4419a5b5e060bb67cb3cbcc66.scope)]
ff48e500225a3ec0: 0000000000000000 ksys_getdents64+156
#13 [ff48e500225a3ec8] ksys_getdents64+0x9c at ffffffff850d200c
ff48e500225a3ed0: filldir64 0000000000000daf
ff48e500225a3ee0: 0000558aae40a0f0 0000558aae40a0d8
ff48e500225a3ef0: ffffffea000007d0 dddc2adfb820a900
ff48e500225a3f00: 0000000000000000 00000000000000d9
ff48e500225a3f10: 0000000000000000 0000000000000000
ff48e500225a3f20: 0000000000000000 ff48e500225a3f58
ff48e500225a3f30: __x64_sys_getdents64+22
#14 [ff48e500225a3f30] __x64_sys_getdents64+0x16 at ffffffff850d20b6
ff48e500225a3f38: do_syscall_64+91
#15 [ff48e500225a3f38] do_syscall_64+0x5b at ffffffff84e041db
ff48e500225a3f40: 0000000000000000 0000000000000000
ff48e500225a3f50: entry_SYSCALL_64_after_hwframe+101
#16 [ff48e500225a3f50] entry_SYSCALL_64_after_hwframe+0x65 at ffffffff858000ad
RIP: 00007f1045604401 RSP: 00007fffa21af410 RFLAGS: 00000246
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f1045604401
RDX: 0000000000000800 RSI: 0000558aae40a0c0 RDI: 0000000000000005
RBP: 0000558aad5bfc40 R8: 0000558aae40a0a0 R9: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000558aae40a0a0
R13: 0000558aad5bfc40 R14: 0000000000000000 R15: 0000000000000000
ORIG_RAX: 00000000000000d9 CS: 0033 SS: 002b
crash-8.0.2> dis -rx xfs_dir2_sf_getdents+304
0xffffffffc6e055b0 <xfs_dir2_sf_getdents>: nopl 0x0(%rax,%rax,1) [FTRACE NOP]
0xffffffffc6e055b5 <xfs_dir2_sf_getdents+0x5>: mov 0x8(%rdx),%r9
0xffffffffc6e055b9 <xfs_dir2_sf_getdents+0x9>: movzbl 0x9(%rdi),%ecx
0xffffffffc6e055bd <xfs_dir2_sf_getdents+0xd>: movabs $0x7fffffff8,%r10
0xffffffffc6e055c7 <xfs_dir2_sf_getdents+0x17>: mov 0x14(%rdi),%r8d
0xffffffffc6e055cb <xfs_dir2_sf_getdents+0x1b>: lea 0x0(,%r9,8),%rax
0xffffffffc6e055d3 <xfs_dir2_sf_getdents+0x23>: and %r10,%rax
0xffffffffc6e055d6 <xfs_dir2_sf_getdents+0x26>: sar %cl,%rax
0xffffffffc6e055d9 <xfs_dir2_sf_getdents+0x29>: cmp %eax,%r8d
0xffffffffc6e055dc <xfs_dir2_sf_getdents+0x2c>: jb 0xffffffffc6e057d9 <xfs_dir2_sf_getdents+0x229>
0xffffffffc6e055e2 <xfs_dir2_sf_getdents+0x32>: push %r15
0xffffffffc6e055e4 <xfs_dir2_sf_getdents+0x34>: shl %cl,%r8
0xffffffffc6e055e7 <xfs_dir2_sf_getdents+0x37>: push %r14
0xffffffffc6e055e9 <xfs_dir2_sf_getdents+0x39>: push %r13
0xffffffffc6e055eb <xfs_dir2_sf_getdents+0x3b>: push %r12
0xffffffffc6e055ed <xfs_dir2_sf_getdents+0x3d>: mov %rsi,%r12
0xffffffffc6e055f0 <xfs_dir2_sf_getdents+0x40>: push %rbp
0xffffffffc6e055f1 <xfs_dir2_sf_getdents+0x41>: mov %rdx,%rbp
0xffffffffc6e055f4 <xfs_dir2_sf_getdents+0x44>: push %rbx
0xffffffffc6e055f5 <xfs_dir2_sf_getdents+0x45>: sub $0x10,%rsp
0xffffffffc6e055f9 <xfs_dir2_sf_getdents+0x49>: mov 0x68(%rsi),%rax
0xffffffffc6e055fd <xfs_dir2_sf_getdents+0x4d>: mov 0x60(%rsi),%r13
0xffffffffc6e05601 <xfs_dir2_sf_getdents+0x51>: mov %rdi,0x8(%rsp)
0xffffffffc6e05606 <xfs_dir2_sf_getdents+0x56>: mov 0x68(%rax),%ecx
0xffffffffc6e05609 <xfs_dir2_sf_getdents+0x59>: mov 0x6c(%rax),%ebx
0xffffffffc6e0560c <xfs_dir2_sf_getdents+0x5c>: add %r8,%rcx
0xffffffffc6e0560f <xfs_dir2_sf_getdents+0x5f>: add %r8,%rbx
0xffffffffc6e05612 <xfs_dir2_sf_getdents+0x62>: sar $0x3,%rcx
0xffffffffc6e05616 <xfs_dir2_sf_getdents+0x66>: sar $0x3,%rbx
0xffffffffc6e0561a <xfs_dir2_sf_getdents+0x6a>: mov %ecx,%eax
0xffffffffc6e0561c <xfs_dir2_sf_getdents+0x6c>: cmp %rax,%r9
0xffffffffc6e0561f <xfs_dir2_sf_getdents+0x6f>: jg 0xffffffffc6e05658 <xfs_dir2_sf_getdents+0xa8>
0xffffffffc6e05621 <xfs_dir2_sf_getdents+0x71>: and $0x7fffffff,%ecx
0xffffffffc6e05627 <xfs_dir2_sf_getdents+0x77>: mov (%rdx),%rax
0xffffffffc6e0562a <xfs_dir2_sf_getdents+0x7a>: mov $0x4,%r9d
0xffffffffc6e05630 <xfs_dir2_sf_getdents+0x80>: mov %rbp,%rdi
0xffffffffc6e05633 <xfs_dir2_sf_getdents+0x83>: mov %rcx,0x8(%rdx)
0xffffffffc6e05637 <xfs_dir2_sf_getdents+0x87>: mov $0x1,%edx
0xffffffffc6e0563c <xfs_dir2_sf_getdents+0x8c>: mov 0x20(%rsi),%r8
0xffffffffc6e05640 <xfs_dir2_sf_getdents+0x90>: mov $0xffffffffc6e49a70,%rsi
0xffffffffc6e05647 <xfs_dir2_sf_getdents+0x97>: call 0xffffffff85a011f0 <__x86_indirect_thunk_rax>
0xffffffffc6e0564c <xfs_dir2_sf_getdents+0x9c>: test %eax,%eax
0xffffffffc6e0564e <xfs_dir2_sf_getdents+0x9e>: jne 0xffffffffc6e0577f <xfs_dir2_sf_getdents+0x1cf>
0xffffffffc6e05654 <xfs_dir2_sf_getdents+0xa4>: mov 0x8(%rbp),%r9
0xffffffffc6e05658 <xfs_dir2_sf_getdents+0xa8>: mov %ebx,%eax
0xffffffffc6e0565a <xfs_dir2_sf_getdents+0xaa>: cmp %r9,%rax
0xffffffffc6e0565d <xfs_dir2_sf_getdents+0xad>: jge 0xffffffffc6e05790 <xfs_dir2_sf_getdents+0x1e0>
0xffffffffc6e05663 <xfs_dir2_sf_getdents+0xb3>: cmpb $0x1,0x1(%r13)
0xffffffffc6e05668 <xfs_dir2_sf_getdents+0xb8>: sbb %rax,%rax
0xffffffffc6e0566b <xfs_dir2_sf_getdents+0xbb>: xor %r15d,%r15d
0xffffffffc6e0566e <xfs_dir2_sf_getdents+0xbe>: and $0xfffffffffffffffc,%rax
0xffffffffc6e05672 <xfs_dir2_sf_getdents+0xc2>: cmpb $0x0,0x0(%r13)
0xffffffffc6e05677 <xfs_dir2_sf_getdents+0xc7>: lea 0xa(%r13,%rax,1),%r14
0xffffffffc6e0567c <xfs_dir2_sf_getdents+0xcc>: jne 0xffffffffc6e0570d <xfs_dir2_sf_getdents+0x15d>
0xffffffffc6e05682 <xfs_dir2_sf_getdents+0xd2>: jmp 0xffffffffc6e05760 <xfs_dir2_sf_getdents+0x1b0>
0xffffffffc6e05687 <xfs_dir2_sf_getdents+0xd7>: mov 0x20(%rdx),%rax
0xffffffffc6e0568b <xfs_dir2_sf_getdents+0xdb>: mov %r14,%rsi
0xffffffffc6e0568e <xfs_dir2_sf_getdents+0xde>: mov %r13,%rdi
0xffffffffc6e05691 <xfs_dir2_sf_getdents+0xe1>: and $0x7fffffff,%ebx
0xffffffffc6e05697 <xfs_dir2_sf_getdents+0xe7>: call 0xffffffff85a011f0 <__x86_indirect_thunk_rax>
0xffffffffc6e0569c <xfs_dir2_sf_getdents+0xec>: mov %r14,%rdi
0xffffffffc6e0569f <xfs_dir2_sf_getdents+0xef>: mov %rax,(%rsp)
0xffffffffc6e056a3 <xfs_dir2_sf_getdents+0xf3>: mov 0x68(%r12),%rax
0xffffffffc6e056a8 <xfs_dir2_sf_getdents+0xf8>: mov 0x10(%rax),%rax
0xffffffffc6e056ac <xfs_dir2_sf_getdents+0xfc>: call 0xffffffff85a011f0 <__x86_indirect_thunk_rax>
0xffffffffc6e056b1 <xfs_dir2_sf_getdents+0x101>: mov %rbx,0x8(%rbp)
0xffffffffc6e056b5 <xfs_dir2_sf_getdents+0x105>: mov (%r12),%rdi
0xffffffffc6e056b9 <xfs_dir2_sf_getdents+0x109>: movzbl %al,%esi
0xffffffffc6e056bc <xfs_dir2_sf_getdents+0x10c>: call 0xffffffffc6e05560 <xfs_dir3_get_dtype>
0xffffffffc6e056c1 <xfs_dir2_sf_getdents+0x111>: movzbl (%r14),%edx
0xffffffffc6e056c5 <xfs_dir2_sf_getdents+0x115>: lea 0x3(%r14),%rsi ii> 0x3( r14) is the second parameter, so r14 stores the isfepi
0xffffffffc6e056c9 <xfs_dir2_sf_getdents+0x119>: mov %rbx,%rcx
0xffffffffc6e056cc <xfs_dir2_sf_getdents+0x11c>: mov 0x0(%rbp),%r11
0xffffffffc6e056d0 <xfs_dir2_sf_getdents+0x120>: movzbl %al,%r9d
0xffffffffc6e056d4 <xfs_dir2_sf_getdents+0x124>: mov (%rsp),%r8
0xffffffffc6e056d8 <xfs_dir2_sf_getdents+0x128>: mov %rbp,%rdi
0xffffffffc6e056db <xfs_dir2_sf_getdents+0x12b>: call 0xffffffff85a01330 <__x86_indirect_thunk_r11>
0xffffffffc6e056e0 <xfs_dir2_sf_getdents+0x130>: test %eax,%eax
crash-8.0.2> dis -rx filldir64+190
0xffffffff850d13e0 <filldir64>: nopl 0x0(%rax,%rax,1) [FTRACE NOP]
0xffffffff850d13e5 <filldir64+0x5>: push %r15
0xffffffff850d13e7 <filldir64+0x7>: push %r14. iii> the second pushes up from the stack frame base is the isfepi
Up to this point, I think the second element in the stack of filldir64 is sfep.
0xffffffff850d13e9 <filldir64+0x9>: push %r13
0xffffffff850d13eb <filldir64+0xb>: lea 0x1b(%rdx),%r13d
0xffffffff850d13ef <filldir64+0xf>: push %r12
0xffffffff850d13f1 <filldir64+0x11>: and $0xfffffff8,%r13d
0xffffffff850d13f5 <filldir64+0x15>: push %rbp
0xffffffff850d13f6 <filldir64+0x16>: push %rbx
0xffffffff850d13f7 <filldir64+0x17>: movl $0xffffffea,0x24(%rdi)
0xffffffff850d13fe <filldir64+0x1e>: cmp %r13d,0x20(%rdi)
0xffffffff850d1402 <filldir64+0x22>: jl 0xffffffff850d14f9 <filldir64+0x119>
0xffffffff850d1408 <filldir64+0x28>: mov %rsi,%r15
0xffffffff850d140b <filldir64+0x2b>: mov 0x18(%rdi),%rsi
0xffffffff850d140f <filldir64+0x2f>: mov %rdi,%rbp
0xffffffff850d1412 <filldir64+0x32>: test %rsi,%rsi
0xffffffff850d1415 <filldir64+0x35>: je 0xffffffff850d143f <filldir64+0x5f>
0xffffffff850d1417 <filldir64+0x37>: mov %gs:0x15c80,%rax
0xffffffff850d1420 <filldir64+0x40>: mov (%rax),%rax
0xffffffff850d1423 <filldir64+0x43>: test $0x4,%al
0xffffffff850d1425 <filldir64+0x45>: jne 0xffffffff850d1500 <filldir64+0x120>
0xffffffff850d142b <filldir64+0x4b>: stac
0xffffffff850d142e <filldir64+0x4e>: xor %eax,%eax
0xffffffff850d1430 <filldir64+0x50>: mov %rcx,0x8(%rsi)
0xffffffff850d1434 <filldir64+0x54>: clac
0xffffffff850d1437 <filldir64+0x57>: test %eax,%eax
0xffffffff850d1439 <filldir64+0x59>: jne 0xffffffff850d14e2 <filldir64+0x102>
0xffffffff850d143f <filldir64+0x5f>: mov 0x10(%rbp),%r12
0xffffffff850d1443 <filldir64+0x63>: stac
0xffffffff850d1446 <filldir64+0x66>: xor %eax,%eax
0xffffffff850d1448 <filldir64+0x68>: mov %r8,(%r12)
0xffffffff850d144c <filldir64+0x6c>: clac
0xffffffff850d144f <filldir64+0x6f>: test %eax,%eax
0xffffffff850d1451 <filldir64+0x71>: jne 0xffffffff850d14e2 <filldir64+0x102>
0xffffffff850d1457 <filldir64+0x77>: stac
0xffffffff850d145a <filldir64+0x7a>: movq $0x0,0x8(%r12)
0xffffffff850d1463 <filldir64+0x83>: clac
0xffffffff850d1466 <filldir64+0x86>: test %eax,%eax
0xffffffff850d1468 <filldir64+0x88>: jne 0xffffffff850d14e2 <filldir64+0x102>
0xffffffff850d146a <filldir64+0x8a>: stac
0xffffffff850d146d <filldir64+0x8d>: mov %r13w,0x10(%r12)
0xffffffff850d1473 <filldir64+0x93>: clac
0xffffffff850d1476 <filldir64+0x96>: test %eax,%eax
0xffffffff850d1478 <filldir64+0x98>: jne 0xffffffff850d14e2 <filldir64+0x102>
0xffffffff850d147a <filldir64+0x9a>: stac
0xffffffff850d147d <filldir64+0x9d>: mov %eax,%ebx
0xffffffff850d147f <filldir64+0x9f>: mov %r9b,0x12(%r12)
0xffffffff850d1484 <filldir64+0xa4>: clac
0xffffffff850d1487 <filldir64+0xa7>: test %ebx,%ebx
0xffffffff850d1489 <filldir64+0xa9>: jne 0xffffffff850d14e2 <filldir64+0x102>
0xffffffff850d148b <filldir64+0xab>: movslq %edx,%r14
0xffffffff850d148e <filldir64+0xae>: mov %r15,%rdi. ii> r15 is the first parameter when calling __check_object_size
0xffffffff850d1491 <filldir64+0xb1>: mov $0x1,%edx
0xffffffff850d1496 <filldir64+0xb6>: mov %r14,%rsi
0xffffffff850d1499 <filldir64+0xb9>: call 0xffffffff850b47c0 <__check_object_size>
0xffffffff850d149e <filldir64+0xbe>: mov %r14,%rdx
Up to this point, r15 is the given object for __check_object_size to validate.
And no one change the r15 until the end, so we can get the given object from stack of usercopy_abort i.e., R15: ff3bb03416f1c4fd.
I check isfepi and the give object by using ikmemi
crash-8.0.2> kmem ff3bb03416f1c4fa
CACHE OBJSIZE ALLOCATED TOTAL SLABS SSIZE NAME
ff3bb00107c0fa00 16 2074327 2860288 11173 4k kmalloc-16
SLAB MEMORY NODE TOTAL ALLOCATED FREE
ff8faa00d05bc700 ff3bb03416f1c000 0 256 76 180
FREE / [ALLOCATED]
ff3bb03416f1c4f0 (cpu 4 cache)
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
ff8faa00d05bc700 3416f1c000 ff3bb00107c0fa00 ff3bb03416f1ca00 1 17ffffc0000100 slab
crash-8.0.2> kmem ff3bb03416f1c4fd
CACHE OBJSIZE ALLOCATED TOTAL SLABS SSIZE NAME
ff3bb00107c0fa00 16 2074327 2860288 11173 4k kmalloc-16
SLAB MEMORY NODE TOTAL ALLOCATED FREE
ff8faa00d05bc700 ff3bb03416f1c000 0 256 76 180
FREE / [ALLOCATED]
ff3bb03416f1c4f0 (cpu 4 cache)
PAGE PHYSICAL MAPPING INDEX CNT FLAGS
ff8faa00d05bc700 3416f1c000 ff3bb00107c0fa00 ff3bb03416f1ca00 1 17ffffc0000100 slab
It seemed the object have been freed.
I have no idea what is going on here.
Sincerely hope that you lot give me some help.
Below is some information about my server.
# xfs_db -r /dev/nvme0n1
xfs_db> version
versionnum [0xbcb5+0x18a] = V5,NLINK,DIRV2,ATTR,ALIGN,LOGV2,EXTFLG,SECTOR,MOREBITS,ATTR2,LAZYSBCOUNT,PROJID32BIT,CRC,FTYPE
Linux kernel version is v4.18 and the OS is Centos7.
Best regards.