dmesg日志如下:
[172602.460977] Unable to handle kernel NULL pointer dereference at virtual address 0000000000000028
[172602.470254] Mem abort info:
[172602.473553] ESR = 0x96000007
[172602.477113] Exception class = DABT (current EL), IL = 32 bits
[172602.483544] SET = 0, FnV = 0
[172602.487110] EA = 0, S1PTW = 0
[172602.490769] Data abort info:
[172602.494158] ISV = 0, ISS = 0x00000007
[172602.498495] CM = 0, WnR = 0
[172602.501986] user pgtable: 64k pages, 48-bit VAs, pgdp = 00000000eb57799a
[172602.509178] [0000000000000028] pgd=0000203f3f0b0003, pud=0000203f3f0b0003, pmd=0000203fd64e0003, pte=0000000000000000
[172602.520275] Internal error: Oops: 96000007 [#1] SMP
...
[172602.627698] Process mkdir (pid: 102562, stack limit = 0x00000000336a1556)
[172602.634974] CPU: 64 PID: 102562 Comm: mkdir Kdump: loaded Tainted: G W OE 4.19.xxx #1
[172602.648038] Hardware name: PowerLeader PR210K/BC82AMDYC, BIOS KL4.41.102.BD.240118.R 01/18/2024
[172602.657214] pstate: 60400009 (nZCv daif +PAN -UAO)
[172602.662509] pc : __nfs3_proc_setacls+0x4c/0x328 [nfsv3]
[172602.668231] lr : nfs3_proc_setacls+0x34/0x50 [nfsv3]
[172602.673689] sp : fffff64fae247c00
[172602.677508] x29: fffff64fae247c00 x28: fffff6301ac90340
[172602.683315] x27: 0000000000000000 x26: 0000000000000000
[172602.689121] x25: 0000000056000000 x24: fffff62fe622b130
[172602.694926] x23: 0000000000000000 x22: fffff62fe6620f18
[172602.700732] x21: 0000000000000000 x20: 0000000000000000
[172602.706538] x19: 0000000000000000 x18: 0000000000000000
[172602.712343] x17: 0000000000000000 x16: ffff4a6e97ea9d50
[172602.718149] x15: 0000000000000000 x14: 32312d30302d3731
[172602.723955] x13: ffff000000000000 x12: ffffffffffffffff
[172602.729761] x11: 0000000000000030 x10: 0000000000000005
[172602.735567] x9 : 000000000000ffff x8 : fffffffffffffff0
[172602.741373] x7 : ffffffffffffffff x6 : fffff62fe598a324
[172602.747179] x5 : fffff64fae247c94 x4 : 0000000000000001
[172602.752985] x3 : fffff64fae247c50 x2 : fffff64fae247c78
[172602.758791] x1 : fffff64fae247c48 x0 : fffff64fae247c84
[172602.764598] Call trace:
[172602.767555] __nfs3_proc_setacls+0x4c/0x328 [nfsv3]
[172602.772930] nfs3_proc_setacls+0x34/0x50 [nfsv3]
[172602.778046] nfs3_proc_mkdir+0x134/0x1c8 [nfsv3]
[172602.783190] nfs_mkdir+0x68/0x190 [nfs]
[172602.787534] vfs_mkdir+0x130/0x1d8
[172602.791439] do_mkdirat+0xf4/0x110
[172602.795344] __arm64_sys_mkdirat+0x28/0x38
[172602.799944] el0_svc_common+0x78/0x130
[172602.804195] el0_svc_handler+0x38/0x78
[172602.808445] el0_svc+0x8/0x1b0
[172602.812006] Code: 910143a3 9101e3a2 a9007c1f 910123a1 (f94016a0)
[172602.818696] SMP: stopping secondary CPUs
[172602.827904] Starting crashdump kernel...
[172602.832332] Bye!faddr2line usr/lib/debug/lib/modules/4.19.90-25.17.v2101.osc.ky10.aarch64/kernel/fs/nfs/nfsv3.ko.debug __nfs3_proc_setacls+0x4c/0x328
__nfs3_proc_setacls+0x4c/0x328:
NFS_SB 于 /usr/src/debug/kernel-4.19.90/linux-4.19.90-25.17.v2101.osc.ky10.aarch64/./include/linux/nfs_fs.h:253
(已内连入)NFS_SERVER 于 /usr/src/debug/kernel-4.19.90/linux-4.19.90-25.17.v2101.osc.ky10.aarch64/./include/linux/nfs_fs.h:263
(已内连入)__nfs3_proc_setacls 于 /usr/src/debug/kernel-4.19.90/linux-4.19.90-25.17.v2101.osc.ky10.aarch64/fs/nfs/nfs3acl.c:161crash发生在__nfs3_proc_setacls() -> NFS_SERVER() -> NFS_SB(inode->i_sb)。
crash> struct inode -ox
struct inode {
...
[0x28] struct super_block *i_sb;
...i_sb的偏移量是0x28，和dmesg日志中的virtual address 0000000000000028对应，所以空指针的对象是struct inode。
再看一下栈信息:
crash> bt
PID: 102562 TASK: fffff6301ac90340 CPU: 64 COMMAND: "mkdir"
#0 [fffff64fae247650] machine_kexec at ffff4a6e97c02ecc
#1 [fffff64fae2476b0] __crash_kexec at ffff4a6e97d0fdc8
#2 [fffff64fae247820] crash_kexec at ffff4a6e97d0fed0
#3 [fffff64fae247850] die at ffff4a6e97bef514
#4 [fffff64fae247890] die_kernel_fault at ffff4a6e97c0a9c4
#5 [fffff64fae2478c0] __do_kernel_fault at ffff4a6e97c0a694
#6 [fffff64fae2478f0] do_page_fault at ffff4a6e98734814
#7 [fffff64fae2479e0] do_translation_fault at ffff4a6e98734d0c
#8 [fffff64fae247a10] do_mem_abort at ffff4a6e97be1284
#9 [fffff64fae247bf0] el1_ia at ffff4a6e97be310c
PC: ffff4a6e7e7e49a4 [__nfs3_proc_setacls+76]
LR: ffff4a6e7e7e520c [nfs3_proc_setacls+52]
SP: fffff64fae247c00 PSTATE: 60400009
X29: fffff64fae247c00 X28: fffff6301ac90340 X27: 0000000000000000
X26: 0000000000000000 X25: 0000000056000000 X24: fffff62fe622b130
X23: 0000000000000000 X22: fffff62fe6620f18 X21: 0000000000000000
X20: 0000000000000000 X19: 0000000000000000 X18: 0000000000000000
X17: 0000000000000000 X16: ffff4a6e97ea9d50 X15: 0000000000000000
X14: 32312d30302d3731 X13: ffff000000000000 X12: ffffffffffffffff
X11: 0000000000000030 X10: 0000000000000005 X9: 000000000000ffff
X8: fffffffffffffff0 X7: ffffffffffffffff X6: fffff62fe598a324
X5: fffff64fae247c94 X4: 0000000000000001 X3: fffff64fae247c50
X2: fffff64fae247c78 X1: fffff64fae247c48 X0: fffff64fae247c84
#10 [fffff64fae247c00] __nfs3_proc_setacls at ffff4a6e7e7e49a0 [nfsv3]
#11 [fffff64fae247cb0] nfs3_proc_setacls at ffff4a6e7e7e5208 [nfsv3]
#12 [fffff64fae247ce0] nfs3_proc_mkdir at ffff4a6e7e7e16d8 [nfsv3]
# 用bt -FF命令还能看到[fffff62fe622b130:dentry]
fffff64fae247cf0: [fffff62fe6620f18:nfs_inode_cache] [fffff62fe622b130:dentry]
#13 [fffff64fae247d30] nfs_mkdir at ffff4a6e7ee53904 [nfs]
#14 [fffff64fae247db0] vfs_mkdir at ffff4a6e97e9a404
#15 [fffff64fae247df0] do_mkdirat at ffff4a6e97e9fe20
#16 [fffff64fae247e40] __arm64_sys_mkdirat at ffff4a6e97e9fe64
#17 [fffff64fae247e60] el0_svc_common at ffff4a6e97bf81dc
#18 [fffff64fae247ea0] el0_svc_handler at ffff4a6e97bf82cc
#19 [fffff64fae247ff0] el0_svc at ffff4a6e97be4084
PC: 0000ffff151a9b74 LR: 00000000004026c4 SP: 0000ffffc1e4d900
X29: 0000ffffc1e4d900 X28: 0000000000000000 X27: 0000000000000001
X26: 00000000000001ed X25: 0000000000000000 X24: 00000000ffffffff
X23: 0000ffffc1e4dae0 X22: 0000000000402100 X21: 00000000000001ed
X20: 0000ffffc1e4fbee X19: 0000ffffc1e4da88 X18: 0000000000000a03
X17: 0000ffff151a9b60 X16: 00000000004191e0 X15: 0000ffff1531ccc0
X14: 0000000000000000 X13: 0000000000000000 X12: 00000000000003f3
X11: 0000000000000004 X10: 0101010101010101 X9: 00000000003fffff
X8: 0000000000000022 X7: 00000000ffffffff X6: 0000000000000000
X5: 00000000004020b8 X4: 00000000000001ed X3: 0000000000000001
X2: 00000000000001ed X1: 0000ffffc1e4fc23 X0: ffffffffffffff9c
ORIG_X0: ffffffffffffff9c SYSCALLNO: 22 PSTATE: 00001000反汇编nfs3_proc_mkdir函数:
crash> dis -lx nfs3_proc_mkdir
...
/usr/src/debug/kernel-4.19.90/linux-4.19.90-25.17.v2101.osc.ky10.aarch64/fs/nfs/nfs3proc.c: 560
0xffff4a6e7e7e16d0 <nfs3_proc_mkdir+0x128>: ldr x0, [x24,#48]
0xffff4a6e7e7e16d4 <nfs3_proc_mkdir+0x12c>: ldp x2, x1, [x29,#64]
0xffff4a6e7e7e16d8 <nfs3_proc_mkdir+0x130>: bl 0xffff4a6e7e7e51d8 <nfs3_proc_setacls>
0xffff4a6e7e7e16dc <nfs3_proc_mkdir+0x134>: mov w20, w0由于d_inode在struct dentry中的偏移量为48，所以可以确定x24寄存器这时存的是struct dentry的值，但后面x24寄存器的值有可能被改变，所以再看后面的函数反汇率:
crash> dis -lx nfs3_proc_setacls
/usr/src/debug/kernel-4.19.90/linux-4.19.90-25.17.v2101.osc.ky10.aarch64/fs/nfs/nfs3acl.c: 247
0xffff4a6e7e7e51d8 <nfs3_proc_setacls>: stp x29, x30, [sp,#-48]!
0xffff4a6e7e7e51dc <nfs3_proc_setacls+0x4>: mov x29, sp
0xffff4a6e7e7e51e0 <nfs3_proc_setacls+0x8>: stp x19, x20, [sp,#16]
0xffff4a6e7e7e51e4 <nfs3_proc_setacls+0xc>: str x21, [sp,#32]
0xffff4a6e7e7e51e8 <nfs3_proc_setacls+0x10>: mov x19, x0
0xffff4a6e7e7e51ec <nfs3_proc_setacls+0x14>: mov x20, x1
0xffff4a6e7e7e51f0 <nfs3_proc_setacls+0x18>: mov x0, x30
0xffff4a6e7e7e51f4 <nfs3_proc_setacls+0x1c>: mov x21, x2
0xffff4a6e7e7e51f8 <nfs3_proc_setacls+0x20>: nop
/usr/src/debug/kernel-4.19.90/linux-4.19.90-25.17.v2101.osc.ky10.aarch64/fs/nfs/nfs3acl.c: 249
0xffff4a6e7e7e51fc <nfs3_proc_setacls+0x24>: mov x2, x21
0xffff4a6e7e7e5200 <nfs3_proc_setacls+0x28>: mov x1, x20
0xffff4a6e7e7e5204 <nfs3_proc_setacls+0x2c>: mov x0, x19
0xffff4a6e7e7e5208 <nfs3_proc_setacls+0x30>: bl 0xffff4a6e7e7e4958 <__nfs3_proc_setacls>
/usr/src/debug/kernel-4.19.90/linux-4.19.90-25.17.v2101.osc.ky10.aarch64/fs/nfs/nfs3acl.c: 250
0xffff4a6e7e7e520c <nfs3_proc_setacls+0x34>: cmn w0, #0x5f
...
crash> dis -lx __nfs3_proc_setacls # 或 dis -rl __nfs3_proc_setacls+0x4c
/usr/src/debug/kernel-4.19.90/linux-4.19.90-25.17.v2101.osc.ky10.aarch64/fs/nfs/nfs3acl.c: 160
0xffff4a6e7e7e4958 <__nfs3_proc_setacls>: stp x29, x30, [sp,#-176]!
0xffff4a6e7e7e495c <__nfs3_proc_setacls+0x4>: mov x29, sp
0xffff4a6e7e7e4960 <__nfs3_proc_setacls+0x8>: stp x19, x20, [sp,#16]
0xffff4a6e7e7e4964 <__nfs3_proc_setacls+0xc>: stp x21, x22, [sp,#32]
0xffff4a6e7e7e4968 <__nfs3_proc_setacls+0x10>: str x23, [sp,#48]
0xffff4a6e7e7e496c <__nfs3_proc_setacls+0x14>: mov x21, x0
0xffff4a6e7e7e4970 <__nfs3_proc_setacls+0x18>: mov x19, x1
0xffff4a6e7e7e4974 <__nfs3_proc_setacls+0x1c>: mov x23, x2
0xffff4a6e7e7e4978 <__nfs3_proc_setacls+0x20>: mov x0, x30
0xffff4a6e7e7e497c <__nfs3_proc_setacls+0x24>: nop
/usr/src/debug/kernel-4.19.90/linux-4.19.90-25.17.v2101.osc.ky10.aarch64/fs/nfs/nfs3acl.c: 170
0xffff4a6e7e7e4980 <__nfs3_proc_setacls+0x28>: stp xzr, xzr, [x29,#88]
/usr/src/debug/kernel-4.19.90/linux-4.19.90-25.17.v2101.osc.ky10.aarch64/fs/nfs/nfs3acl.c: 164
0xffff4a6e7e7e4984 <__nfs3_proc_setacls+0x2c>: add x0, x29, #0x84
0xffff4a6e7e7e4988 <__nfs3_proc_setacls+0x30>: add x5, x29, #0x94
/usr/src/debug/kernel-4.19.90/linux-4.19.90-25.17.v2101.osc.ky10.aarch64/fs/nfs/nfs3acl.c: 170
0xffff4a6e7e7e498c <__nfs3_proc_setacls+0x34>: stp xzr, xzr, [x29,#104]
/usr/src/debug/kernel-4.19.90/linux-4.19.90-25.17.v2101.osc.ky10.aarch64/fs/nfs/nfs3acl.c: 164
0xffff4a6e7e7e4990 <__nfs3_proc_setacls+0x38>: mov w4, #0x1 // #1
0xffff4a6e7e7e4994 <__nfs3_proc_setacls+0x3c>: add x3, x29, #0x50
/usr/src/debug/kernel-4.19.90/linux-4.19.90-25.17.v2101.osc.ky10.aarch64/fs/nfs/nfs3acl.c: 170
0xffff4a6e7e7e4998 <__nfs3_proc_setacls+0x40>: add x2, x29, #0x78
/usr/src/debug/kernel-4.19.90/linux-4.19.90-25.17.v2101.osc.ky10.aarch64/fs/nfs/nfs3acl.c: 164
0xffff4a6e7e7e499c <__nfs3_proc_setacls+0x44>: stp xzr, xzr, [x0]
/usr/src/debug/kernel-4.19.90/linux-4.19.90-25.17.v2101.osc.ky10.aarch64/fs/nfs/nfs3acl.c: 170
0xffff4a6e7e7e49a0 <__nfs3_proc_setacls+0x48>: add x1, x29, #0x48
/usr/src/debug/kernel-4.19.90/linux-4.19.90-25.17.v2101.osc.ky10.aarch64/./include/linux/nfs_fs.h: 253
0xffff4a6e7e7e49a4 <__nfs3_proc_setacls+0x4c>: ldr x0, [x21,#40]
...x24没有被改变，所以崩溃时x24寄存器中存的就是struct dentry对象的值:
struct dentry fffff62fe622b130 # x24寄存器的值
struct dentry {
...
d_inode = 0x0,
...
}这也从另一个角度证明了nfs3_proc_setacls()和__nfs3_proc_setacls()函数中传入的struct inode为空指针。
patchset: nfs_instantiate() might succeed leaving dentry negative unhashed