kernel trace with "General protection fault in user access. Non-canonical address?"

[alban]

On newer kernels, traceloop sometimes triggers a kernel stack to be printed in dmesg:

kernel: General protection fault in user access. Non-canonical address?

Details

kernel: ------------[ cut here ]------------
kernel: General protection fault in user access. Non-canonical address?
kernel: WARNING: CPU: 1 PID: 4564 at ../../../../../../../../usr/src/linux-5.4.16-coreos/arch/x86/mm/extable.c:126 ex_handler_uaccess+0x4d/0x60
kernel: Modules linked in: veth xt_statistic xt_nat ip_vs_sh ip_vs_wrr ip_vs_rr ip_vs ip6table_nat ip6_tables iptable_mangle xt_comment xt_mark xt_conntrack xt_MASQUERADE nf_conntrack_netlink nfnetlink xfrm_u>
kernel: CPU: 1 PID: 4564 Comm: traceloop Not tainted 5.4.16-flatcar #1
kernel: Hardware name: Amazon EC2 t3.small/, BIOS 1.0 10/16/2017
kernel: RIP: 0010:ex_handler_uaccess+0x4d/0x60
kernel: Code: 83 c4 08 b8 01 00 00 00 5b c3 80 3d f4 d1 48 01 00 75 dc 48 c7 c7 a0 72 15 90 48 89 34 24 c6 05 e0 d1 48 01 01 e8 6c 11 01 00 <0f> 0b 48 8b 34 24 eb bd 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44
kernel: RSP: 0018:ffffa13f80bfb9e8 EFLAGS: 00010282
kernel: RAX: 0000000000000000 RBX: ffffffff8fa0272c RCX: 0000000000000000
kernel: RDX: 000000000000003f RSI: ffffffff9344475f RDI: 0000000000000246
kernel: RBP: 000000000000000d R08: ffffffff93444720 R09: 000000000000003f
kernel: R10: 0000000000000000 R11: 00000000000011d4 R12: 0000000000000000
kernel: R13: 0000000000000000 R14: 0000000000000000 R15: 0000000000000000
kernel: FS:  00007f394bb2c700(0000) GS:ffff93867b500000(0000) knlGS:0000000000000000
kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
kernel: CR2: 0000000000000000 CR3: 00000000731f6003 CR4: 00000000007606e0
kernel: DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
kernel: DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
kernel: PKRU: 55555554
kernel: Call Trace:
kernel:  fixup_exception+0x43/0x56
kernel:  do_general_protection+0x49/0x150
kernel:  general_protection+0x28/0x30
kernel: RIP: 0010:copy_user_generic_unrolled+0x86/0xc0
kernel: Code: 4c 8b 5e 38 4c 89 47 20 4c 89 4f 28 4c 89 57 30 4c 89 5f 38 48 8d 76 40 48 8d 7f 40 ff c9 75 b6 89 d1 83 e2 07 c1 e9 03 74 12 <4c> 8b 06 4c 89 07 48 8d 76 08 48 8d 7f 08 ff c9 75 ee 21 d2 74 10
kernel: RSP: 0018:ffffa13f80bfbaf8 EFLAGS: 00050202
kernel: RAX: 0000000000000002 RBX: ffff9386169c0000 RCX: 0000000000000001
kernel: RDX: 0000000000000000 RSI: 0100000000000000 RDI: ffffa13f80bfbc90
kernel: RBP: 0000000000000008 R08: ffffffff8f18840a R09: ffff93867572b000
kernel: R10: ffff93867572b630 R11: ffff93867adad000 R12: 00007ffffffff000
kernel: R13: 0100000000000000 R14: ffffa13f80bfbc90 R15: 0000000000000001
kernel:  ? ___bpf_prog_run+0x94a/0x13b0
kernel:  __probe_kernel_read+0x54/0x80
kernel:  bpf_probe_read+0x2d/0x60
kernel:  ___bpf_prog_run+0xa01/0x13b0
kernel:  __bpf_prog_run512+0x3e/0x60
kernel:  ? ___bpf_prog_run+0x94a/0x13b0
kernel:  ? __switch_to_asm+0x40/0x70
kernel:  ? __switch_to_asm+0x34/0x70
kernel:  ? __switch_to_asm+0x40/0x70
kernel:  ? __switch_to_asm+0x34/0x70
kernel:  ? __switch_to_asm+0x40/0x70
kernel:  ? __switch_to_asm+0x34/0x70
kernel:  ? __switch_to_asm+0x40/0x70
kernel:  ? __switch_to_asm+0x40/0x70
kernel:  ? __switch_to_asm+0x40/0x70
kernel:  ? __switch_to_asm+0x34/0x70
kernel:  ? __switch_to_asm+0x40/0x70
kernel:  ? __switch_to_asm+0x34/0x70
kernel:  ? __switch_to_asm+0x40/0x70
kernel:  ? __switch_to_asm+0x34/0x70
kernel:  ? legitimize_path.isra.44+0x2d/0x60
kernel:  ? unlazy_walk+0x42/0x70
kernel:  ? terminate_walk+0x7a/0xe0
kernel:  ? path_lookupat.isra.50+0xa3/0x220
kernel:  ? __switch_to_asm+0x40/0x70
kernel:  ? alloc_htab_elem+0x180/0x280
kernel:  trace_call_bpf+0x82/0x100
kernel:  ? htab_map_update_elem+0x25b/0x450
kernel:  perf_trace_run_bpf_submit+0x4e/0xc0
kernel:  perf_trace_sys_enter+0xf7/0x160
kernel:  syscall_trace_enter+0x2a9/0x2c0
kernel:  do_syscall_64+0xdb/0x120
kernel:  entry_SYSCALL_64_after_hwframe+0x44/0xa9
kernel: RIP: 0033:0x4b07f0
kernel: Code: 8b 7c 24 10 48 8b 74 24 18 48 8b 54 24 20 49 c7 c2 00 00 00 00 49 c7 c0 00 00 00 00 49 c7 c1 00 00 00 00 48 8b 44 24 08 0f 05 <48> 3d 01 f0 ff ff 76 20 48 c7 44 24 28 ff ff ff ff 48 c7 44 24 30
kernel: RSP: 002b:000000c001445690 EFLAGS: 00000206 ORIG_RAX: 000000000000010d
kernel: RAX: ffffffffffffffda RBX: 000000c00004a000 RCX: 00000000004b07f0
kernel: RDX: 0000000000000000 RSI: 000000c001a9647a RDI: ffffffffffffff9c
kernel: RBP: 000000c0014456e8 R08: 0000000000000000 R09: 0000000000000000
kernel: R10: 0000000000000000 R11: 0000000000000206 R12: ffffffffffffffff
kernel: R13: 0000000000000088 R14: 0000000000000087 R15: 0000000000000200
kernel: ---[ end trace a69fe79250905b65 ]---

For more information:

• torvalds/linux@00c4237 (warning added in Linux 5.1)
• torvalds/linux@6ae08ae (BPF helpers bpf_probe_read_{user, kernel}() added in Linux 5.5)

Traceloop should not use bpf_probe_read() but should use the new bpf_probe_read_{user, kernel}() when possible, depending on the Linux version (see kernel-versions.md). I'm hoping it will help, but I have not tried it.

Gobpf currently does not allow to load different implementation of a BPF program depending on the kernel version. However, @shang-wang wrote DataDog/gobpf#1 which could help with the function SetKprobeForSection(). It could be upstreamed ;) There are some examples how to use that function here.