The eBPF verifier is a vital part of the eBPF stack. The verifier checks all eBPF bytecode that is loaded into the kernel:

This ensures that the bytecode is valid, that it doesn’t access potentially unsafe memory (like dereferencing null pointers), and that all programs terminate. This is vital for safe-guarding the kernel against common issues.

The only problem? The verifier output can be quite hard to understand. Consider for example the following verifier error log which only lists the bytecode:

libbpf: prog 'enterOpenat2': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
;  @ <stdin>:15
0: (b7) r1 = 0                        ; R1_w=0
;  @ <stdin>:16
1: (7b) *(u64 *)(r10 -8) = r1         ; R1_w=0 R10=fp0 fp-8_w=0
2: (7b) *(u64 *)(r10 -16) = r1        ; R1_w=0 R10=fp0 fp-16_w=0
3: (bf) r6 = r10                      ; R6_w=fp0 R10=fp0
4: (07) r6 += -16                     ; R6_w=fp-16
;  @ <stdin>:17
5: (bf) r1 = r6                       ; R1_w=fp-16 R6_w=fp-16
6: (b7) r2 = 16                       ; R2_w=16
7: (85) call bpf_get_current_comm#16          ; R0_w=scalar() fp-8_w=mmmmmmmm fp-16_w=mmmmmmmm
;  @ <stdin>:18
8: (18) r1 = 0xffff95b5c6d05800       ; R1_w=map_ptr(map=map,ks=16,vs=4)
10: (bf) r2 = r6                      ; R2_w=fp-16 R6_w=fp-16
11: (85) call bpf_map_lookup_elem#1   ; R0=map_value_or_null(id=1,map=map,ks=16,vs=4)
;  @ <stdin>:19
12: (61) r1 = *(u32 *)(r0 +0)
R0 invalid mem access 'map_value_or_null'
processed 12 insns (limit 1000000) max_states_per_insn 0 total_states 1 peak_states 1 mark_read 1
-- END PROG LOAD LOG --

For the related program:

struct {
    __uint (type, BPF_MAP_TYPE_HASH);
    __uint (key_size, sizeof(u8[16]));
    __uint (value_size, sizeof(u32));
    __uint (max_entries, 256);
} map SEC(".maps");

// ...

u32 *counter = bpf_map_lookup_elem(&comm);
*(counter) = (*(counter)) + 1;

It’s not entirely obvious what the error is and how to resolve it, especially for people new to eBPF.

This is why I’m starting eBPF Verifier Errors Project:

A project that aims to collect as many eBPF verifier logs as possible, with the related source code, a description of the issue, and how to resolve them. This creates a database where people can look up verifier errors and possibly use it as a data source for developer tooling (like AI chatbots).

The errors are collected in the form of issues in the ebpf-verifier-errors repository on GitHub. You can obtain a list on the command line via:

curl -s -H "https://api.github.com/repos/parttimenerd/ebpf-verifier-errors/issues?labels=submission" \
     | jq -r '.[] | "\(.title)\n\(.body)\n-----"'

How to Submit

Please consider submitting your verifier errors today. It’s not hard; just fill out the issue form, and you’re done:

Any submission counts. We use GitHub issues because it makes submitting as easy as possible. No need to clone the repository and submit a pull request. Feel free to submit C, Rust, and Java code. If you find a submission that can improved, just add a comment.

If you’re unsure what a submission should look like, there are already five submissions by Dylan Reimerink and me, and possibly many more when you read this:

Conclusion

Verifier errors can be cryptic and frustrating, but we can change that. Let’s make them less scary and enrich the eBPF ecosystem. I’m happy that Dylan of ebpf-docs fame has agreed to join this worthy cause.

I pledge to contribute all eBPF verifier errors that I encounter and I look forward to your contributions.

This article is part of my work in the SapMachine team at SAP, making profiling and debugging easier for everyone.