The eBPF Verifier Errors Project

The eBPF verifier is a vital part of the eBPF stack. The verifier checks all eBPF bytecode that is loaded into the kernel:

Courtesy of Mohammed Aboullaite

This ensures that the bytecode is valid, that it doesn’t access potentially unsafe memory (like dereferencing null pointers), and that all programs terminate. This is vital for safe-guarding the kernel against common issues.

The only problem? The verifier output can be quite hard to understand. Consider for example the following verifier error log which only lists the bytecode:

libbpf: prog 'enterOpenat2': -- BEGIN PROG LOAD LOG --
0: R1=ctx() R10=fp0
;  @ <stdin>:15
0: (b7) r1 = 0                        ; R1_w=0
;  @ <stdin>:16
1: (7b) *(u64 *)(r10 -8) = r1         ; R1_w=0 R10=fp0 fp-8_w=0
2: (7b) *(u64 *)(r10 -16) = r1        ; R1_w=0 R10=fp0 fp-16_w=0
3: (bf) r6 = r10                      ; R6_w=fp0 R10=fp0
4: (07) r6 += -16                     ; R6_w=fp-16
;  @ <stdin>:17
5: (bf) r1 = r6                       ; R1_w=fp-16 R6_w=fp-16
6: (b7) r2 = 16                       ; R2_w=16
7: (85) call bpf_get_current_comm#16          ; R0_w=scalar() fp-8_w=mmmmmmmm fp-16_w=mmmmmmmm
;  @ <stdin>:18
8: (18) r1 = 0xffff95b5c6d05800       ; R1_w=map_ptr(map=map,ks=16,vs=4)
10: (bf) r2 = r6                      ; R2_w=fp-16 R6_w=fp-16
11: (85) call bpf_map_lookup_elem#1   ; R0=map_value_or_null(id=1,map=map,ks=16,vs=4)
;  @ <stdin>:19
12: (61) r1 = *(u32 *)(r0 +0)
R0 invalid mem access 'map_value_or_null'
processed 12 insns (limit 1000000) max_states_per_insn 0 total_states 1 peak_states 1 mark_read 1
-- END PROG LOAD LOG --

For the related program:

struct {
    __uint (type, BPF_MAP_TYPE_HASH);
    __uint (key_size, sizeof(u8[16]));
    __uint (value_size, sizeof(u32));
    __uint (max_entries, 256);
} map SEC(".maps");

// ...

u32 *counter = bpf_map_lookup_elem(&comm);
*(counter) = (*(counter)) + 1;

It’s not entirely obvious what the error is and how to resolve it, especially for people new to eBPF.

This is why I’m starting eBPF Verifier Errors Project:

A project that aims to collect as many eBPF verifier logs as possible, with the related source code, a description of the issue, and how to resolve them. This creates a database where people can look up verifier errors and possibly use it as a data source for developer tooling (like AI chatbots).

The errors are collected in the form of issues in the ebpf-verifier-errors repository on GitHub. You can obtain a list on the command line via:

curl -s -H "https://api.github.com/repos/parttimenerd/ebpf-verifier-errors/issues?labels=submission" \
     | jq -r '.[] | "\(.title)\n\(.body)\n-----"'

How to Submit

Please consider submitting your verifier errors today. It’s not hard; just fill out the issue form, and you’re done:

Any submission counts. We use GitHub issues because it makes submitting as easy as possible. No need to clone the repository and submit a pull request. Feel free to submit C, Rust, and Java code. If you find a submission that can improved, just add a comment.

If you’re unsure what a submission should look like, there are already five submissions by Dylan Reimerink and me, and possibly many more when you read this:

Conclusion

Verifier errors can be cryptic and frustrating, but we can change that. Let’s make them less scary and enrich the eBPF ecosystem. I’m happy that Dylan of ebpf-docs fame has agreed to join this worthy cause.

I pledge to contribute all eBPF verifier errors that I encounter and I look forward to your contributions.

This article is part of my work in the SapMachine team at SAP, making profiling and debugging easier for everyone.

Author

  • Johannes Bechberger

    Johannes Bechberger is a JVM developer working on profilers and their underlying technology in the SapMachine team at SAP. This includes improvements to async-profiler and its ecosystem, a website to view the different JFR event types, and improvements to the FirefoxProfiler, making it usable in the Java world. He started at SAP in 2022 after two years of research studies at the KIT in the field of Java security analyses. His work today is comprised of many open-source contributions and his blog, where he writes regularly on in-depth profiling and debugging topics, and of working on his JEP Candidate 435 to add a new profiling API to the OpenJDK.

    View all posts

New posts like these come out at least every two weeks, to get notified about new posts, follow me on Twitter, Mastodon, or LinkedIn, or join the newsletter:

Leave a Reply

Your email address will not be published. Required fields are marked *