Welcome back to my series on ebpf. In the last post, I told you about writing eBPF applications in pure Java using my new Java compiler plugin. This week, we’re extending the XDP example from last week (and Hello eBPF: XDP-based Packet Filter (9)) to also capture outgoing packets using a Traffic Control (TC) classifier.

Before we go into the details, first, the demo of the PacketLogger:

The logger captures the incoming and outgoing IP packets with their IP address, their protocol (TCP, UDP, OTHER), the TCP/UDP port, and the packet length. But before I show you how I implemented all this in Java, here is a short introduction to the Linux network stack:

Linux Network Stack

When a packet comes over the internet into our system, it first enters via the network devices. Our XDP hook runs on the network device driver to check the incoming packets as early as possible. The packet is then passed onto the Linux network stack and is checked by the TC classifiers:

Foreshadowing for next week: For a firewall, we want to check the incoming packets as early as possible to reduce the load of incoming packets on the system, off-loading the checking from the CPU to the network device where possible. It’s not as important for outgoing packets as the CPU generates the packages.

For our example, we chose to log the incoming packets at the XDP and outgoing packets at the TC levels. We can’t use XDP on outgoing packets, because it is ingress only. To learn more about the differences between TC and XDP hooks, I recommend reading Hangbin Liu’s article tc/BPF and XDP/BPF.

TC can be used for many more:

TC can be used for a number of use cases, all of them having to do with the manipulation of traffic. TC is for example used to implement QoS (Quality of Service) allowing latency sensitive traffic like VoIP (Voice over IP) to be processed ahead of lets say web traffic. It can also drop packets to simulate packet-loss, add latency to simulate distant clients or apply bandwidth limitations for applications or users, to name a few.

TC allows an admin to filter traffic using a hierarchical model of qdiscs (Queuing DISCipline). A root qdisc is attached to a network interface with certain actions. This qdisc can also have child qdiscs which will be used over the root if their filter matches the traffic. This program type allows us to implement such a filter in eBPF.

ebPF-Docs

Now to it’s usage in Java:

TC and XDP Interface

The TC interface is similar to the XDP interface (as seen in Hello eBPF: Write your eBPF application in Pure Java (12)):

public interface TCHook {

    @BPFFunction(section = "tc")
    @NotUsableInJava
    default sk_action tcHandleIngress(Ptr<__sk_buff> packet) {
        return sk_action.SK_PASS;
    }

    @BPFFunction(section = "tc")
    @NotUsableInJava
    default sk_action tcHandleEgress(Ptr<__sk_buff> packet) {
        return sk_action.SK_PASS;
    }
}

Instances of sk_buff/__sk_buff (the latter is the stable version) are allocated when the packet first enters the network stack, both from the direction of the application and the direction of the network device:

@Type(
      noCCodeGeneration = true,
      cType = "struct __sk_buff"
  )
  @NotUsableInJava
  public static class __sk_buff extends Struct {
    public @Unsigned int len;

    public @Unsigned int pkt_type;

    public @Unsigned int mark;

    public @Unsigned int queue_mapping;

    public @Unsigned int protocol;

    public @Unsigned int vlan_present;
    
    // ...
}

The fields are populated during the packet’s passage through the network stack, so not all fields are accessible in every hook. In the TC hook, we can only access a subset:

And many more fields, more in ebpf-docs.

The most important fields for our use case are the data and data_end fields because they point to the same data as the equally named fields in the XDP context.

Now to the implementation:

Implementation in Java

We first define the stored packet info and the ring buffer:

@Type
enum PacketDirection implements Enum<PacketDirection> {
    INCOMING, OUTGOING
}

@Type
enum Protocol implements Enum<Protocol> {
    TCP, UDP, OTHER
}

@Type
static class PacketInfo {
    PacketDirection direction;
    Protocol protocol;
    IPAddress source;
    IPAddress destination;
    int port;
    int length;
}

@BPFMapDefinition(maxEntries = 4096 * 256)
BPFRingBuffer<PacketInfo> packetLog;

We then implement the XDPHook and TCHook interfaces:

@Override
public xdp_action xdpHandlePacket(Ptr<xdp_md> ctx) {
    handlePacket(PacketDirection.INCOMING,
            Ptr.voidPointer(ctx.val().data),
            Ptr.voidPointer(ctx.val().data_end));
    return xdp_action.XDP_PASS;
}

@Override
public sk_action tcHandleEgress(Ptr<__sk_buff> packet) {
    handlePacket(PacketDirection.OUTGOING,
            Ptr.voidPointer(packet.val().data),
            Ptr.voidPointer(packet.val().data_end));
    return sk_action.SK_PASS;
}

Both call the same handlePacket method, making the implementation far simpler. The method first allocates a PacketInfo instance on the stack parses the packet, collects all the info, and then stores the PacketInfo instance in the ring buffer:

@BPFFunction
@AlwaysInline
void handlePacket(PacketDirection direction, Ptr<?> start, Ptr<?> end) {
    PacketInfo info = new PacketInfo();
    info.direction = direction;
    if (parsePacket(start, end, Ptr.of(info))) {
        var ptr = packetLog.reserve();
        if (ptr != null) {
            ptr.set(info);
            packetLog.submit(ptr);
        }
    }
}

To attach our hooks to all non-internal/loopback interfaces and print the packet log, we just call the appropriate methods in the user-land:

public static void main(String[] args) throws InterruptedException {
    try (PacketLogger program = BPFProgram.load(PacketLogger.class)) {

        // set the ring buffer call back
        program.packetLog.setCallback((info) -> {
            if (info.direction == PacketDirection.INCOMING) {
                System.out.print("Incoming from " +
                        NetworkUtil.intToIpAddress(info.source.ipv4)
                                .getHostAddress());
            } else {
                System.out.print("Outgoing to   " +
                        NetworkUtil.intToIpAddress(info.destination.ipv4)
                                .getHostAddress());
            }
            System.out.printf(" protocol %s port %5d length %d%n",
                    info.protocol, info.port, info.length);
        });

        // attach the XDP and TC hooks
        program.xdpAttach();
        program.tcAttachEgress();

        // consume the packet log
        while (true) {
            program.consumeAndThrow();
            Thread.sleep(500);
        }
    }
}

I will spare you the code for parsing the packets themselves; if you’re interested, look at the code on GitHub. Parsing the packets, especially IPv6 packets, is more complicated than one might think, so the parsing code is a Java adaptation of the sample code from the xdp-project.

Conclusion

TC and XDP hooks allow us to monitor incoming and outgoing packets easily. With hello-ebpf, it is possible to write all this in pure Java, defining all data structures once and using them both in user and kernel land. Of course, all of this is a work in progress, and I had found a few bugs while implementing the largest program yet with my eBPF library. The resulting C code has around 150 lines of code (excluding blank lines).

But this is only the beginning. Join me next week when I extend the PacketLogger into a proper Firewall written purely in Java.

This article is part of my work in the SapMachine team at SAP, making profiling and debugging easier for everyone. Thanks to Dylan Reimerink for answering all my ebpf-related questions; the whole hello-ebpf project would have been much harder without your support.