Hot-patching the JVM to hook native Java agents

Over a year ago, I wrote a blog post called Who instruments the instrumenters? together with Mikaël Francoeur on how we debugged the Java instrumentation code. In the meantime, I gave a more detailed talk on this topic at VoxxedDays Amsterdam. The meta-agent that I developed for this worked well for Java agents/instrumenters, but what about native agents? Marco Sussitz found my agent and asked exactly this question. Native agents are agents that utilize the JVMTI API to, for example, modify class bytecode; however, they are not written in Java. With this blog post, I’m proud to announce that the meta-agent now supports instrumenting native agents.

TL;DR: Meta-agent allows you to see how an agent, native or Java, transforms bytecode.

There are many examples of native agents, like DynaTrace‘s monitoring agent or async-profiler‘s method tracer. I’m using the latter in my example here, as it’s open-source and readily available. The method tracer instruments the Java bytecode to trace the execution time of specific methods. You can find more about it in the async-profiler forum.

As a sample program, we use Loop.java:

public class Loop {
    public static void main(String[] args) 
      throws InterruptedException {
        while (true) Thread.sleep(1000);
    }
}

Let’s trace the Thrread.sleep method and use the meta-agent to see what async-profiler does with the bytecode:

java -agentpath:native/libnative_agent.dylib \
     -javaagent:target/meta-agent.jar=server \
     -agentpath:libasyncProfiler.dylib=start,trace=java.lang.Thread.sleep,file=duration.html \
     Loop.java

This opens a server at localhost:7071 and we check how async-profiler modified the Thread class:

So we can now instrument native agents like any other Java agent. And the part: As all Java agents are built on top of the libinstrument native agent, we can also see what any Java agent is doing. For example, we can see that the Java instrumentation agent instruments itself:

So I finally built an instrumenter that can essentially instrument my instrumentation agent, which in turn instruments other instrumentation agents. Another benefit is that the instrumenter can find every modification of any Java agent.

Continue reading →