Who killed the JVM? Attaching a debugger twice

A few weeks back, I told you about on-demand debugging in my Level-up your Java Debugging Skills with on-demand Debugging blog post, enabling you to delay a debugging session till:

  • You gave orders via jcmd (onjcmd=y option), a feature contributed by the SapMachine team
  • the program threw a specific exception (onthrow=<exception>)
  • The program threw an uncaught exception (onuncaught=y)

This is quite useful because the JDWP agent has to do substantial initialization before it can start listening for the attaching debugger:

The triggering event invokes the bulk of the initialization, including creation of threads and monitors, transport setup, and installation of a new event callback which handles the complete set of events.

Comment in JDWP-agent Source Code

Other things, like class loading, were slower with an attached debugger in older JDK versions (see JDK-8227269).

But what happens after you end the debugging session? Is your debugged program aborted, and if not, can you reattach your debugger at a later point in time? The answer is as always: It depends. Or, more precisely: It depends on the remote debugger you’re using and how you terminate the debugging session.

But why should you disconnect and then reattach a debugger? It allows you to not run the debugger during longer ignorable stretches of your application’s execution. The overhead of running the JDWP agent waiting for a connection is minimal compared to the plethora of events sent from the agent to the debugger during a debugging session (like class loading events, see A short primer on Java debugging internals).

Before we cover how to (re)attach a debugger in IDEs, we’ll see how this works on the JDWP/JDI level:

On JVM Level

The JDWP agent does not prevent the debugger from reattaching. There are two ways that Debugging sessions can be closed by the debugger: dispose and exit. Disposing of a connection via the JDWP Dispose command is the least intrusive way. This command is exposed to the debugger in JDI via the VirtualMachine#dispose() method:

Invalidates this virtual machine mirror. The communication channel to the target VM is closed, and the target VM prepares to accept another subsequent connection from this debugger or another debugger, including the following tasks:

Any current method invocations executing in the target VM are continued after the disconnection. Upon completion of any such method invocation, the invoking thread continues from the location where it was originally stopped.

JDI Documentation

This essentially means that disposing of a debugging connection does not prevent the currently debugged application from continuing to run.

The other way is using the exit command, exposed as VirtualMachine#exit(int exitCode):

Causes the mirrored VM to terminate with the given error code. All resources associated with this VirtualMachine are freed. If the mirrored VM is remote, the communication channel to it will be closed.

JDI DOCUMENTATION

This, of course, prevents the debugger from reattaching.

Reattaching with IDEs

NetBeans, IntelliJ IDEA, and Eclipse all support reattaching after ending a debugging session by just creating a new remote debugging session. Be aware that this only works straightforwardly when using remote debugging, as the local debugging UI is usually directly intertwined with the UI for running the application. I would recommend trying remote debugging once in a while, even when debugging on your local machine, to be able to use all the advanced features.

Terminating an Application with IDEs

NetBeans is the only IDE of the three that does not support this (as far as I can ascertain). IntelliJ IDEA and Eclipse both support it, with Eclipse having the more straight-forward UI:

If the terminate button is not active, then you might have to tick the Allow termination of remote VM check-box in the remote configuration settings:

IntelliJ IDEA’s UI is, in this instance, arguably less discoverable: To terminate the application, you have to close the specific debugging session tab explicitly.

This then results in a popup that offers you the ability to terminate:

Conclusion

The ability to disconnect and then reattach debuggers is helpful for many complex debugging scenarios and can help you debug faster. Being able to terminate the application directly from the debugger is an additional time saver when working with remote debugging sessions. Both are often overlooked gems of Java debugging, showing once more how versatile the JDWP agent and UI debuggers are.

I hope you enjoyed this addendum to my Level-up your Java Debugging Skills with on-demand Debugging blog post. If you want even more debugging from me, come to my talk on debugging at JUG Karlsruhe on the 7th of November, to the ConFoo conference in Montreal on the 23rd of February, and hopefully, next year a conference or user group near you.

This article is part of my work in the SapMachine team at SAP, making profiling and debugging easier for everyone. It was supported by rainy weather and the subsequent afternoon in a cafe in Bratislava:

JDWP, onthrow and a mysterious error

In my previous Java-related blog post called Level-up your Java Debugging Skills with on-demand Debugging, I showed you how to use the onthrow option of the JDWP agent to start the debugging session on the first throw of a specific exception. This gave us a mysterious error in JDB:

And I asked if somebody had any ideas. No one had, but I was at Devoxx Belgium and happened to talk with Aleksey Shipilev about it at the Corretto booth:

Two OpenJDK developers having fun at Devoxx: Investigating a jdb bug with Shipilev
in the Coretto booth (Tweet)

We got a rough idea of what was happening, and now that I’m back from Devoxx, I have the time to investigate it properly. But to recap: How can you use the onthrow option and reproduce the bug?

Recap

We use a simple example program with throws and catches the exception Ex twice:

public class OnThrowAndJCmd {

    public static void main(String[] args) throws InterruptedException {
        System.out.println("Hello world!");
        try {
            throw new Ex("");
        } catch (Ex e) {
            System.out.println("Caught");
        }
        try {
            throw new Ex("");
        } catch (Ex e) {
            System.out.println("Caught");
        }
        for (int i = 0; i < 1000; i++) {
            System.out.print(i + " ");
            Thread.sleep(2000);
        }
    }
}

class Ex extends RuntimeException {
    public Ex(String msg) {
        super(msg);
    }
}

We then use one terminal to run the program with the JDWP agent attached and the other to run JDB:

java "-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:5005,onthrow=Ex,launch=exit" src/test/java/OnThrowAndJCmd.java

# in another terminal
jdb -attach 5005

Then JDB prints us the expected error trace:

Exception in thread "event-handler" java.lang.NullPointerException: Cannot invoke "com.sun.jdi.ObjectReference.referenceType()" because the return value of "com.sun.jdi.event.ExceptionEvent.exception()" is null
        at jdk.jdi/com.sun.tools.example.debug.tty.TTY.exceptionEvent(TTY.java:171)
        at jdk.jdi/com.sun.tools.example.debug.tty.EventHandler.exceptionEvent(EventHandler.java:295)
        at jdk.jdi/com.sun.tools.example.debug.tty.EventHandler.handleEvent(EventHandler.java:133)
        at jdk.jdi/com.sun.tools.example.debug.tty.EventHandler.run(EventHandler.java:78)
        at java.base/java.lang.Thread.run(Thread.java:1583)

This might be, and I’m foreshadowing, the reason why IDEs like IntelliJ IDEA don’t support attaching to a JDWP agent with onthrow enabled.

Remember that this issue might be fixed with your current JDK; the bug is reproducible with a JDK build older than the 10th of October.

Update: This bug does not appear in JDK 1.4, but in JDK 1.5 and ever since.

Looking for the culprit

In our preliminary investigation, Aleksey and I realized that JDB was probably not to blame. The problem is that the JDWP-agent sends an exception event after JDB is attached, related to the thrown Ex exception, but this event does not adhere to the specification. The JDWP specification tells us that every exception event contains the following:

TypeNameDescription
intrequestIDRequest that generated event 
threadIDthreadThread with exception
locationlocationLocation of exception throw (or first non-native location after throw if thrown from a native method) 
tagged-objectIDexceptionThrown exception 
locationcatchLocationLocation of catch, or 0 if not caught. An exception is considered to be caught if, at the point of the throw, the current location is dynamically enclosed in a try statement that handles the exception. […]

So clearly, none of the properties should be null in our case. Exception events are written in the writeExceptionEvent method of the JDWP agent. We can modify this method to print all accessed exception fields and check that the problem really is related to the agent. For good measure, we also tell JDB to get notified of all other triggered Ex exceptions (> catch Ex), so we can obtain the printed information for the initial and the second exception:

Exception event: 
    thread: 0x0
    clazz: 0x0
    method: 0x0
    location: 0
    object: 0x0
    catch_clazz: 0x0
    catch_method: 0x0
    catch_location: 0
Caught
Exception event: 
    thread: 0x12b50fb02
    clazz: 0x12b50fb0a
    method: 0x12b188290
    location: 36
    object: 0x12b50fb12
    catch_clazz: 0x12b50fb1a
    catch_method: 0x12b188290
    catch_location: 37

This clearly shows that the exception that started the debugging session was not sent correctly.

How does onthrow work?

When the JDWP agent starts, it registers a JVMTI Exception event callback called cbEarlyException via SetEventCallBacks:

void JNICALL Exception(
  jvmtiEnv *jvmti_env,
  JNIEnv* jni_env, 
  jthread thread,
  jmethodID method, 
  jlocation location, 
  jobject exception, 
  jmethodID catch_method, 
  jlocation catch_location)

Exception events are generated whenever an exception is first detected in a Java programming language method.

JVMTI Documentation

On every exception, this handler checks if the exception has the name passed to the onthrow option. If the exception matches, then the agent initializes the debugging session:

The only problem here is that cbEarlyException is passed all the exception information but doesn’t pass it to the initialize method. This causes the JDWP-agent to send out an Exception event with all fields being null, as you saw in the previous section.

Fixing the bug

Now that we know exactly what went wrong, we can create an issue in the official JDK Bug System (JDK-8317920). Then, we can fix it by creating the event in the cbEarlyException handler itself and passing it to the new opt_info parameter of the initialize method (see GitHub):

static void JNICALL
cbEarlyException(jvmtiEnv *jvmti_env, JNIEnv *env,
        jthread thread, jmethodID method, jlocation location,
        jobject exception,
        jmethodID catch_method, jlocation catch_location)
{
    // ...
    EventInfo info;
    info.ei = EI_EXCEPTION;
    info.thread = thread;
    info.clazz = JNI_FUNC_PTR(env,GetObjectClass)(env, exception);
    info.method = method;
    info.location = location;
    info.object = exception;
    if (gdata->vthreadsSupported) {
        info.is_vthread = isVThread(thread);
    }
    info.u.exception.catch_clazz = getMethodClass(jvmti_env, catch_method);
    info.u.exception.catch_method = catch_method;
    info.u.exception.catch_location = catch_location;
    // ... // check if exception matches
    initialize(env, thread, EI_EXCEPTION, &info);
    // ...
}

The related Pull Request on GitHub is #16145. It will hopefully be merged soon. The last time someone reported and fixed an issue related to the onthrow option was in early 2002, so it is the first change in more than 20 years. The issue was about onthrow requiring the launch option to be present.

It works (even with your IDE)

With this fix in place, it works. JDB even selects the main thread as the current thread:

➜ jdb -attach 5005
Set uncaught java.lang.Throwable
Set deferred uncaught java.lang.Throwable
Initializing jdb ...
> 
Exception occurred: Ex (to be caught at: OnThrowAndJCmd.main(), line=7 bci=18)"thread=main", OnThrowAndJCmd.main(), line=6 bci=17

main[1]

But does fixing this issue also mean that IDEs like IntelliJ IDEA now support attaching to agents with onthrow enabled? Yes, at least if we set a breakpoint somewhere after the first exception has been thrown (like with the onjcmd option):

Conclusion

Collaborating with other people from different companies in an Open-Source project is great. Aleksey found the bug interesting enough to spend half an hour looking into it with me, which persuaded me to look into it again after returning from Devoxx. Fixing these bugs allows users to fully use on-demand debugging, speeding up their error-finding sessions.

I hope you liked this walk down the rabbit hole. See you next week with another article on debugging or my trip to Devoxx trip (or both). Feel free to ask any questions or to suggest new article ideas.

I’ll be giving a few talks on debugging this autumn; you can find specifics on my Talks page. I’m happy to speak on your meet-up or user group; just let me know.

This article is part of my work in the SapMachine team at SAP, making profiling and debugging easier for everyone.

Level-up your Java Debugging Skills with on-demand Debugging

Debugging is one of the most common tasks in software development, so one would assume that all features of debuggers have ample coverage in tutorials and guides. Yet there are three hidden gems of the Java Debugging (JDWP) agent that allow you to delay the start of the debugging session till

  • you gave orders via jcmd (onjcmd=y option)
  • the program threw a specific exception (onthrow=<exception>)
  • the program threw an uncaught exception (onuncaught=y)

Before I tell you more about the specific options, I want to start with the basics of how to apply them:

Option Application

When you debug remotely in your IDE (IntelliJ IDEA in my case), the “Debug Configurations” dialog tells you which options you should pass to your remote JVM:

Just append more options by adding them to the -agentlib option, or by setting the _JAVA_JDWP_OPTIONS environment variable, which is comma-appended to the options.

All options only work correctly in the server mode (server=y) of the JDWP agent (suspend=y or suspend=n seem to exhibit the same behavior with onjcmd).

I’m now showing you how the three hidden gems work:

JCmd triggered debugging

There are often cases where the code that you want to debug is executed later in your program’s run or after a specific issue appears. So don’t waste time running the debugging session from the start of your program, but use the onjcmd=y option to tell the JDWP agent to wait with the debugging session till it is triggered via jcmd:

 ➜ java "-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:5005,onjcmd=y" src/test/java/OnThrowAndJCmd.java &
 ➜ echo $! # get pid
 # wait some time and then start debugging on demand
 ➜ jcmd $! VM.start_java_debugging
jcmd 97145 VM.start_java_debugging
97145:
Debugging has been started.
Transport : dt_socket
Address : *:5005

jps is your friend if you want to find the process id of an already running JVM.

I created a sample class in my java-dbg repository on GitHub with a small sample program for this article. To use JCmd triggered with our IDE, we first have to create a remote debug configuration (see previous section); we can then start the sample program in the shell and trigger the start of the debugging session. Then, we start the remote debug configuration in the IDE and debug our program:

A similar feature long existed in the SAPJVM. In 2019 Christoph Langer from SAP decided to add it to the OpenJDK, where it was implemented in JDK 12 and has been there ever since. It is one of the many significant contributions of the SapMachine team.

Disclaimer: I’m part of this magnificent team, albeit not in 2019.

Exception triggered debugging

Far older than jcmd triggered are exception-triggered debugging sessions. There are two types:

  1. The throwing of a specific exception (byte-code or normal name, inner classes with $) can start the debugging session by using onthrow=<exception>. This is especially nice if you want to debug the cause of this specific exception. This feature can easily be used in combination with your favorite IDE.
  2. The existence of an uncaught exception can trigger the start of a debugging session by using onuncaught=y. The debugging context is your outermost main method, but it’s still helpful if you want to inspect the exception or the state of your application. A problem is that you cannot use the debuggers from IntelliJ IDEA or NetBeans to explore the context; you have to use the command line debugger jdb instead.

Due to historical reasons, you also have to supply a command that is executed when the debugging session starts via the launch option, but setting it to exit works just fine.

Using both trigger types is similar to the JCmd triggered debugging:

 ➜ java "-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:5005,onthrow=Ex,launch=exit" src/test/java/OnThrowAndJCmd.java
# or
 ➜ java "-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:5005,onuncaught=y,launch=exit" src/test/java/OnThrowAndJCmd.java

If you’re okay with using jdb, you can also use the launch option to call a script that starts jdb in a new tmux session, in our case, tmux_jdb.sh:

#!/bin/sh
tmux new-session -d -s jdb -- jdb -attach $2

We run our application using the JDWP agent with the onthrow=Ex,launch=sh tmux_jdb.sh option to start the jdb the first time the Ex exception is thrown and attach to the tmux session:

➜ java "-agentlib:jdwp=transport=dt_socket,server=y,suspend=y,address=*:5005,onthrow=Ex,launch=sh tmux_jdb.sh" src/test/java/OnThrowAndJCmd.java
# in another console after the exception is thrown
➜ tmux attach -t jdb

Where we can explore the current state of the application:

Debugging a specific exception has never been easier.

jdb and the JDWP on* options aren’t as widely used as graphical debuggers, so you might still find some bugs. I don’t know whether the stack trace in the second-to-last screenshot is a bug. Feel free to comment if you know the answer.

How to discover these features

You can either be like me and just drop into the JDK source and look into the debugInit.c file, the official documentation, or you use help option, which prints the following with JDK 21:

➜ java "-agentlib:jdwp=help"
               Java Debugger JDWP Agent Library
               --------------------------------

  (See the "VM Invocation Options" section of the JPDA
   "Connection and Invocation Details" document for more information.)

jdwp usage: java -agentlib:jdwp=[help]|[<option>=<value>, ...]

Option Name and Value            Description                       Default
---------------------            -----------                       -------
suspend=y|n                      wait on startup?                  y
transport=<name>                 transport spec                    none
address=<listen/attach address>  transport spec                    ""
server=y|n                       listen for debugger?              n
launch=<command line>            run debugger on event             none
onthrow=<exception name>         debug on throw                    none
onuncaught=y|n                   debug on any uncaught?            n
onjcmd=y|n                       start debug via jcmd?             n
timeout=<timeout value>          for listen/attach in milliseconds n
includevirtualthreads=y|n        List of all threads includes virtual threads as well as platform threads.
                                                                   n
mutf8=y|n                        output modified utf-8             n
quiet=y|n                        control over terminal messages    n

Obsolete Options
----------------
strict=y|n
stdalloc=y|n

Examples
--------
  - Using sockets connect to a debugger at a specific address:
    java -agentlib:jdwp=transport=dt_socket,address=localhost:8000 ...
  - Using sockets listen for a debugger to attach:
    java -agentlib:jdwp=transport=dt_socket,server=y,suspend=y ...

Notes
-----
  - A timeout value of 0 (the default) is no timeout.

Warnings
--------
  - The older -Xrunjdwp interface can still be used, but will be removed in
    a future release, for example:
        java -Xrunjdwp:[help]|[<option>=<value>, ...]

Of course, this only gives you a glance at the options, so reading the source code still revealed much of what I had before.

Conclusion

Hidden gems are everywhere in the Java ecosystem, even in widely used tools like debugging agents. Especially onthrow and onjcmd can improve the performance of on-demand debugging, as this allows us to trigger the start of the debugging session from outside the debugger.

I hope you can apply your newly gained knowledge the next time you have a complex problem to debug. Still curious about debugging? Come back next week for another blog post.

This article is part of my work in the SapMachine team at SAP, making profiling and debugging easier for everyone. Thanks to Thomas Darimont, with whom I discovered the hidden features while preparing for my talks on Java Debugging. I wrote this article on the train to Devoxx Belgium.