Java Thread Dumps for Daemons With jstack

jstack is a really helpful utility that comes standard with most linux JDK versions. It allows you to generate java thread dumps in situations where kill -3 won't work. kill -3 (aka kill -QUIT) will dump a java process's threads to stderr — but this, of course, works only when you still have access to stderr. If you're running a java process as a daemon (like a jetty or tomcat or jboss etc server), stderr usually is inaccessible.

Fortunately, jstack allows you to generate thread dumps without needing to read stderr from the original java process. jstack will dump the threads to jstack's own stdout, which you can pipe to a file, or through a pager, or just send directly to your terminial. There a few tricks to using it, however, which don't seem to be documented anywhere:

1. Use the right jstack executable

jstack usually will work only if it came with the same exact JDK version as the target JVM process is running. Since a lot of servers end up having several different JVM versions installed on them, it's important to make sure that the version of jstack you're trying to use is the right one — the jstack executable at /usr/bin/jstack won't necessarily be correct. And since jstack doesn't accept a -version flag, it's pretty hard to tell which version /usr/bin/jstack actually is.

So the most reliable way to run jstack is from the bin directory of the JDK which you're using to run the target JVM process. On ubuntu, this usually will be a subdirectory of one of the JDKs in the /usr/lib/jvm directory (like /usr/lib/jvm/java-6-openjdk-amd64 for the 64-bit version of the java 6 JDK). In that case, you might run jstack like this (when the target JVM's process ID is 12345):

/usr/lib/jvm/java-6-openjdk-amd64/bin/jstack 12345

2. Run jstack as the same user as the target JVM

You need to run jstack as the same user as which the target JVM is running. For example, if you're running jetty as a user named jetty, (and the jetty process ID is 12345) use sudo to execute jstack as the jetty user:

sudo -u jetty jstack 12345

(I learned this trick from Michael Moser's jstack - the missing manual blog post — apparently jstack uses a named pipe to communicate with the target JVM process, and that pipe's permissions allow only the user who created the target JVM process to read or write the pipe.)

3. Try, try again

Sometimes, even if you do those first two things, jstack will still tell you to go get bent (or some other inscrutable error message of similar intent). I've found that if I just try running it again a couple of times, jstack magically will work on the second or third try.

4. Don't use -F

Even though jstack sometimes itself will suggest that you try -F (particularly if you've got a version mismatch between jstack and the target JVM), resist the temptation to "force" it. When you use jstack with the -F option, jstack will actually stop the target process (ie kill -STOP). Only use the -F option if your app is already good and hung (because it certainly will be once you use -F).