While running final tests, we noticed extremely erratic runtime behavior. This was difficult to reproduce, but very frustrating because it meant the application runtime increased from 5 to almost 14 minutes. This article describes how we diagnosed the issue and distilled it down to a few test cases. These might help us find some answers. Perhaps we’ve discovered a defect or at least a strange edge case in the JVM.

When our Java code processes certain data first, it compiles machine code that contains many uncommon trap markers, which seem to affect runtime behavior (this is just an observation, not a firm conclusion). When different data is processed later, the JIT compiler sticks with its initial compilation, and the runtime suffers by a factor of 2 to 5. If we change the order of the data, the runtime behavior improves dramatically, and even the data that previously led to the potentially poor code now runs fine.

We can change the code to mitigate the problem, but we are still interested in understanding why Java’s JIT compiler reaches its conclusions and persists with them. The code in question is compatible with almost any Java version and doesn’t use any advanced features.