Path 1 · lesson 4 of 4
JIT compilation, profiling and warm-up
How HotSpot moves a method from the interpreter through C1 to C2, what profiling and inlining buy, why code is deoptimised, and why the first requests after a deploy are slow.
What you will learn
- Describe the five compilation levels and the usual path of a hot method through them
- Explain what the profile collected by the lower tiers is used for
- Read a compilation log line by line, including OSR and 'made not entrant'
- Explain why compiled code is thrown away and what that costs
- Explain why the first requests after a deploy are slow and choose a mitigation
- Recognise a full code cache and say what it does to the service
Before you start
The concept
A JVM does not run your code at one speed. Every method starts in the interpreter, which is slow but needs no preparation. While the code runs, the JVM counts calls and loop iterations. Methods that run often are compiled to machine code in the background, in two steps: quickly by the C1 compiler, then, using what was observed while the C1 code ran, thoroughly by the C2 compiler. This is tiered compilation, and it is why the same request can cost ten times more CPU in the first second of a process than in the tenth minute.
| Level | Runs as | Profiling | Purpose |
|---|---|---|---|
| 0 | Interpreter | Yes | Start at once. Find out which methods are worth compiling. |
| 1 | C1 code, fully optimised by C1 | None | Trivial methods such as getters, where C2 could not do better. |
| 2 | C1 code | Call and loop counters only | A cheaper stand-in for level 3 when the C2 queue is long. |
| 3 | C1 code | Full: types seen at each call site, branches taken, counters | Run several times faster than the interpreter while collecting the profile C2 needs. |
| 4 | C2 code | None | Peak speed. Optimised for what the profile says the program actually does. |
The lab program handles a small 'request': parse an order string into lines, price each line through an interface, add up. It runs 12 batches of 2,000 requests and prints the CPU time of each batch.
Program outputjava -Xmx128m JitWarmup.java warmup: CPU per request while the JIT works
Real JVM output$ java -Xmx128m JitWarmup.java warmup started at 304 ms after JVM start, 12 batches of 2,000 requests batch 1 done at 459 ms: 6,928 us of CPU ( 3.46 us per request) batch 2 done at 486 ms: 1,774 us of CPU ( 0.89 us per request) batch 3 done at 498 ms: 1,346 us of CPU ( 0.67 us per request) batch 4 done at 522 ms: 1,355 us of CPU ( 0.68 us per request) batch 5 done at 534 ms: 1,218 us of CPU ( 0.61 us per request) batch 6 done at 546 ms: 1,167 us of CPU ( 0.58 us per request) batch 7 done at 560 ms: 1,193 us of CPU ( 0.60 us per request) batch 8 done at 571 ms: 1,239 us of CPU ( 0.62 us per request) batch 9 done at 592 ms: 932 us of CPU ( 0.47 us per request) batch 10 done at 606 ms: 628 us of CPU ( 0.31 us per request) batch 11 done at 607 ms: 583 us of CPU ( 0.29 us per request) batch 12 done at 624 ms: 559 us of CPU ( 0.28 us per request) steady state: 200,000 more requests, 0.27 us of CPU per request checksum 11876704000
Why it exists
Warm-up is invisible on a long-running process and very visible around deploys. A new instance starts with nothing compiled. If it receives its full share of traffic at once, every request runs interpreted code, takes several times longer, and holds a thread for that long. With a rolling deploy this happens to each instance in turn; with autoscaling it happens exactly when load is highest.
- Latency spikes after every deploy. p99 jumps for the first seconds to minutes on each new instance, then settles without anyone doing anything.
- Timeouts and retries during rollouts. Slow first requests exceed client timeouts, retries add load to instances that are also cold, and thread pools fill. The case Checkout latency climbs shows what a full pool looks like from the outside.
- CPU throttling makes it worse. The compiler threads need CPU at the same moment as the cold requests. In a container with a low CPU limit they compete, and warm-up takes longer.
- Misleading measurements. A benchmark or load test that measures the first minute measures the interpreter and C1, not the code that runs in production an hour later.
The JIT also explains two rarer problems: a latency step in the middle of the day when the JVM deoptimises hot code because the program started to behave differently, and a service that becomes permanently slower when the code cache fills up and compilation stops.
How it works inside
Counters, thresholds and background compilation
The interpreter and the level 2 and 3 code count invocations of each method and iterations of each loop (back-edges). When the counters pass a threshold, the method is put in a compile queue. On this JDK, -XX:+PrintFlagsFinal shows the base values: Tier3InvocationThreshold=200 for the step to C1 and Tier4InvocationThreshold=5000 for the step to C2, with separate thresholds that also count loop iterations. The thresholds are not fixed: the policy scales them up when the compile queues are long, so a busy compiler delays further compilations.
- Compilation happens on separate threads. The request thread keeps running the old version until the new one is installed. On a machine with 2 CPUs this JVM uses 2 compiler threads (
CICompilerCount=2), which share the CPUs with your threads. - A long loop does not wait for the method to return. When a loop has iterated often enough, the JVM compiles a version of the method that can be entered in the middle of the loop and switches to it while the loop runs. This is on-stack replacement (OSR), marked
%in the log. - Level 3 code is slower than level 2 because of the profiling it does. HotSpot's policy notes about 30%. That is why a method goes to level 2 first when C2 is busy.
Reading the compilation log
Program outputjava -Xlog:jit+compilation=debug:file=jit.log JitWarmup.java warmup, then grep JitWarmup jit.log
Real JVM output$ grep JitWarmup jit.log [0.409s][debug][jit,compilation] 143 1 JitWarmup$Line::quantity (5 bytes) [0.409s][debug][jit,compilation] 144 1 JitWarmup$Line::unitCents (5 bytes) [0.422s][debug][jit,compilation] 154 3 JitWarmup$Line::<init> (15 bytes) [0.423s][debug][jit,compilation] 155 3 JitWarmup::lineTotal (11 bytes) [0.423s][debug][jit,compilation] 156 3 JitWarmup$Standard::apply (8 bytes) [0.434s][debug][jit,compilation] 169 3 JitWarmup::parse (78 bytes) [0.435s][debug][jit,compilation] 168 3 JitWarmup::handle (54 bytes) [0.496s][debug][jit,compilation] 165 4 JitWarmup$Standard::apply (8 bytes) [0.496s][debug][jit,compilation] 156 3 JitWarmup$Standard::apply (8 bytes) made not entrant [0.505s][debug][jit,compilation] 185 4 JitWarmup::parse (78 bytes) [0.572s][debug][jit,compilation] 169 3 JitWarmup::parse (78 bytes) made not entrant [0.572s][debug][jit,compilation] 186 4 JitWarmup::handle (54 bytes) [0.613s][debug][jit,compilation] 168 3 JitWarmup::handle (54 bytes) made not entrant [0.706s][debug][jit,compilation] 229 % 3 JitWarmup::warmup @ 146 (218 bytes) [0.719s][debug][jit,compilation] 230 3 JitWarmup::warmup (218 bytes) [0.780s][debug][jit,compilation] 231 % 4 JitWarmup::warmup @ 146 (218 bytes) [0.812s][debug][jit,compilation] 229 % 3 JitWarmup::warmup @ 146 (218 bytes) made not entrant [1.008s][debug][jit,compilation] 231 % 4 JitWarmup::warmup @ 146 (218 bytes) made not entrant
lineTotal, apply, parse and handle are compiled at level 3; batch 2 is four times cheaper. At 0.50 to 0.57 s, parse and handle are compiled at level 4, and each time the level 3 version is marked made not entrant: no new calls may enter it, because a better version exists. Batches 10 to 12 run the C2 code. The % lines are OSR compilations of the loop in warmup itself (@ 146 is the bytecode index of the loop). made not entrant here is routine replacement, not a problem.| Field | Values | Meaning |
|---|---|---|
| Time | [0.505s] or a number of ms | When the compilation was logged. -XX:+PrintCompilation prints milliseconds since JVM start with no brackets. |
| Compile id | 185 | Sequence number of the compilation. The same id appears again when that version is invalidated. |
| Flags | % s ! b n | % OSR, s synchronized method, ! has exception handlers, b blocking compilation, n wrapper for a native method |
| Level | 0 to 4 | The tier, as in the table above |
| Method | JitWarmup::parse (78 bytes) | Class, method and size of its bytecode. @ 146 after the name is the OSR entry point. |
| Message | made not entrant | This version must not be entered any more: replaced by a higher tier, or invalidated by a deoptimisation |
Program outputThe classic form: -XX:+PrintCompilation
Real JVM output$ java -Xmx128m -XX:+PrintCompilation JitWarmup.java warmup | head -12 109 1 3 java.lang.Object::<init> (1 bytes) 110 2 n 0 jdk.internal.misc.Unsafe::getReferenceVolatile (native) 115 4 3 java.lang.String::isLatin1 (19 bytes) 115 3 3 java.lang.String::hashCode (60 bytes) 119 5 3 java.lang.String::coder (15 bytes) 121 6 3 java.lang.String::length (11 bytes) 121 7 3 jdk.internal.util.Preconditions::checkIndex (18 bytes) 129 8 3 java.lang.String::charAt (25 bytes) 130 9 n 0 java.lang.System::arraycopy (native) (static) 134 11 n 0 jdk.internal.misc.Unsafe::getByte (native) 138 10 3 java.lang.ref.Reference::reachabilityFence (1 bytes) 138 12 3 java.util.concurrent.ConcurrentHashMap::tabAt (22 bytes)
String::hashCode, String::charAt). n 0 lines are native method wrappers. This output is mixed into your program's output without synchronisation, so for anything but a quick look use -Xlog:jit+compilation=debug:file=jit.log, which writes whole lines to a file.What the profile is for: inlining and speculation
C2's most important optimisation is inlining: replacing a call with the body of the called method. It removes the call, and it lets every later optimisation see across the method boundary: escape analysis, constant folding, removal of repeated checks. Small methods are always candidates (-XX:MaxInlineSize, 35 bytes of bytecode). Hot methods are inlined up to a larger size (-XX:FreqInlineSize, 325 on this platform).
A call through an interface or a non-final method cannot be inlined unless the compiler knows which implementation will be called. The profile tells it. Level 3 code records the classes actually seen at each call site (up to 2 per site on this JDK, TypeProfileWidth=2). If only one class ever appeared, C2 inlines that implementation behind a cheap type check. This is speculation: the compiled code is correct only as long as the program keeps behaving as it did while it was profiled. The same applies to branches that were never taken and to checks that never failed: C2 leaves them out and installs a trap instead.
Deoptimisation
When a speculation turns out wrong, the thread hits the trap. The JVM discards the compiled frame, continues the method in the interpreter at the same point, usually invalidates the compiled version, and compiles it again later with the new facts. Nothing breaks, but for a short time hot code runs at interpreter speed, and the compiler has work to do again.
Program outputA second implementation appears: java -Xlog:jit+compilation=debug:file=jit.log JitWarmup.java deopt
Real JVM outputphase 1 starts at 367 ms: every call so far passes a Standard rule phase 2 starts at 482 ms: half of the calls now pass a Reduced rule done at 546 ms, checksum 30950850000 $ grep JitWarmup jit.log ... [0.395s][debug][jit,compilation] 1418 4 JitWarmup::parse (78 bytes) [0.409s][debug][jit,compilation] 1416 3 JitWarmup::parse (78 bytes) made not entrant [0.409s][debug][jit,compilation] 1419 4 JitWarmup::handle (54 bytes) [0.414s][debug][jit,compilation] 1417 3 JitWarmup::handle (54 bytes) made not entrant ... [0.483s][debug][jit,compilation] 1422 % 4 JitWarmup::run @ 5 (40 bytes) made not entrant [0.483s][debug][jit,compilation] 1419 4 JitWarmup::handle (54 bytes) made not entrant [0.488s][debug][jit,compilation] 1428 3 JitWarmup$Reduced::apply (12 bytes) [0.489s][debug][jit,compilation] 1429 % 4 JitWarmup::handle @ 13 (54 bytes) [0.492s][debug][jit,compilation] 1431 4 JitWarmup$Reduced::apply (12 bytes) [0.492s][debug][jit,compilation] 1428 3 JitWarmup$Reduced::apply (12 bytes) made not entrant [0.492s][debug][jit,compilation] 1432 4 JitWarmup::handle (54 bytes)
handle passes a Standard pricing rule, and C2 compiles handle (id 1419, level 4) at 0.409 s for exactly that. Phase 2 starts at 482 ms and passes a Reduced rule on every second call. One millisecond later, at 0.483 s, compilation 1419 is made not entrant: its assumption failed. For the next 9 ms calls run in lower tiers or in an OSR version (id 1429, compiled at 0.489 s), until a new level 4 version (id 1432) is installed at 0.492 s. In a service the gap is longer, because the method is larger and the compiler has a queue. The other line at 0.483 s, for run @ 5, is a second, unrelated trap: the exit of the first loop, a branch that had never been taken in 300,000 iterations.JFRjfr view deoptimizations-by-reason and deoptimizations-by-site, and one jdk.Deoptimization event
Real JVM output Deoptimization by Reason
Reason Count
---------------------- ----------------
unstable_if 2
predicate 1
Deoptimization by Site
Method Line Number Bytecode Index Count
--------------------------------------------------------------- ----------- -------------- -----
JitWarmup.run(JitWarmup$PricingRule, JitWarmup$PricingRule) 102 9 1
java.util.HashMap.putVal(int, Object, Object, boolean, boolean) 661 236 1
JitWarmup.handle(String, JitWarmup$PricingRule) 39 20 1
...
jdk.Deoptimization {
startTime = 22:08:39.538 (2026-11-15)
compileId = 1419
compiler = "c2"
method = JitWarmup.handle(String, JitWarmup$PricingRule)
lineNumber = 39
bci = 20
instruction = "ifeq"
reason = "predicate"
action = "maybe_recompile"
eventThread = "main" (javaThreadId = 1)
}handle names compile id 1419, the one that was invalidated in the log above, at source line 39, which is the loop that calls rule.apply(...). reason = "predicate" is HotSpot's code for 'compiler generated predicate failed': a check that C2 created itself, typically one it moved in front of a loop. unstable_if, the reason of the other two events, means a branch that was predicted never to be taken was taken. action = "maybe_recompile" says what happens next. A handful of deoptimisations after startup or after a change in traffic is normal. Hundreds per minute at the same site are worth a look.The code cache
Compiled code is stored in the code cache, a fixed region outside the heap: 240 MB reserved by default with tiered compilation. It is split into three segments: non-method code such as the interpreter, profiled code (levels 2 and 3, short-lived) and non-profiled code (levels 1 and 4, long-lived). The JVM removes code that is no longer used. If the cache still fills up, the JIT stops compiling. Code that is already compiled keeps running; everything else stays in the interpreter or in whatever tier it had reached.
Program outputA workload with thousands of hot methods, with the default code cache and with 2.5 MB
Real JVM output$ java -Xmx128m -XX:+PrintCodeCache JitWarmup.java codecache round 1 done at 2832 ms: 20 compiler runs used 643 ms of CPU round 2 done at 3964 ms: 20 compiler runs used 260 ms of CPU round 3 done at 4832 ms: 20 compiler runs used 203 ms of CPU ... round 8 done at 9203 ms: 20 compiler runs used 153 ms of CPU ... CodeCache: size=245760Kb, used=9456Kb, max_used=9513Kb, free=236301Kb total_blobs=3934, nmethods=3491, adapters=348, full_count=0 Compilation: enabled, stopped_count=0, restarted_count=0 $ java -Xmx128m -XX:ReservedCodeCacheSize=2496k -XX:+PrintCodeCache JitWarmup.java codecache round 1 done at 4880 ms: 20 compiler runs used 995 ms of CPU round 2 done at 6954 ms: 20 compiler runs used 444 ms of CPU round 3 done at 8619 ms: 20 compiler runs used 359 ms of CPU [8.832s][warning][codecache] CodeCache is full. Compiler has been disabled. ... total_blobs=1043, nmethods=600, adapters=348, full_count=1 Compilation: disabled (not enough contiguous free space left), stopped_count=1, restarted_count=0 round 4 done at 9935 ms: 20 compiler runs used 352 ms of CPU round 5 done at 11325 ms: 20 compiler runs used 335 ms of CPU ... round 8 done at 15503 ms: 20 compiler runs used 329 ms of CPU
-XX:ReservedCodeCacheSize=2496k the cache is full in round 4: the JVM prints CodeCache is full. Compiler has been disabled., full_count=1, and Compilation: disabled. From then on the rounds stay at about 330 ms, more than twice the steady state of the first run, for the rest of the process's life. Only 600 methods were compiled. Nothing fails and no exception is thrown: the service is just slower, and the only evidence is one warning line in the log.Architecture
Drawing diagram…
Flowchart. A method starts at level 0 in the interpreter. After about 200 calls it is compiled by C1 at level 3 with full profiling, or at level 2 first if the C2 queue is long. Trivial methods go to level 1. After about 5000 calls with a complete profile it is compiled by C2 at level 4. If an assumption of the level 4 code fails, the method is deoptimised back to level 0. Level 3 code is stored in the profiled segment of the code cache, level 1 and 4 code in the non-profiled segment.
flowchart LR L0["Level 0, interpreter with profiling"] -->|"about 200 calls"| L3["Level 3, C1 with full profiling"] L0 -->|"C2 queue is long"| L2["Level 2, C1 with counters only"] L2 --> L3 L3 -->|"trivial method"| L1["Level 1, C1 without profiling"] L3 -->|"about 5000 calls, profile ready"| L4["Level 4, C2 optimised for the profile"] L4 -->|"assumption fails, deoptimise"| L0 L3 --> CC["Code cache, profiled segment"] L4 --> CC2["Code cache, non-profiled segment"] L1 --> CC2
- The numbers on the arrows are the base thresholds of this JDK. Loops reach them through iterations as well as calls, and the JVM raises them when the compilers are busy.
- Level 3 exists to feed level 4. Its code is larger and slower than it could be, because it records types and branches as it runs.
- The path back from level 4 to level 0 is the cost of speculation. After it, the method climbs the tiers again with a more complete profile.
- Each compiled version takes space in the code cache. A method can have several versions there for a while: the level 3 version is removed some time after level 4 has replaced it.
Step by step
Drawing diagram…
Sequence diagram. A request thread calls a method in the interpreter, which counts calls. When the counter passes a threshold the method is queued for C1, which installs level 3 code while requests keep running interpreted. Later calls run the level 3 code and record a profile. When the next threshold passes, C2 compiles the method with that profile and installs level 4 code. When a type arrives that the profile never saw, the compiled code traps, the call continues in the interpreter, and the method is recompiled.
sequenceDiagram participant R as Request thread participant I as Interpreter participant C1 as C1 compiler thread participant C2 as C2 compiler thread participant K as Code cache R->>I: Call handle, first requests I->>I: Count calls and loops I->>C1: Counter passed, queue handle Note over R,I: Requests keep running interpreted C1->>K: Install level 3 code R->>K: Later calls run level 3 code and record a profile K->>C2: Counter passed, queue handle with its profile Note over R,K: Requests keep running level 3 code C2->>K: Install level 4 code, level 3 made not entrant R->>K: Calls run level 4 code R->>K: A type arrives that the profile never saw K-->>I: Trap, continue this call in the interpreter I->>C2: Recompile handle with the new profile
- The request thread never waits for a compilation. It runs whatever version is installed at the moment of the call, which is why warm-up is gradual and not a step.
- The compiler threads need CPU time of their own. During warm-up a process uses more CPU than its request rate explains.
- The profile is collected by real calls. What those calls look like decides what the level 4 code is optimised for.
- The last three arrows are a deoptimisation. The call that hit the trap completes correctly, only slower.
In a real system
A pricing service is deployed with a rolling update, three instances, one at a time. Each new instance passes its readiness check about 20 seconds after start and immediately receives a third of the traffic, 400 requests per second. For the next 30 to 60 seconds its p99 is several times the normal value, some requests exceed the caller's 500 ms timeout, and the caller's retries land on the other instances. Then it settles. The dashboards show three bumps per deploy and nobody can find a bug.
Program outputThe three speeds of the same code: C2, C1 only, interpreter only
Real JVM output$ java -Xmx128m JitWarmup.java warmup ... steady state: 200,000 more requests, 0.27 us of CPU per request ... $ java -Xmx128m -XX:TieredStopAtLevel=1 JitWarmup.java warmup ... steady state: 200,000 more requests, 0.70 us of CPU per request ... $ java -Xmx128m -Xint JitWarmup.java warmup ... steady state: 200,000 more requests, 18.48 us of CPU per request
-XX:TieredStopAtLevel=1 (C1 only, no profiling, no C2): 0.70 µs. With -Xint (interpreter only): 18.48 µs, 68 times the C2 figure, and it never improves. A cold instance is somewhere between the last and the first number for each method on the request path, and moves towards the first as the methods are compiled. -Xint and -XX:TieredStopAtLevel=1 are diagnostic settings; they show what each tier contributes.There is no flag that removes warm-up on JDK 21, because the fast code depends on a profile that exists only after the code has run. What you can do is decide where the warm-up happens:
- Warm up before taking traffic. Run the real request path with realistic inputs during startup, before the instance reports ready. The implementation below does this with an
ApplicationRunner, which Spring Boot runs before it marks the application as accepting traffic. - Ramp traffic up. If the load balancer or service mesh supports slow start, let a new instance receive a small share first. The JIT warms up on real traffic and the cost is spread thinly.
- Give the compiler CPU. A container limited to a fraction of a CPU compiles slowly. Warm-up is the moment a JVM needs the most CPU per request.
- Do not judge by the first minute. Exclude a warm-up period from load tests and benchmarks, or measure it separately on purpose.
To see compilation in a running service, JFR is the least intrusive tool. jfr view compiler-statistics recording.jfr gives totals (methods compiled, total compile time, OSR compilations) and jfr view deoptimizations-by-site the invalidations. Note that the per-compilation event jdk.Compilation is recorded only for compilations that take longer than 1,000 ms in JFR's default settings (100 ms in the profile settings), so jfr view longest-compilations is usually empty; it was in the lab. For a full list, use -Xlog:jit+compilation=debug.
Java and Spring Boot
A warm-up that runs before a Spring Boot 3.5 service accepts traffic. It drives the real service bean with a realistic mix of requests for a bounded time. The code was compiled here against stub versions of the Spring types and was not run inside Spring; PricingService, QuoteRequest and WarmupSamples stand for your own types.
package com.example.pricing;import java.time.Duration;import java.util.List;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.boot.ApplicationArguments;import org.springframework.boot.ApplicationRunner;import org.springframework.stereotype.Component;/** Runs the hot request path before the application reports ready, so the JIT has compiled it. */@Componentpublic class JitWarmupRunner implements ApplicationRunner { private static final Logger log = LoggerFactory.getLogger(JitWarmupRunner.class); private static final int MAX_CALLS = 30_000; private static final Duration BUDGET = Duration.ofSeconds(15); private final PricingService pricing; private final List<QuoteRequest> samples; public JitWarmupRunner(PricingService pricing, WarmupSamples warmupSamples) { this.pricing = pricing; this.samples = warmupSamples.realisticMix(); } @Override public void run(ApplicationArguments args) { long start = System.nanoTime(); long deadline = start + BUDGET.toNanos(); long checksum = 0; int calls = 0; while (calls < MAX_CALLS && System.nanoTime() < deadline) { QuoteRequest request = samples.get(calls % samples.size()); try { checksum += pricing.quote(request).totalCents(); } catch (RuntimeException e) { log.warn("warm-up call failed, continuing: {}", e.toString()); } calls++; } log.info("JIT warm-up: {} calls in {} ms (checksum {})", calls, (System.nanoTime() - start) / 1_000_000, checksum); }}Why it is written this way
- An
ApplicationRunnerruns after the context has started and before Spring Boot publishes the readiness stateACCEPTING_TRAFFIC. A Kubernetes readiness probe bound to that state keeps the instance out of the load balancer untilrunreturns. - 30,000 calls is well past the 5,000 calls at which C2 becomes interested, with room for the methods further down the call tree and for the compile queue. The time budget stops a slow dependency from blocking startup for ever.
- The samples are a mix: every customer type, pricing rule and payload shape that production sends. The profile recorded here decides what C2 optimises for. One fixed request teaches the compiler that only one case exists.
- The call goes through the real
PricingServicebean, with its proxies, validation and serialisation, because those are the methods that need compiling. Calling an internal helper directly warms up the wrong code. - The result is used (
checksum), so the compiler cannot remove the work. Warm-up calls must have no side effects: no writes, no messages, no charges. - The log line records how long warm-up took. It is startup time you chose to spend; watch it.
# What is the compiler doing right now?jcmd "$PID" Compiler.queue # methods waiting to be compiledjcmd "$PID" Compiler.codecache # code cache size, usage and whether compilation is enabled# Record compilation activity for one minute from a running servicejcmd "$PID" JFR.start name=jit duration=60s filename=/tmp/jit.jfrjfr view compiler-statistics /tmp/jit.jfrjfr view deoptimizations-by-site /tmp/jit.jfrjfr view deoptimizations-by-reason /tmp/jit.jfr# Every compilation, from process start (set at launch)# java -Xlog:jit+compilation=debug:file=/var/log/app/jit.log -jar app.jargrep -c "made not entrant" /var/log/app/jit.loggrep "com.example.pricing" /var/log/app/jit.log | tail -20# The warning to alert ongrep "CodeCache is full" /var/log/app/*.logWhy it is written this way
Compiler.queueduring the first minute after start shows the backlog: a long queue means requests are still running in lower tiers.Compiler.codecacheprints the same block as-XX:+PrintCodeCachein the lab: look atusedagainstsize, and atCompilation: enabled.- The compilation log is verbose during startup (thousands of lines) and quiet afterwards. Write it to a file, not to standard output.
- Spring Boot's Micrometer integration publishes
jvm.compilation.time, the accumulated compile time. A value that keeps rising quickly long after startup means the JVM is still compiling or recompiling.
Wrong, then right
Two attempts at warming up the pricing service. The first one runs too late and teaches the compiler the wrong thing. The second is the runner from the implementation section, shown next to it.
Incorrect implementation
package com.example.pricing;import org.springframework.boot.context.event.ApplicationReadyEvent;import org.springframework.context.event.EventListener;import org.springframework.scheduling.annotation.Async;import org.springframework.stereotype.Component;@Componentpublic class WarmupListener { private final PricingService pricing; public WarmupListener(PricingService pricing) { this.pricing = pricing; } @Async @EventListener(ApplicationReadyEvent.class) public void warmUp() { QuoteRequest request = QuoteRequest.standard("WARMUP", 1, 1999); for (int i = 0; i < 100; i++) { pricing.quote(request); } }}Why it is written this way
ApplicationReadyEventis published after the runners, immediately before the readiness state changes toACCEPTING_TRAFFIC. With@Asyncthe listener returns at once, so the instance is ready while the loop has barely started.- 100 calls are below even the roughly 200 that C1 waits for, and far below the thousands that C2 needs.
- One request object, one customer type, one pricing rule: every call site on the path sees exactly one class and every branch goes the same way.
- The result of
quoteis ignored. For this small loop it does not matter, but in a longer one the compiler may remove work whose result is never used.
Why it fails
- The warm-up races with real traffic and loses. The readiness state flips as soon as the asynchronous listener has been started, the load balancer sends requests, and they run in the interpreter and in C1 code, the slow tiers the warm-up was supposed to get past.
- Even if it ran in time, 100 calls would leave every method below C2's threshold. The first few thousand real requests would still do the profiling and wait for the level 4 code.
- If the loop were made long enough, the single input would make it worse in a different way. C2 would compile the path for the one rule and the one branch pattern it saw. The first real request with a different rule fails a type check, and the hot method is deoptimised and recompiled under load. The lab's
deoptrun shows exactly this: a method compiled for one implementation and invalidated 1 ms after a second one appeared. - In production the symptom is the same latency bump after each deploy as with no warm-up at all, plus a log line that says the warm-up succeeded.
Corrected implementation
package com.example.pricing;import java.time.Duration;import java.util.List;import org.slf4j.Logger;import org.slf4j.LoggerFactory;import org.springframework.boot.ApplicationArguments;import org.springframework.boot.ApplicationRunner;import org.springframework.stereotype.Component;@Componentpublic class JitWarmupRunner implements ApplicationRunner { private static final Logger log = LoggerFactory.getLogger(JitWarmupRunner.class); private static final int MAX_CALLS = 30_000; private static final Duration BUDGET = Duration.ofSeconds(15); private final PricingService pricing; private final List<QuoteRequest> samples; public JitWarmupRunner(PricingService pricing, WarmupSamples warmupSamples) { this.pricing = pricing; this.samples = warmupSamples.realisticMix(); // all rule types and payload shapes production sends } @Override public void run(ApplicationArguments args) { // synchronous: readiness waits for it long deadline = System.nanoTime() + BUDGET.toNanos(); long checksum = 0; int calls = 0; while (calls < MAX_CALLS && System.nanoTime() < deadline) { checksum += pricing.quote(samples.get(calls % samples.size())).totalCents(); calls++; } log.info("JIT warm-up: {} calls, checksum {}", calls, checksum); }}Why it is written this way
- A runner is executed on the startup thread before the application is marked ready. No
@Async, no event: the ordering is the point. - The loop is bounded by calls and by time. Whichever comes first ends it, and startup continues.
- Rotating through a realistic mix gives each call site the same set of types it will see in production, so the level 4 code does not have to be thrown away when real traffic arrives.
- The checksum is logged, so every result is used and the compiler cannot discard the calls.
Trade-offs
- Startup takes up to 15 seconds longer per instance. A rolling deploy of 30 instances takes minutes longer, and autoscaling reacts later. The budget is a trade between a slower rollout and slow first requests.
- Warm-up data has to be maintained. If production starts sending a new kind of request and the samples do not include it, the warm-up is quietly less useful and may cause the deoptimisation it was meant to prevent.
- The warm-up runs real code, so it needs real dependencies or safe substitutes. A path that writes to a database or calls a payment provider cannot be exercised with fake orders unless it has a dry-run mode.
- It warms only what it calls. Rare endpoints stay cold, which is usually acceptable; do not try to cover everything.
- It uses CPU during startup, at the same time as class loading and bean creation. In a container with a tight CPU limit, startup may be throttled and take longer than the budget suggests.
Configuration
Flags that show or limit what the JIT does. Defaults are from the JDK 21 java command reference, or from -XX:+PrintFlagsFinal on OpenJDK 21.0.13 for Linux x64 where the reference gives none.
-XX:+TieredCompilationDefault enabled
- What it does
- Use both compilers: C1 with profiling first, C2 afterwards.
- When to change it
- Leave it on. Without it startup is slower, and the default code cache size drops to 48 MB.
-XX:TieredStopAtLevelDefault 4 (PrintFlagsFinal)
- What it does
- The highest tier methods may reach.
1means C1 only, without profiling. - When to change it
- A diagnostic or special-purpose setting.
1shortens warm-up and lowers peak speed (0.70 µs against 0.27 µs per request in the lab). Consider it only for short-lived processes.
-XX:ReservedCodeCacheSizeDefault 240 MB (48 MB with -XX:-TieredCompilation)
- What it does
- Maximum size of the code cache. When it is full, compilation stops.
- When to change it
- Do not lower it to save memory without measuring
Compiler.codecacheafter a long run. Raise it if you seeCodeCache is full.
-XX:+UseCodeCacheFlushingDefault enabled
- What it does
- The JVM removes compiled code that is no longer used to make room before giving up on compilation.
- When to change it
- Leave it on.
-Xlog:jit+compilation=debugDefault off
- What it does
- One line per compilation and per invalidation, in the unified logging format.
-XX:+PrintCompilationprints the same to standard output. - When to change it
- Write it to a file when investigating warm-up or deoptimisation. It is verbose at startup.
-XX:MaxInlineSizeDefault 35 (bytes of bytecode)
- What it does
- Methods up to this size can be inlined by C2 even when they are not hot. Hot methods have a larger limit,
-XX:FreqInlineSize(platform dependent, 325 here). - When to change it
- Do not tune it. Know it: small methods are free to call from hot code, very large hot methods are not inlined.
-XintDefault off
- What it does
- Interpreter only. No compilation at all.
- When to change it
- Never in production. Useful once, to see what the JIT is worth: 68 times slower in the lab.
Hands-on exercise
Watch a method climb the tiers and fall back
Run the lab program on your own JDK 21, match the drop in CPU per request to lines of the compilation log, compare the three execution modes, and provoke a deoptimisation.
Steps · 0 of 6 done
What you should see
Batch 1 costs several times more CPU per request than batch 2, and batch 12 less again (3.46, 0.89 and 0.28 µs here). The level 3 compilation of handle falls inside batch 1 and the level 4 compilation a few batches later, after which the per-request cost reaches its floor. The % line is the OSR compilation of the loop in warmup; made not entrant marks a level 3 version replaced by level 4. Steady state is roughly 0.3 µs with defaults, 0.7 µs with C1 only and 18 µs interpreted. In the deopt run a level 4 version of handle is made not entrant within a few milliseconds of phase 2 starting, and a new level 4 version follows. The code cache run prints CodeCache is full. Compiler has been disabled. and ends with Compilation: disabled. Your absolute numbers will differ.
What can go wrong
Sending full traffic to a cold instance
- Symptom
- p99 latency and timeouts rise for 30 to 60 seconds on each new instance during every rolling deploy or scale-out, then recover by themselves.
- Cause
- The request path is still interpreted or running C1 code, several times slower than the steady state, and the compiler threads compete with request threads for CPU.
- Fix
- Warm the request path before readiness, or ramp traffic to new instances gradually. Make sure the container has CPU to compile with.
Measuring the first minute
- Symptom
- A benchmark shows a new implementation as slower, or a load test fails its latency target, but production numbers an hour after deploy are fine.
- Cause
- The measurement covers interpreter and C1 execution and the compilation itself. Short tests may never reach level 4.
- Fix
- Discard a warm-up phase, run long enough to reach steady state, and for microbenchmarks use a harness such as JMH that handles warm-up and prevents the compiler from removing the measured code.
Warming up with unrealistic input
- Symptom
- Warm-up is in place, yet the first real requests still cause a latency step, and JFR shows deoptimisations in hot methods right after traffic starts.
- Cause
- The profile recorded during warm-up saw one type per call site and one outcome per branch. C2 compiled for that. Real traffic breaks the assumptions and the code is invalidated.
- Fix
- Feed the warm-up a mix that covers the types and branches production uses. Check
jfr view deoptimizations-by-siteafter a deploy.
Shrinking the code cache to save memory
- Symptom
- Some time after start, throughput drops and CPU per request rises, permanently. The log contains
CodeCache is full. Compiler has been disabled.once. - Cause
-XX:ReservedCodeCacheSizewas set below what the application's compiled code needs. When the cache fills, compilation stops and newly hot code stays slow.- Fix
- Remove the setting or raise it. Check
jcmd <pid> Compiler.codecacheafter hours of real traffic before choosing any limit, and alert on the warning.
Reading every 'made not entrant' as a problem
- Symptom
- Someone counts thousands of
made not entrantlines in the compilation log and concludes the JVM is thrashing. - Cause
- The same message is printed when a level 3 version is retired because the level 4 version is ready. That is the normal path of every hot method.
- Fix
- Look at the level of the invalidated version. Level 2 or 3 after a level 4 compilation of the same method is routine. A level 4 version made not entrant is a deoptimisation; use JFR's deoptimisation views for those.
How to troubleshoot
Slow requests for the first minute after each deploy
Check
jcmd <pid> Compiler.queue (in the first seconds after start), and Micrometer `jvm.compilation.time` against process uptime
- How to read it
- A long queue and fast-rising compile time mean the JIT is still catching up and requests run in lower tiers. This is warm-up, not a regression in the code.
Need to know whether a specific method has been compiled, and at which level
Check
java -Xlog:jit+compilation=debug:file=jit.log ... then grep 'com.example.pricing.PricingService::quote' jit.log
- How to read it
- The level column shows the tier of each version, in order. A last line at level 4 without
made not entrantmeans the method runs C2 code. No line at all means it never became hot.
A latency step in the middle of the day with no deploy
Check
jfr view deoptimizations-by-site recording.jfr and jfr view deoptimizations-by-reason recording.jfr
- How to read it
- Deoptimisations concentrated in a hot method at that time mean the program started behaving differently: a new implementation behind an interface, a branch that had never been taken, a feature flag switched on.
Throughput dropped and stays low; CPU per request is up
Check
grep 'CodeCache is full' app.log and jcmd <pid> Compiler.codecache
- How to read it
Compilation: disabledor afull_countabove 0 means the code cache filled and the JIT stopped. Restart with a larger-XX:ReservedCodeCacheSize, or remove a setting that lowered it.
CPU is high during startup although no requests arrive yet
Check
jcmd <pid> Thread.print | grep -E 'C1 CompilerThread|C2 CompilerThread'
- How to read it
- The compiler threads appear in the thread dump with their CPU time. High CPU on them during startup and warm-up is expected. If they stay busy for many minutes, look for continuous class generation or repeated deoptimisation.
Key takeaways
- Every method starts in the interpreter. Hot methods are compiled by C1 with profiling (level 3) and then by C2 using that profile (level 4).
- Compilation runs on background threads. Requests use whatever version is installed, so speed improves gradually over the first seconds to minutes.
- C2's speed comes from inlining and from assumptions taken from the profile. When an assumption fails, the code is deoptimised and compiled again.
- In a compilation log,
%is on-stack replacement andmade not entrantretires a version: routine for level 3, a deoptimisation for level 4. - A cold instance is several times slower than a warm one. Warm the real request path with realistic input before readiness, or ramp traffic up.
- A full code cache disables the JIT without any error. The only sign is one warning line and a service that stays slow.
Practical check
Small scenarios, one decision each. Pick what you would do, then read why.
Scenario 1 of 3
A service is rolled out one instance at a time. Each new instance shows p99 of 900 ms for about 40 seconds after it becomes ready, against 80 ms normally. CPU on the new instance is at its limit during that time. GC pauses are under 5 ms throughout, and the old instances are unaffected.
What is the most likely cause, and what addresses it?
Scenario 2 of 3
A compilation log contains, in this order: 412 3 OrderService::price (210 bytes), later 655 4 OrderService::price (210 bytes), then 412 3 OrderService::price (210 bytes) made not entrant. A colleague says the method was deoptimised and wants to investigate.
What do the three lines show?
Scenario 3 of 3
To reduce container memory, a team sets -XX:ReservedCodeCacheSize=32m on a large Spring Boot service. After two hours of traffic, throughput falls by a third and stays there. There are no errors. The log has one line: CodeCache is full. Compiler has been disabled.
What happened, and what is the fix?
References
References: JDK 21: the java command (-XX:+TieredCompilation, -XX:ReservedCodeCacheSize, -XX:+PrintCompilation, -XX:MaxInlineSize, -Xint) ↗ · JDK 21 HotSpot Virtual Machine Guide: Performance Enhancements (tiered compilation, segmented code cache) ↗ · OpenJDK 21 source: compilationPolicy.hpp (the compilation levels and transitions) ↗ · OpenJDK 21 source: deoptimization.hpp (deoptimisation reasons and actions) ↗ · OpenJDK 21 source: compileTask.cpp (the fields of a compilation log line) ↗ · JDK 21: the jcmd command (Compiler.queue, Compiler.codecache) ↗ · Spring Boot 3.5 reference: SpringApplication (application events, runners, availability states) ↗ · Report a problem