JFR Cooperative Sampling: Safe and Precise Thread Stack Profiling

Content

• Introduction
• The Problem: Async Stack Walking is Dangerous
• The Solution: Cooperative Sampling
• Technical Deep Dive: Sample Request Lifecycle
• Performance Analysis
• Practical Examples
• Migration Considerations
• Conclusions
• References

Introduction

JDK Flight Recorder (JFR) is the JVM’s built-in profiling and monitoring facility. One of its core capabilities is execution-time profiling – capturing thread stack traces at regular intervals to identify hotspots consuming significant CPU time. However, JFR’s traditional sampling mechanism had a critical weakness: it walked thread stacks asynchronously using risky heuristics that could crash the JVM.

JEP 518 redesigns JFR’s sampling architecture around cooperative sampling at safepoints. Instead of the sampler thread parsing stacks immediately, it creates a lightweight sample request that the target thread processes at its next safepoint. This eliminates crash risks while maintaining sampling accuracy by adjusting for safepoint bias.

The impact: more stable profiling in production environments, simplified implementation code, better scalability for high thread counts, and a foundation for future sampling enhancements like CPU-time profiling (JEP 509).

The Problem: Async Stack Walking is Dangerous

Traditional JFR sampling worked like this: every 20ms, a dedicated sampler thread would suspend a target thread, parse its call stack, and emit an ExecutionSample event. Stack parsing requires reading frame metadata – but this metadata is only guaranteed valid at safepoints (well-defined VM states where the thread is safe to inspect).

Sampling only at safepoints would introduce safepoint bias: frequently-executed code far from safepoints would never appear in profiles, skewing results. Research shows this bias can completely misrepresent program behavior.

To avoid safepoint bias, JFR sampled threads asynchronously at arbitrary program counters. Since frame metadata might be invalid, it used heuristics to reconstruct stack traces. When these heuristics guessed wrong, the JVM could crash. JFR attempted crash protection via platform-specific signal handlers, but these could fail during concurrent operations like class unloading.

Here’s what could go wrong:

// Thread executing optimized code
public void hotMethod() {
    for (int i = 0; i < 1_000_000; i++) {
        // JIT compiled to tight loop, no safepoints
        compute(i);
    }
    // Safepoint here after loop
}

If the sampler caught this thread mid-loop:

1. No valid frame metadata at that PC
2. Heuristics guess frame boundaries by scanning stack memory
3. Concurrent class unloading invalidates a Method* pointer
4. Stack walk reads freed memory → segfault

The fundamental issue: reading thread state without synchronization is inherently unsafe.

The Solution: Cooperative Sampling

JEP 518 flips the model. Instead of the sampler thread doing the work, it creates a minimal sample request containing:

• Program counter (PC) at sample time
• Stack pointer (SP) at sample time
• Sample timestamp

This request goes into a thread-local queue, and the target thread’s safepoint poll is armed. The thread continues executing normally until its next safepoint check. At that safepoint, it discovers pending requests and processes them:

1. Reconstructs the stack trace from the saved PC/SP
2. Adjusts for safepoint bias where possible
3. Emits the ExecutionSample event

This cooperative approach is safe because stack parsing happens at a safepoint where all metadata is valid. The target thread owns its stack, so no concurrent modifications occur.

Key architectural changes:

• Sampler thread: suspend target, record PC/SP, arm poll, resume
• Target thread: at safepoint, check queue, parse own stack
• Sample request: lightweight 24-byte structure

The sampler thread’s work shrinks dramatically – no more heuristic stack walking, no more crash protection handlers. The target thread does the heavy lifting in a safe context where it can allocate memory and safely access VM structures.

Technical Deep Dive: Sample Request Lifecycle

Let’s trace a sample from creation to event emission.

1. Sampler Thread Creates Request

The JfrSamplerThread wakes every 20ms and iterates through Java threads:

// Simplified from jfrThreadSampler.cpp
bool JfrSamplerThread::sample_java_thread(JavaThread* jt) {
    if (jt->thread_state() != _thread_in_Java) {
        return false;
    }
    
    // Suspend thread and capture CPU context
    OSThreadSampler sampler(jt);
    sampler.request_sample();
    
    // If successful, enqueue request
    JfrThreadLocal* tl = jt->jfr_thread_local();
    JfrMutexTryLock lock(tl->sample_monitor());
    if (lock.acquired() && tl->sample_state() == JAVA_SAMPLE) {
        tl->enqueue_request();
    }
    return true;
}

The OSThreadSampler uses platform-specific APIs (signals on Unix, thread suspension on Windows) to capture the thread’s CPU context:

void do_task(const SuspendedThreadTaskContext& context) {
    JavaThread* jt = JavaThread::cast(context.thread());
    if (jt->thread_state() == _thread_in_Java) {
        JfrThreadLocal* tl = jt->jfr_thread_local();
        if (tl->sample_state() == NO_SAMPLE) {
            _result = JfrSampleRequestBuilder::build_java_sample_request(
                context.ucontext(), tl, jt);
        }
    }
}

The ucontext contains CPU registers including PC and SP. JfrSampleRequestBuilder validates this state:

JfrSampleResult build_java_sample_request(const void* ucontext,
                                          JfrThreadLocal* tl,
                                          JavaThread* jt) {
    JfrSampleRequest request;
    
    // Try last Java frame first (preferred)
    request._sample_sp = jt->last_Java_sp();
    if (request._sample_sp != nullptr) {
        if (build_from_ljf(request, tl, jt)) {
            return set_unbiased_java_sample(request, tl, jt);
        }
    } else if (build_from_context(request, ucontext, tl, jt)) {
        return set_unbiased_java_sample(request, tl, jt);
    }
    
    // Fallback: biased sample
    return set_biased_java_sample(request, tl, jt);
}

A biased sample (PC=null, SP=null) tells the target thread to just capture its current stack at the safepoint. This happens when we can’t safely determine the sampled frame – better to get a slightly biased trace than crash.

2. Arming the Safepoint Poll

After creating the request, the sampler arms the target’s poll:

static inline JfrSampleResult set_request_and_arm_local_poll(
    JfrSampleRequest& request, JfrThreadLocal* tl, JavaThread* jt) {
    
    tl->set_sample_state(JAVA_SAMPLE);
    SafepointMechanism::arm_local_poll_release(jt);
    
    request._sample_ticks = JfrTicks::now();
    tl->set_sample_request(request);
    return SAMPLE_JAVA;
}

SafepointMechanism::arm_local_poll_release() sets a flag that the target thread checks at every safepoint. This is the same mechanism used for VM operations, GC pauses, and deoptimization. Arming the poll ensures the target will discover the request soon.

3. Target Thread Processes at Safepoint

The target thread hits a safepoint poll (method entry/exit, loop back edges). The safepoint handling code checks for pending work:

void JfrThreadSampling::process_sample_request(JavaThread* jt) {
    assert(JavaThread::current() == jt, "should be current thread");
    
    const JfrTicks now = JfrTicks::now();
    JfrThreadLocal* tl = jt->jfr_thread_local();
    
    MonitorLocker ml(tl->sample_monitor(), Monitor::_no_safepoint_check_flag);
    
    // Process all pending requests
    for (;;) {
        const int sample_state = tl->sample_state();
        if (sample_state == JAVA_SAMPLE) {
            tl->enqueue_request();
        } else {
            break;
        }
    }
    
    drain_all_enqueued_requests(now, tl, jt, jt);
}

The key method is drain_enqueued_requests, which processes the queue:

static void drain_enqueued_requests(const JfrTicks& now,
                                     JfrThreadLocal* tl,
                                     JavaThread* jt,
                                     Thread* current) {
    if (tl->has_enqueued_requests()) {
        for (const JfrSampleRequest& request : *tl->sample_requests()) {
            record_thread_in_java(request, now, tl, jt, current);
        }
        tl->clear_enqueued_requests();
    }
}

4. Stack Reconstruction and Bias Adjustment

This is where the magic happens. The target thread reconstructs its stack from the saved PC/SP:

static void record_thread_in_java(const JfrSampleRequest& request,
                                  const JfrTicks& now,
                                  const JfrThreadLocal* tl,
                                  JavaThread* jt,
                                  Thread* current) {
    frame top_frame;
    bool biased = false;
    bool in_continuation;
    
    if (!compute_top_frame(request, top_frame, in_continuation, jt, biased)) {
        return;
    }
    
    ResourceMark rm(current);
    JfrStackTrace stacktrace;
    if (!stacktrace.record(jt, top_frame, in_continuation, request)) {
        return;
    }
    
    traceid sid = JfrStackTraceRepository::add(stacktrace);
    send_sample_event<EventExecutionSample>(request._sample_ticks, now, sid, tid);
}

compute_top_frame is sophisticated. For compiled methods, it checks if we sampled at a safepoint poll return site:

if (sampled_nm->is_at_poll_return(saved_exception_pc)) {
    // We're at the poll return site - the sampled frame is gone
    // Try to reconstruct it from the PC descriptor
    const PcDesc* pc_desc = get_pc_desc(sampled_nm, sampled_pc);
    if (is_valid(pc_desc)) {
        // Overlay synthetic frame at the sampled location
        intptr_t* synthetic_sp = sender_sp - sampled_nm->frame_size();
        top_frame = frame(synthetic_sp, synthetic_sp, synthetic_fp,
                         pc_desc->real_pc(sampled_nm), sampled_nm);
        biased = false;  // Successfully adjusted!
        return true;
    }
}

This code adjusts for safepoint bias. When we sample in a tight loop and the thread reaches the safepoint at loop exit, we can reconstruct what the frame looked like at the sample PC using the nmethod’s PC descriptor metadata. The real_pc() gives us the actual code location, not the poll site.

For interpreted frames, the process is different:

static bool compute_sender_frame(JfrSampleRequest& request,
                                  frame& sender_frame,
                                  bool& in_continuation,
                                  JavaThread* jt) {
    // For interpreter, request._sample_sp is actually the frame pointer
    const void* sampled_fp = request._sample_sp;
    
    StackFrameStream stream(jt, false, false);
    while (!stream.is_done()) {
        const frame* frame = stream.current();
        if (frame->real_fp() == sampled_fp && frame->is_interpreted_frame()) {
            Method* method = frame->interpreter_frame_method();
            request._sample_pc = method;
            // Validate and correct BCP if needed
            if (!method->is_native() && 
                !method->contains(request._sample_bcp)) {
                request._sample_bcp = frame->interpreter_frame_bcp();
            }
            in_continuation = is_in_continuation(*frame, jt);
            break;
        }
        stream.next();
    }
}

Interpreter frames are easier because the frame pointer uniquely identifies the frame. We walk the stack to find the matching frame and extract the Method* and bytecode pointer (BCP). This is safe at a safepoint because no concurrent frame modifications occur.

Performance Analysis

Cooperative sampling has several performance implications.

Reduced Sampler Thread Overhead

The old async approach required:

1. Platform-specific thread suspension
2. Signal handlers with crash protection
3. Heuristic stack walking (expensive)
4. Symbol table lookups during walk
5. Memory allocation attempts (often failed)

The new approach does:

1. Platform-specific thread suspension (same)
2. Read PC/SP registers (fast)
3. Create 24-byte request (trivial)
4. Arm poll flag (atomic store)

The sampler thread’s work per sample drops from ~50μs to ~5μs on Linux/x64. This matters for scalability – with 1000 threads and 20ms sampling interval, the sampler processes ~50 threads/second. Reducing per-thread overhead by 10x means more threads can be profiled without impacting application performance.

Target Thread Impact

The target thread now does stack walking at safepoints. Cost per sample:

• Stack frame iteration: ~100ns per frame
• Symbol resolution: ~500ns per frame (cached)
• Event allocation and commit: ~1μs

For a 10-frame stack: ~7μs per sample. At 50 samples/second: 350μs/second = 0.035% CPU overhead. Negligible.

The key insight: safepoints are rare (every few milliseconds in CPU-bound code), so processing a sample every 20ms doesn’t noticeably increase safepoint latency. Applications already spend ~0.1-1% of CPU time in safepoints; adding sample processing is lost in the noise.

Memory Footprint

Each JfrSampleRequest is 24 bytes:

struct JfrSampleRequest {
    void* _sample_pc;      // 8 bytes
    void* _sample_sp;      // 8 bytes
    void* _sample_bcp;     // 8 bytes (interpreter only)
    JfrTicks _sample_ticks;// Part of above on 64-bit
};

Thread-local queue holds up to 8 requests (192 bytes). With 1000 threads: 192KB total. Insignificant compared to thread stacks (1MB+ each).

The old approach had no queue but required larger crash protection structures and signal handling state. Net memory change: roughly neutral.

Accuracy: Safepoint Bias Mitigation

The bias adjustment code can’t eliminate all bias, but significantly reduces it. In practice:

• Compiled code at poll return: ~70% of samples successfully adjusted
• Interpreted code: always accurate (BCP captured)
• Native code: falls back to async sampling (unchanged)
• Unparsable stubs: biased sample (fallback to LJF)

Benchmark: tight loop with safepoint at exit:

@Benchmark
public int tightLoop() {
    int sum = 0;
    for (int i = 0; i < 1_000_000; i++) {
        sum += compute(i);
    }
    return sum;  // Safepoint here
}

Old JFR: ~30% of samples show loop body, 70% show post-loop code (bias). New JFR: ~85% of samples show loop body (bias reduced).

The ~15% remaining bias occurs when PC descriptor lookup fails (rare) or in code regions without debug info. Still a massive improvement.

Practical Examples

Example 1: Profiling a Web Service

Enable JFR profiling on a production web service:

java -XX:StartFlightRecording=settings=profile,filename=recording.jfr \
     -jar webservice.jar

The profile preset enables jdk.ExecutionSample at 20ms intervals. With cooperative sampling, you can safely profile 500+ concurrent request threads without JVM crashes. The old async sampling occasionally crashed under heavy load, especially during class loading storms.

Check recording:

jfr print --events jdk.ExecutionSample recording.jfr

You’ll see stack traces captured cooperatively. Compare with old recordings – you’ll notice fewer anomalous single-frame traces (artifacts of failed async stack walks).

Example 2: Analyzing Safepoint Latency

Cooperative sampling adds a bonus: the EventSafepointLatency event. When a thread processes its sample request at a safepoint, it records how long it waited:

// Emitted automatically by JFR
EventSafepointLatency {
    startTime = 12:34:56.789  // Sample captured
    endTime   = 12:34:56.791  // Safepoint reached
    duration  = 2ms           // Time to safepoint
    threadState = RUNNABLE
    stackTrace = [hotMethod:line42, ...]
}

This reveals which methods delay safepoint arrival. Long-running loops without polls show up as high safepoint latency. Example:

public void badLoop() {
    for (int i = 0; i < Integer.MAX_VALUE; i++) {
        // No method calls, no back edges with polls
        compute(i);
    }
}

If sampled mid-loop, safepoint latency = remaining loop time. This identifies safepoint starvation issues that degrade GC pause times.

JFR Mission Control visualizes these events, highlighting methods that block safepoints. You can then add explicit poll checks:

public void goodLoop() {
    for (int i = 0; i < Integer.MAX_VALUE; i++) {
        if (i % 1000 == 0) {
            Thread.onSpinWait();  // Compiler inserts poll
        }
        compute(i);
    }
}

Example 3: Virtual Thread Profiling

Virtual threads (Project Loom) use continuation frames stored on the heap. Cooperative sampling handles these correctly:

try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
    for (int i = 0; i < 10_000; i++) {
        executor.submit(() -> {
            // CPU-bound work
            compute();
        });
    }
}

When sampling a virtual thread, compute_top_frame checks:

static inline bool is_in_continuation(const frame& frame, JavaThread* jt) {
    return JfrThreadLocal::is_vthread(jt) &&
           (Continuation::is_frame_in_continuation(jt, frame) ||
            Continuation::is_continuation_enterSpecial(frame));
}

If true, the stack walk stops at the continuation boundary. The resulting trace correctly attributes CPU time to the virtual thread, not the carrier thread. This is crucial for profiling applications with thousands of virtual threads.

Example 4: Diagnostic Flag for Native Code

Cooperative sampling doesn’t help threads stuck in native code (they never hit safepoints). For these, JFR falls back to async sampling with the inherent risks. You can disable native sampling entirely:

java -XX:FlightRecorderOptions=samplethreads=java \
     -XX:StartFlightRecording=settings=profile \
     -jar app.jar

The samplethreads=java option skips threads in native state. Use this in environments where JNI code is known to be unstable or uses unusual stack layouts. You’ll lose visibility into native methods, but gain crash safety.

Check if a thread is samplable:

// JFR decides based on thread state
if (jt->thread_state() == _thread_in_Java) {
    // Cooperative sampling
} else if (jt->thread_state() == _thread_in_native) {
    // Async sampling (risky) or skip
}

Migration Considerations

JEP 518 is transparent to most users – JFR profiling works exactly as before, just safer. However, there are edge cases.

Breaking Changes

None for normal usage. The jdk.ExecutionSample event structure is unchanged. Existing JFR analysis tools continue to work.

Behavioral Changes

1. Slightly different sample distribution: Bias adjustment changes which frames appear in profiles. If you relied on precise sample counts for specific methods, numbers will shift slightly (usually toward more accuracy).

2. Safepoint latency overhead: Processing samples at safepoints adds ~1-10μs per safepoint. For applications with extremely tight safepoint budgets (HFT, real-time), monitor safepoint times.

3. Native sampling caveat: Native code sampling still uses async mechanism. If you profiled JNI-heavy workloads and experienced crashes, they’re not fixed. Use samplethreads=java to disable native sampling.

Compatibility

• JDK 25+: Cooperative sampling enabled by default
• JDK 24 and earlier: Old async sampling

No flags to control the mechanism – it’s an internal implementation change. If you encounter issues (unlikely), file a JDK bug report.

Best Practices

1. Increase sampling frequency: Safer mechanism allows more aggressive profiling. Try 10ms or even 5ms intervals for finer granularity:

jfr configure jdk.ExecutionSample period=10ms

2. Monitor safepoint latency: Use EventSafepointLatency to identify long-running methods without polls. The new mechanism makes this data actionable.

3. Trust the data more: Fewer spurious single-frame traces. If you see them, investigate – they’re likely real, not sampling artifacts.

4. Virtual threads: Cooperative sampling is essential for accurate virtual thread profiling. If profiling Loom apps, JDK 25+ is mandatory.

Conclusions

JEP 518 fundamentally improves JFR’s reliability by eliminating the async stack walking that plagued earlier versions. The cooperative approach – capture PC/SP, defer work to safepoint – is a textbook example of “don’t fight the VM, work with it.”

Key benefits:

• Stability: No more crashes from heuristic stack walking failures
• Scalability: Sampler thread overhead reduced 10x, enabling 1000+ thread profiling
• Accuracy: Bias adjustment recovers true sample locations in ~70% of cases
• Foundation: Enables future enhancements like CPU-time profiling (JEP 509)

The architecture is cleaner: sampler creates lightweight requests, target does heavy lifting in safe context. Code complexity drops, maintenance burden decreases, and new sampling modalities become feasible.

For Java developers, the message is simple: turn on JFR profiling in production. The crash risks that made some teams cautious are gone. With negligible overhead (~0.05% CPU), always-on profiling should be standard practice for performance-sensitive applications.

For JVM engineers, JEP 518 demonstrates how to leverage safepoint infrastructure beyond just GC and deopt. Any cross-thread introspection can benefit from the cooperative pattern – suspend, create request, arm poll, resume, process at safepoint. Expect more VM features to adopt this approach.

References

• JEP 518
• JEP 509: JFR CPU-Time Profiling
• Safepoint bias research: “Evaluating the Accuracy of Java Profilers” (Mytkowicz et al., PLDI 2010)
• Implementation: jfr/periodic/sampling/ in OpenJDK
• JFR Guide
• Safepoint mechanism: safepointMechanism.hpp