Fix Kotlin coroutine context isolation and continuation ownership by mcculls · Pull Request #12914 · DataDog/dd-trace-java

mcculls · 2026-10-16T14:48:38Z

What Does This Do

Fix Kotlin coroutine context isolation and trace-continuation ownership:

Give builder-created child coroutines fresh Datadog context elements, preserving their position and callback order in the supplied context.
Let only the owning coroutine release an element's continuation, so scoped coroutines cannot release their caller's trace hold. Handle children launched with an already-cancelled parent.
Avoid redundant continuations for structured children when their parent already keeps the same local root span's trace open.

Add regression coverage for context replacement, callback order, and trace lifetimes. Reuse the flow and lifecycle tests across coroutines 1.6.4, 1.8.1, and the latest 1.x release (locked to 1.11.0), alongside the minimum-version suite on 1.3.0. Regenerate the module lockfile.

Motivation

Reusing a parent's context element can mix independent coroutine state, while completing a scoped coroutine can release the parent's continuation too early. These fixes preserve context isolation and keep traces open until their coroutine work finishes.

Additional Notes

Extracted from #12864. This PR retains the existing context capture and swap/restore behavior; the shared scope-stack implementation and AgentTracer/CoreTracer/scope-manager changes remain in PR #12864.

Validation on JDK 17:

All four coroutine version suites pass: 562 tests per matrix, no failures or skips.
Note that these tests currently force use of the legacy context manager.
Compilation and module spotlessCheck pass.

Extract the Kotlin fixes and independent regression tests from PR #12864. Give builder-created coroutines fresh context elements while preserving context element order. Only release an element's continuation when its owning coroutine completes, including children of already-cancelled jobs. Skip redundant trace continuations for eligible structured children. Keep the existing context capture and swap/restore behavior. Parent live stack seeding, shared scope stacks, AgentTracer/CoreTracer/scope-manager changes, and the handoff tests requiring them are deliberately excluded. Include context replacement and trace-lifetime regressions, run the latest dependency suite on kotlinx-coroutines 1.8, and regenerate the module lockfile. Source: #12864 Validation: test and latestDepTest on JDK 17 (276 tests per run), with both legacy and non-legacy context managers; module spotlessCheck.

Add coroutines16Test and coroutines18Test suites pinned to 1.6.4 and 1.8.1, reusing the existing latest-dependency test sources and shared fixtures. Keep the minimum-version suite on 1.3.0 and advance latestDepTest to 1.+, currently locked to 1.11.0. Give the flow builders explicit Int element types so the shared tests compile with the newer dependency using the existing Kotlin compiler. Regenerate the module dependency lockfile for all suites. Validation: all four suites on JDK 17 with both legacy and non-legacy context managers (562 tests per matrix); module spotlessCheck.

Identify the trace held open by a coroutine or its parent using the local root AgentSpan rather than AgentTraceCollector. Compare local root span identity when deciding whether a structured child needs its own trace continuation, and rename the ownership fields and helper accordingly. Validation: all four coroutine version suites on JDK 17 with legacy and non-legacy context managers (562 tests per matrix); module spotlessCheck.

datadog-datadog-prod-us1-2 · 2026-10-16T15:11:55Z

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 59.79% (-0.05%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 3717bb6 | Docs | Give us feedback!}

dd-octo-sts · 2026-10-16T15:17:58Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	13.95 s	13.89 s	[-0.3%; +1.2%] (no difference)
startup:insecure-bank:tracing:Agent	12.85 s	12.88 s	[-0.9%; +0.5%] (no difference)
startup:petclinic:appsec:Agent	17.09 s	16.84 s	[+0.4%; +2.6%] (maybe worse)
startup:petclinic:iast:Agent	16.94 s	16.93 s	[-0.8%; +0.9%] (no difference)
startup:petclinic:profiling:Agent	16.62 s	16.77 s	[-1.8%; -0.0%] (maybe better)
startup:petclinic:sca:Agent	16.47 s	16.77 s	[-6.1%; +2.4%] (no difference)
startup:petclinic:tracing:Agent	16.08 s	16.06 s	[-0.8%; +0.9%] (no difference)

Commit: 3717bb6f · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

chatgpt-codex-connector · 2026-10-16T16:10:05Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-16T16:14:23.397304Z	`9a8b48c`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

ygree

Nice catch!

dd-octo-sts · 2026-10-19T09:31:12Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-10-19T09:31:17Z

View all feedbacks in Devflow UI.

2026-10-19 09:31:16 UTC ℹ️ Start processing command /merge

2026-10-19 09:31:21 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).

2026-10-19 10:12:08 UTC ❌ MergeQueue: The build pipeline contains failing jobs for this merge request

Build pipeline has failing jobs for 726ca5e:

test_inst: [ibm8, 2/8]

⚠️ Do NOT retry failed jobs directly (why?).

What to do next?

Investigate the failures and when ready, re-add your pull request to the queue!
If your PR checks are green, try to rebase/merge. It might be because the CI run is a bit old.
Any question, go check the FAQ.

Details

Since those jobs are not marked as being allowed to fail, the pipeline will most likely fail.
Therefore, and to allow other builds to be processed, this merge request has been rejected and the pipeline got canceled.

mcculls · 2026-10-19T10:17:55Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-10-19T10:18:00Z

View all feedbacks in Devflow UI.

2026-10-19 10:17:59 UTC ℹ️ Start processing command /merge

2026-10-19 10:18:04 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).

2026-10-19 10:33:33 UTC ⚠️ MergeQueue: This merge request build was cancelled

stuart.mcculloch@datadoghq.com cancelled this merge request build

mcculls · 2026-10-19T10:32:56Z

/merge --cancel

gh-worker-devflow-routing-ef8351 · 2026-10-19T10:33:01Z

View all feedbacks in Devflow UI.

2026-10-19 10:33:01 UTC ℹ️ Start processing command /merge --cancel

…atadog element

…manager

dd-octo-sts · 2026-10-19T13:01:28Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-10-19T13:01:32Z

View all feedbacks in Devflow UI.

2026-10-19 13:01:32 UTC ℹ️ Start processing command /merge

2026-10-19 13:01:36 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).

2026-10-19 14:16:32 UTC ℹ️ MergeQueue: This merge request was merged

mcculls added 3 commits October 16, 2026 15:35

mcculls added type: bug fix Bug fix comp: context propagation Trace context propagation inst: kotlin Kotlin instrumentation tag: ai generated Largely based on code generated by an AI or LLM and removed comp: context propagation Trace context propagation labels Oct 16, 2026

mcculls marked this pull request as ready for review October 16, 2026 16:09

mcculls requested review from a team as code owners October 16, 2026 16:09

mcculls requested review from PerfectSlayer and ygree and removed request for a team October 16, 2026 16:09

mcculls requested a review from amarziali October 16, 2026 16:10

This comment was marked as outdated.

Sign in to view

mcculls mentioned this pull request Oct 16, 2026

Fix Kotlin coroutines losing scopes when resumed on another thread #12864

Open

ygree reviewed Oct 16, 2026

View reviewed changes

Comment thread dd-java-agent/instrumentation/kotlin-coroutines-1.3/build.gradle Outdated

mcculls added 2 commits October 16, 2026 18:27

Release completed coroutine owners with a sentinel

c81cde8

Align coroutine test dependencies with their core modules

d4ed998

ygree approved these changes Oct 16, 2026

View reviewed changes

amarziali approved these changes Oct 19, 2026

View reviewed changes

mcculls added this pull request to the merge queue Oct 19, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 19, 2026

mcculls added 2 commits October 19, 2026 11:55

Preserve active span changes when a scoped coroutine reinstalls the D…

54df510

…atadog element

Mark this lazy coroutine test as flaky when using the legacy context …

3717bb6

…manager

mcculls added this pull request to the merge queue Oct 19, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 19, 2026

gh-worker-dd-mergequeue-cf854d Bot merged commit ff29f41 into master Oct 19, 2026
609 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the mcculls/kotlin-independent-fixes branch October 19, 2026 14:16

github-actions Bot added this to the 1.68.0 milestone Oct 19, 2026

Conversation

mcculls commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

Motivation

Additional Notes

Uh oh!

datadog-datadog-prod-us1-2 Bot commented Oct 16, 2026 • edited by datadog-prod-us1-3 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dd-octo-sts Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

chatgpt-codex-connector Bot commented Oct 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

ygree left a comment

Choose a reason for hiding this comment

Uh oh!

dd-octo-sts Bot commented Oct 19, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What to do next?

Uh oh!

Uh oh!

mcculls commented Oct 19, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mcculls commented Oct 19, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dd-octo-sts Bot commented Oct 19, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Oct 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mcculls commented Oct 16, 2026 •

edited

Loading

datadog-datadog-prod-us1-2 Bot commented Oct 16, 2026 •

edited by datadog-prod-us1-3 Bot

Loading

dd-octo-sts Bot commented Oct 16, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 16, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Oct 19, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Oct 19, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Oct 19, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Oct 19, 2026 •

edited

Loading