Support test automation exclusions in Karate

[calvinbayer]

What Does This Do

Support test automation exclusions in Karate. Wire test declaration metadata into Datadog policy selection and reporting, preserving assigned Test Management properties.

Motivation

Allow individual tests to prevent Datadog quarantine/disabling and exclude Datadog ATR, EFD and Attempt to Fix. Developer selection and framework retries retain their existing semantics.

Additional Notes

Stacked on codex/cucumber-automation-exclusions; review this PR against its base for the framework-specific change. Shared public API and policy are introduced by the first JUnit4 PR. No refactor commit was needed.

Validation: Both Karate1 and Karate2 module spotlessApply and instrumented exclusion regressions passed on installed JDK21. Each protected scenario executes once with blocking failure, annotation tags and preserved assignment tags.

Snapshot dependency for inherited physical source paths: DataDog/test-environment#604.

test-environment-branch: codex/java-inherited-test-source-snapshots

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - sbt-scalatest

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	58.26	56.55	$\color{red}{\blacktriangle}$ +1.71	56.55	$\color{red}{\blacktriangle}$ +1.71	177/417
agentEvpProxy	58.40	n/a	n/a	n/a	n/a	-

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - nebula-release-plugin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	35.18	37.15	$\color{green}{\blacktriangledown}$ -1.97	37.15	$\color{green}{\blacktriangledown}$ -1.97	88/221
agentless	37.09	37.15	$\color{green}{\blacktriangledown}$ -0.06	37.15	$\color{green}{\blacktriangledown}$ -0.06	88/194
agentlessCodeCoverage	43.90	46.30	$\color{green}{\blacktriangledown}$ -2.40	45.38	$\color{green}{\blacktriangledown}$ -1.48	88/193
agentlessLineCoverage	56.99	56.55	$\color{red}{\blacktriangle}$ +0.44	56.55	$\color{red}{\blacktriangle}$ +0.44	88/193

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - heliboard

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	11.36	9.73	$\color{red}{\blacktriangle}$ +1.63	9.92	$\color{red}{\blacktriangle}$ +1.44	88/193

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - netflix-zuul

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	89.85	89.58	$\color{red}{\blacktriangle}$ +0.27	89.58	$\color{red}{\blacktriangle}$ +0.27	88/222
agentless	82.18	82.69	$\color{green}{\blacktriangledown}$ -0.51	82.69	$\color{green}{\blacktriangledown}$ -0.51	88/193
agentlessCodeCoverage	96.91	97.04	$\color{green}{\blacktriangledown}$ -0.13	97.04	$\color{green}{\blacktriangledown}$ -0.13	88/193
agentlessLineCoverage	113.43	113.88	$\color{green}{\blacktriangledown}$ -0.45	113.88	$\color{green}{\blacktriangledown}$ -0.45	89/194

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - pass4s

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	11.13	10.54	$\color{red}{\blacktriangle}$ +0.59	10.13	$\color{red}{\blacktriangle}$ +1.00	89/220
agentless	12.80	9.92	$\color{red}{\blacktriangle}$ +2.88	9.92	$\color{red}{\blacktriangle}$ +2.88	88/190
agentlessCodeCoverage	15.14	17.38	$\color{green}{\blacktriangledown}$ -2.24	17.73	$\color{green}{\blacktriangledown}$ -2.59	80/175

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - reactive-streams-jvm

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	22.04	22.54	$\color{green}{\blacktriangledown}$ -0.50	22.54	$\color{green}{\blacktriangledown}$ -0.50	118/252
agentless	18.64	19.59	$\color{green}{\blacktriangledown}$ -0.95	19.59	$\color{green}{\blacktriangledown}$ -0.95	114/216
agentlessCodeCoverage	20.16	20.39	$\color{green}{\blacktriangledown}$ -0.23	20.39	$\color{green}{\blacktriangledown}$ -0.23	115/217
agentlessLineCoverage	26.40	27.52	$\color{green}{\blacktriangledown}$ -1.12	27.52	$\color{green}{\blacktriangledown}$ -1.12	79/153

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - sonar-kotlin

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	14.71	13.40	$\color{red}{\blacktriangle}$ +1.31	13.40	$\color{red}{\blacktriangle}$ +1.31	100/235
agentless	11.98	12.37	$\color{green}{\blacktriangledown}$ -0.39	12.37	$\color{green}{\blacktriangledown}$ -0.39	100/205
agentlessCodeCoverage	15.73	15.72	$\color{red}{\blacktriangle}$ +0.01	15.41	$\color{red}{\blacktriangle}$ +0.32	100/205
agentlessLineCoverage	17.44	17.38	$\color{red}{\blacktriangle}$ +0.06	17.73	$\color{green}{\blacktriangledown}$ -0.29	100/205

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - jolokia

Job Status: 🟢 success

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	97.26	97.04	$\color{red}{\blacktriangle}$ +0.22	95.12	$\color{red}{\blacktriangle}$ +2.14	103/237
agentless	92.84	91.39	$\color{red}{\blacktriangle}$ +1.45	91.39	$\color{red}{\blacktriangle}$ +1.45	103/209
agentlessCodeCoverage	100.69	101.00	$\color{green}{\blacktriangledown}$ -0.31	99.00	$\color{red}{\blacktriangle}$ +1.69	103/209
agentlessLineCoverage	104.18	103.04	$\color{red}{\blacktriangle}$ +1.14	101.00	$\color{red}{\blacktriangle}$ +3.18	115/227

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[datadog-official]

🎯 Code Coverage (details)
• Patch Coverage: 100.00%
• Overall Coverage: 59.92% (+0.00%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 2d3b70b | Docs | Give us feedback!}

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - okhttp

Job Status: 🔴 failed

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	20.00	19.20	$\color{red}{\blacktriangle}$ +0.80	19.59	$\color{red}{\blacktriangle}$ +0.41	103/236
agentless	19.33	18.45	$\color{red}{\blacktriangle}$ +0.88	19.20	$\color{red}{\blacktriangle}$ +0.13	102/211
agentlessCodeCoverage	22.42	21.65	$\color{red}{\blacktriangle}$ +0.77	22.54	$\color{green}{\blacktriangledown}$ -0.12	104/215
agentlessLineCoverage	40.76	37.91	$\color{red}{\blacktriangle}$ +2.85	38.67	$\color{red}{\blacktriangle}$ +2.09	101/209

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - spring_boot

Job Status: 🔴 failed

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	9.47	16.36	$\color{green}{\blacktriangledown}$ -6.89	16.36	$\color{green}{\blacktriangledown}$ -6.89	100/233
agentless	10.07	9.92	$\color{red}{\blacktriangle}$ +0.15	9.92	$\color{red}{\blacktriangle}$ +0.15	100/205
agentlessCodeCoverage	13.18	13.94	$\color{green}{\blacktriangledown}$ -0.76	13.67	$\color{green}{\blacktriangledown}$ -0.49	100/207
agentlessLineCoverage	22.40	22.99	$\color{green}{\blacktriangledown}$ -0.59	22.54	$\color{green}{\blacktriangledown}$ -0.14	100/205

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.00 s	13.97 s	[-0.5%; +0.9%] (no difference)
startup:insecure-bank:tracing:Agent	12.92 s	12.95 s	[-1.1%; +0.7%] (no difference)
startup:petclinic:appsec:Agent	17.10 s	16.98 s	[-0.1%; +1.6%] (no difference)
startup:petclinic:iast:Agent	16.83 s	17.00 s	[-1.7%; -0.3%] (maybe better)
startup:petclinic:profiling:Agent	16.69 s	16.72 s	[-1.4%; +1.1%] (no difference)
startup:petclinic:sca:Agent	17.02 s	16.90 s	[-0.2%; +1.7%] (no difference)
startup:petclinic:tracing:Agent	16.07 s	16.37 s	[-3.0%; -0.8%] (maybe better)

Commit: 2d3b70be · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-10-19T22:17:07.831595Z	9fda4c1	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[cit-pr-commenter-54b7da]

CI Visibility Test Environment - sonar-java

Job Status: 🔴 failed

Scenario	This PR (%)	7d median	Δ 7d	30d median	Δ 30d	runs (7d/30d)
agent	26.83	14.51	$\color{red}{\blacktriangle}$ +12.32	14.23	$\color{red}{\blacktriangle}$ +12.60	100/233
agentless	27.30	12.62	$\color{red}{\blacktriangle}$ +14.68	12.87	$\color{red}{\blacktriangle}$ +14.43	100/207
agentlessCodeCoverage	82.12	79.45	$\color{red}{\blacktriangle}$ +2.67	79.45	$\color{red}{\blacktriangle}$ +2.67	100/207
agentlessLineCoverage	135.42	120.92	$\color{red}{\blacktriangle}$ +14.50	123.36	$\color{red}{\blacktriangle}$ +12.06	100/208

Baseline: median of @test.tracer_overhead on main (gitlab) over the last 7/30 days, per OSS project & scenario. Δ = this PR − baseline median; red ▲ = more overhead, green ▽ = less overhead than baseline.

Tests run to verify that CI Visibility behavior has not regressed in the current PR.

[datadog-official]

More details

Both Karate adapters now pass scenario tags into exclusion-aware skip decisions, consistent with existing retry policy selection and reporting. The completed static review identified no actionable regressions.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 9fda4c1}

[datadog-official]

TestNG can re-enable Datadog retries after an annotation transformer explicitly disables them, causing tests to execute repeatedly despite the framework override.

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 1b32475}

---

Findings that could not be posted inline

dd-java-agent/instrumentation/testng/testng-common/src/main/java/datadog/trace/instrumentation/testng/TestNGUtils.java:78

Preserve transformer overrides that disable retries

When an unexcluded TestNG test's annotation transformer selects DisabledRetryAnalyzer, hasFrameworkOverride() is true, but the disabled-analyzer check falls through to installing Datadog's RetryAnalyzer. With ATR or EFD enabled, a test intended to run once can execute repeatedly. Previously, the user transformer's decision prevailed. Preserve explicit framework overrides separately from the default absence of a retry analyzer.

    if (RetryAnnotationTransformer.hasFrameworkOverride(annotation)) {
      return;
    }
    if (TestExclusions.noAutoRetry(source, Arrays.asList(method.getGroups()))) {

React: 👍 useful · 👎 wrong · 😕 unclear · 👀 right, but not acting on it

What each reaction means

• 👍 Useful: correct, and you're fixing it or will.
• 👎 Wrong: the finding is incorrect, for example the bug can't happen or the code was misread.
• 😕 Unclear: you can't tell what it means or why it matters.
• 👀 Right, not acting: correct, but intended, out of scope for this PR, a known risk, or too minor.

_{🤖 Bits Code Review · Open Bits AI session}

[calvinbayer]

Addressed the finding in this review with the inherited fix d308a2b.

The regression first reproduced the bug: an explicit DisabledRetryAnalyzer override still executed six times with TestNG 7.0 ATR, and five/three times with latest TestNG ATR/EFD. The transformer now preserves every explicit framework override before installing Datadog policy. The full configured suite passes 308 cases with 101 capability skips, plus Muzzle, forbidden-API and module checks. The supported scenarios execute exactly once and emit no retry metadata.

[calvinbayer]

@autotest Review

[datadog-official]

More details

Both Karate versions pass scenario tags into skip selection consistently with their existing execution-policy and reporting paths, preserving the metadata needed to honor exclusions and retain Test Management assignments.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 2d52691}

[calvinbayer]

@autotest Review

[datadog-official]

More details

Both Karate integrations pass scenario tags into exclusion-aware skip selection, consistent with existing retry selection and reporting paths. The static review found no reportable defect.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Bits Code Review · Commit 2d3b70b}