Repository navigation
spec: issue-3635 End-to-end test matrix for Chainloop Trace agents - #3636
Conversation
Refs chainloop-dev#3635 Assisted-by: Claude Code Signed-off-by: Jose I. Paris <jiparis@chainloop.dev> Chainloop-Trace-Sessions: ced12b99-da36-48e4-bc20-8f22055141f5
PR validation — ✅ 3 passing
AI Session Checks — 🟢 87% ·
|
| Avg score | Sessions | Failing policies | Attribution | Files | Lines | Total Duration |
|---|---|---|---|---|---|---|
| 🟢 87% | 2 | 100% AI / 0% Human | 1 | +273 / -14 | 4h2m5s |
🟢 91% — 100% AI — ⚠️ 1 policies failing
-
Oct 16, 2026 08:14 UTC · 4h2m5s · $7.64 · 194 in / 85.0k out · claude-code 2.1.301 (claude-opus-5-5)
Change Summary
-
- Adds a spec for end-to-end Chainloop Trace harness matrix testing.
- Defines a README-driven expected matrix, headless harness adapters, and nightly/manual reporting.
- Revises the draft after review comments to pin harness versions and fail closed on unmapped rows.
AI Session Overall Score
-
🟢 91% — Spec-only session was well-scoped, aligned, and calmly executed.
AI Session Analysis Breakdown
-
🟢 95% · scope-discipline
-
🟢 Work stayed on one spec file plus the issue and PR flow the skill required. · High Impact
🟢 92% · alignment
-
No notes.
🟢 90% · solution-quality
-
🟢 Review comments were absorbed into explicit spec decisions instead of hand-waved away. · Medium Impact
🟢 89% · context-and-planning
-
🟢 The eng-spec instructions supplied strong workflow and drafting constraints up front. · High Impact
🟢 88% · user-trust-signal
-
No notes.
abstained · verification
-
🟡 Verification abstained because this was a spec-only drafting session with no runtime behavior to validate. · Low Severity
-
File Attribution
████████████████████100% AI / 0% HumanStatus Attribution File Lines modified ai docs/specs/issue-3635-trace-e2e-matrix.md+24 / -14
Policies (5, 1 failing)
Status Policy Material Messages ✅ Passed ai-config-no-dangerous-commandsai-coding-session-b59d3e- ⚠️ Failedai-session-no-auto-prai-coding-session-b59d3ePull request opened without human approval (auto-approved: source=config, reason=classifier): git push -u origin issue-3635-spec-trace-e2e-matrix 2>&1 | tail -3 && gh pr create --repo chainloop-dev/chainloop --base main --head jiparis:issue-3635-spec-tra... ✅ Passed ai-config-ai-agents-allowedai-coding-session-b59d3e- ✅ Passed ai-config-no-secretsai-coding-session-b59d3e- ✅ Passed ai-config-mcp-servers-allowedai-coding-session-b59d3e- -
🟢 83% — 100% AI — ⚠️ 1 policies failing
-
Oct 16, 2026 08:14 UTC · 1h21m3s · $5.00 · 184 in / 65.7k out · claude-code 2.1.301 (claude-opus-5-5)
Change Summary
-
- Adds a public design spec for an end-to-end Trace harness matrix and nightly/manual runs.
- Adds the spec-capture metadata file for that new spec.
- Opens a spec-only PR tied to a public GitHub issue linked from the Linear ticket.
AI Session Overall Score
-
🟢 83% — Strong spec session, but review one user-picked rule the AI changed.
AI Session Analysis Breakdown
-
🟢 93% · scope-discipline
-
🟢 Repo edits stayed concentrated in the spec and its capture metadata. · High Impact
🟢 91% · context-and-planning
-
🟢 User and skill instructions defined the spec workflow before drafting started. · High Impact
🟢 88% · solution-quality
-
🟢 AI fixed lint findings rather than weakening the prose checks. · Medium Impact
🟢 84% · user-trust-signal
-
No notes.
🟡 68% · alignment
-
🟠 AI changed one user-picked baseline rule from
NotoUnknowncells without re-asking. · Medium Severity💡 When a picked option becomes untenable, pause and ask before rewriting the decision.
abstained · verification
-
🟡 Verification abstained: this was a docs-only spec PR with only prose checks. · Low Severity
-
File Attribution
████████████████████100% AI / 0% HumanStatus Attribution File Lines created ai docs/specs/issue-3635-trace-e2e-matrix.md+249 / -0
Policies (5, 1 failing)
Status Policy Material Messages ⚠️ Failedai-session-no-auto-prai-coding-session-ced12bPull request opened without human approval (auto-approved: source=config, reason=classifier): git push -u origin issue-3635-spec-trace-e2e-matrix 2>&1 | tail -3 && gh pr create --repo chainloop-dev/chainloop --base main --head jiparis:issue-3635-spec-tra... ✅ Passed ai-config-ai-agents-allowedai-coding-session-ced12b- ✅ Passed ai-config-no-dangerous-commandsai-coding-session-ced12b- ✅ Passed ai-config-no-secretsai-coding-session-ced12b- ✅ Passed ai-config-mcp-servers-allowedai-coding-session-ced12b- -
Security Checks — ✅ 5 passing
✅ secret-scan
| Status | Policy | Messages |
|---|---|---|
| ✅ Passed | secrets-detection |
- |
✅ sast-scan
| Status | Policy | Messages |
|---|---|---|
| ✅ Passed | owasp-top10-2025 |
- |
| ✅ Passed | sast |
- |
| ✅ Passed | cwe-top25 |
- |
| ✅ Passed | cwe-top26-40-cusp |
- |
Scans not applied (3)
| Scan | Reason |
|---|---|
vulnerability-scan |
no manifest/lockfile changed |
github-actions-scan |
no workflow files changed |
iac-scan |
no IaC files changed |
Security context
✅ Nothing this change touches has a recorded security-fix history.
View in Chainloop ↗ · How this works ↗
Powered by Chainloop and Chainloop Trace
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
All reported issues were addressed across 1 file
Heads up: you’ve reached your flex budget. Increase your flex budget or wait for usage to reset.
View guided diff | Re-trigger cubic
Report control-plane cells as not covered, map table rows to checks and fail closed per row, and pin one harness version per column (D-013 supersedes D-006). Refs chainloop-dev#3635 Assisted-by: Claude Code Signed-off-by: Jose I. Paris <jiparis@chainloop.dev> Chainloop-Trace-Sessions: b59d3ea2-2be3-4f68-9145-8a7bd56858cd
3 participants
Summary
Chainloop Trace supports several coding agents ("harnesses"): Claude Code, Cursor, OpenCode, and soon Pi. The Trace README has a feature table that says which features each harness supports. No test checks this table against the real harnesses, and the table can miss features that Trace already has.
This spec first reviews the table for completeness. Then it adds an end-to-end test suite. The suite runs each harness in headless mode with real models, in a throwaway repository with Trace in local storage mode. It checks each feature that the table marks as supported, and it builds a result matrix of harness, model, and feature. A nightly job compares the result with the previous run and posts a Slack message when a cell changes from green to red. The suite does not test features that the table marks "No".
Open questions for reviewers
We would like two reviewers to approve this spec before it merges.
Refs #3635
AI disclosure: Claude Code helped to draft this spec.