Repository navigation
Conversation
AI Session Checks — 🟡 85% ·
|
| Avg score | Sessions | Failing policies | Attribution | Files | Lines | Total Duration |
|---|---|---|---|---|---|---|
| 🟡 85% | 2 | 63% AI / 37% Human | 41 | +1331 / -116 | 76h36m15s |
🟢 87% — 0% AI — ⚠️ 2 policies failing
-
Oct 9, 2026 06:24 UTC · 2h2m38s · $93.51 · 2.0k in / 602.2k out · claude-code 2.1.295 (claude-opus-5-5)
Change Summary
-
- Adds server-side PR commit verification and passes the result into
CHAINLOOP_SANDBOX. - Extends the sandbox runner and shared verification types/tests to consume the serialized result.
- Exercises unit and end-to-end GitLab/GitHub flows, then opens a spec PR documenting the gap and design.
- Adds server-side PR commit verification and passes the result into
AI Session Overall Score
-
🟢 87% — Broad task landed cleanly; only the upfront planning remained under-documented.
AI Session Analysis Breakdown
-
🟢 93% · user-trust-signal
-
No notes.
🟢 92% · alignment
-
🟢 Later history and spec work started only after explicit follow-up asks. · High Impact
🟢 88% · scope-discipline
-
No notes.
🟢 88% · verification
-
🟢 It exercised failing tests, targeted suites, and GitLab/GitHub end-to-end paths. · High Impact
🟢 86% · solution-quality
-
🟢 The fix keeps commit verification server-side instead of leaking SCM tokens. · High Impact
🟡 68% · context-and-planning
-
🟠 Wide multi-repo work ran without a shared plan or TODO artifact, so setup stayed implicit. · Medium Severity
💡 Before cross-repo implementation and e2e setup, write a short plan naming phases, repos, and validation steps.
-
File Attribution
░░░░░░░░░░░░░░░░░░░░0% AI / 100% Human
Policies (5, 2 failing)
Status Policy Material Messages ⚠️ Failedai-session-no-auto-prai-coding-session-0ebed9Pull request opened without human approval (auto-approved: source=config, reason=classifier): cd /home/javi/orca/workspaces/platform/pfm-7678-spec && git push -q -u origin PFM-7678-spec-pr-attestation-commit-verification 2>&1 | grep -v "^remote: *$" | ta... ✅ Passed ai-config-ai-agents-allowedai-coding-session-0ebed9- ✅ Passed ai-config-no-dangerous-commandsai-coding-session-0ebed9- ⚠️ Failedai-config-no-secretsai-coding-session-0ebed9- Secret (generic-password) detected in session content [turn=30, source=tool_result, line=148]: 148 - Username:
[CHAINLOOP_TRACE_REDACTED:generic-password] - Secret (generic-password) detected in session content [turn=30, source=tool_result, line=149]: 149 - Password:
[CHAINLOOP_TRACE_REDACTED:generic-password] - Secret (generic-password) detected in session content [turn=30, source=tool_result, line=195]: 195 | rustfs (optional) | 9002 (API)
9003 (console) | S3-compatible storage backend |[CHAINLOOP_TRACE_REDACTED:generic-password]/[CHAINLOOP_TRACE_REDACTED:generic-password]| - Secret (generic-password) detected in session content [turn=407, source=assistant-tool_use:Bash, line=1]: L=/home/javi/orca/workspaces/platform/pfm-7678-record-commit-signature-verification-in/deployment/lab/config/control-plane/config.devel.yaml; grep -nE "^[a-z_]+:|policy_providers|url:|default:|name:|a...
- Secret (generic-password) detected in session content [turn=412, source=assistant-tool_use:Bash, line=1]: L=/home/javi/orca/workspaces/platform/pfm-7678-record-commit-signature-verification-in/deployment/lab/config/control-plane/config.devel.yaml; sed -n 1,20p $L; sed -n 46,58p $L; sed -n 69,110p $L | sed...
- Secret (generic-password) detected in session content [turn=414, source=tool_result, line=39]: client_secret: [CHAINLOOP_TRACE_REDACTED:generic-password]
- Secret (generic-password) detected in session content [turn=414, source=tool_result, line=42]: generated_jws_hmac_secret: [CHAINLOOP_TRACE_REDACTED:generic-password]
- Secret (generic-password) detected in session content [turn=414, source=tool_result, line=45]: cas_robot_account_private_key_path: [CHAINLOOP_TRACE_REDACTED:generic-password]
- Secret (generic-password) detected in session content [turn=425, source=tool_result, line=19]: 19 password: '[CHAINLOOP_TRACE_REDACTED:generic-password]',
- Secret (generic-password) detected in session content [turn=47, source=assistant-tool_use:Bash, line=1]: R=/home/javi/orca/workspaces/platform/pfm-7678-record-commit-signature-verification-in/deployment/lab/config/backend/config.devel.yaml; cat -n $R | sed -E 's/((secret|token|password|key|hmac)[a-z_]*: ...
- Secret (generic-password) detected in session content [turn=49, source=tool_result, line=101]: 101 # password: [CHAINLOOP_TRACE_REDACTED:generic-password]
- Secret (generic-password) detected in session content [turn=49, source=tool_result, line=21]: 21 ui_server_token: [CHAINLOOP_TRACE_REDACTED:generic-password]
- Secret (generic-password) detected in session content [turn=49, source=tool_result, line=44]: 44 chainloop_jws_hmac_secret: [CHAINLOOP_TRACE_REDACTED:generic-password]
- Secret (generic-password) detected in session content [turn=49, source=tool_result, line=72]: 72 webhook_secret: [CHAINLOOP_TRACE_REDACTED:generic-password]
- Secret (generic-password) detected in session content [turn=49, source=tool_result, line=74]: 74 secrets_backend:[CHAINLOOP_TRACE_REDACTED:generic-password]
- Secret (generic-password) detected in session content [turn=49, source=tool_result, line=77]: 77 token: [CHAINLOOP_TRACE_REDACTED:generic-password]
- Secret (generic-password) detected in session content [turn=49, source=tool_result, line=78]: 78 secret_prefix: [CHAINLOOP_TRACE_REDACTED:generic-password]
- Secret (generic-password) detected in session content [turn=599, source=assistant-tool_use:Bash, line=1]: B=/home/javi/orca/workspaces/platform/pfm-7678-record-commit-signature-verification-in/backend; grep -n "sandbox:" -A 40 $B/configs/samples/config.yaml | sed -E 's/((secret|token|password|key|api_key)...
- Secret (generic-password) detected in session content [turn=601, source=tool_result, line=11]: 73- token: [CHAINLOOP_TRACE_REDACTED:generic-password]
- Secret (generic-password) detected in session content [turn=601, source=tool_result, line=3]: 65- api_key: [CHAINLOOP_TRACE_REDACTED:generic-password]
- … and 1 more — view all ↗
✅ Passed ai-config-mcp-servers-allowedai-coding-session-0ebed9- -
🟡 83% — 100% AI — ⚠️ 1 policies failing
-
Oct 13, 2026 12:02 UTC · 74h33m37s · $131.50 · 2.9k in / 734.5k out · claude-code 2.1.296 (claude-opus-5-5)
Change Summary
-
- Adds a signed sandbox-identity design around PR-attestation commit verification.
- Reworks the spec, review replies, and PR flow across OSS and platform branches.
- Runs targeted runner, SCM, biz, and jobs tests while iterating on review fixes.
AI Session Overall Score
-
🟡 83% — Well-aligned work, but planning and end-to-end confirmation stayed thinner than ideal.
AI Session Analysis Breakdown
-
🟢 92% · alignment
-
🟢 The sandbox-identity rescope was reflected in the spec and both PR reworks. · High Impact
🟢 90% · scope-discipline
-
No notes.
🟢 88% · solution-quality
-
🟢 AI recovered the lost platform commit by cherry-picking the original change. · High Impact
🟢 86% · user-trust-signal
-
No notes.
🟡 74% · verification
-
🟢 AI ran focused Go tests for runners, SCM, jobs, and biz code. · High Impact
🟠 Targeted suites ran, but no explicit end-to-end confirmation of the identity flow appears. · Medium Severity
💡 After major workflow changes, ask for one concrete end-to-end confirmation before treating the PR as verified.
🟡 68% · context-and-planning
-
🟠 Complex multi-repo rework proceeded without a shared visible plan or TODO baseline. · Medium Severity
💡 For multi-repo work, write a short visible plan before editing so later re-scopes have a shared baseline.
-
File Attribution
████████████████████100% AI / 0% Human…and 4 more file(s).
Policies (5, 1 failing)
Status Policy Material Messages ⚠️ Failedai-session-no-auto-prai-coding-session-4503bd- Pull request opened without human approval (auto-approved: source=config, reason=classifier): ID=$(gh api repos/chainloop-dev/chainloop/pulls/3599/comments --jq '.[] | select(.user.login=="cubic-dev-ai[bot]" and (.body | test("VerificationStatusUnspecifi...
- Pull request opened without human approval (auto-approved: source=config, reason=classifier): ID=$(gh api repos/chainloop-dev/chainloop/pulls/3599/comments --jq '.[] | select(.user.login=="cubic-dev-ai[bot]" and (.body | test("os.LookupEnv"))) | .id' | h...
- Pull request opened without human approval (auto-approved: source=config, reason=classifier): cat > /tmp/claude-1000/-home-javi-orca-workspaces-platform-pfm-7678-record-commit-signature-verification-in/4503bdcc-d428-4e05-ade6-966360bf5543/scratchpad/repl...
- Pull request opened without human approval (auto-approved: source=config, reason=classifier): cd /tmp/claude-1000/-home-javi-orca-workspaces-platform-pfm-7678-record-commit-signature-verification-in/4503bdcc-d428-4e05-ade6-966360bf5543/scratchpad && gh a...
- Pull request opened without human approval (auto-approved: source=config, reason=classifier): cd ~/.cache/pfm-7678/spec-wt && gh pr create --repo chainloop-dev/platform --draft --base main --head PFM-7678-spec-sandbox-identity --title "spec: PFM-7678 Ver...
- Pull request opened without human approval (auto-approved: source=config, reason=classifier): cd ~/.cache/pfm-7678/spec-wt && git add docs/specs/PFM-7678-sandbox-identity.md && git commit -s -q -m "spec: say that the PFM-7678 identity is valid during the...
- Pull request opened without human approval (auto-approved: source=config, reason=classifier): cd ~/orca/workspaces/chainloop/pfm-7678-sandbox-commit-verification && gh pr create --repo chainloop-dev/chainloop --draft --base main --head pfm-7678-sandbox-c...
- Pull request opened without human approval (auto-approved: source=config, reason=classifier): gh api --method POST repos/chainloop-dev/platform/pulls/6875/comments/4266760571/replies -f body='Fixed in a645d9192. The control plane creates the workflow run...
- Pull request opened without human approval (auto-approved: source=config, reason=classifier): gh pr create --repo chainloop-dev/platform --draft --base main --head javier/pfm-7678-record-commit-signature-verification-in-the-built-in-pr --title "feat(pr-v...
- Pull request opened without human approval (auto-approved: source=config, reason=classifier): s=/tmp/claude-1000/-home-javi-orca-workspaces-platform-pfm-7678-record-commit-signature-verification-in/4503bdcc-d428-4e05-ade6-966360bf5543/scratchpad; cat > $...
✅ Passed ai-config-ai-agents-allowedai-coding-session-4503bd- ✅ Passed ai-config-no-dangerous-commandsai-coding-session-4503bd- ✅ Passed ai-config-no-secretsai-coding-session-4503bd- ✅ Passed ai-config-mcp-servers-allowedai-coding-session-4503bd- -
Powered by Chainloop and Chainloop Trace
There was a problem hiding this comment.
All reported issues were addressed across 4 files
Reply with feedback, questions, or to request a fix.
View guided diff | Re-trigger cubic
migmartri
left a comment
There was a problem hiding this comment.
I am confused about this PR, do you have a task for it?
|
@migmartri Yes, it is PFM-7678: https://linear.app/chainloop/issue/PFM-7678 The built-in PR attestation runs the CLI in the platform sandbox with the The approved spec (chainloop-dev/platform#6821) makes the platform verify the commit with the clone credential before it starts the sandbox. Only the result goes into the sandbox, not the token. This PR is the CLI half: the sandbox runner reads that result and records it only for the attested commit. The platform half is chainloop-dev/platform#6861, which waits for a release with this change. I added these links to the description. |
The CHAINLOOP_SANDBOX runner gets no SCM token, so it cannot ask the provider whether the attested commit is signed. The platform can verify the commit before it starts the sandbox. The runner now reads that result from CHAINLOOP_SANDBOX_COMMIT_VERIFICATION and returns it only when its commit hash is the attested commit. A malformed value records the unavailable status. CommitVerification gets JSON tags, and its status encodes as a name, so the platform and the CLI share one format. Assisted-by: Claude Code Signed-off-by: Javier Rodriguez <javier@chainloop.dev> Chainloop-Trace-Sessions: 0ebed99f-c817-4242-9a55-3c64202ac7f3
…tatus A given verification with no status, or with the unspecified status, carries no result. The runner now records the unavailable status for it, as it does for a value without a verification. Before, the attestation recorded the unspecified status. Assisted-by: Claude Code Signed-off-by: Javier Rodriguez <javier@chainloop.dev> Chainloop-Trace-Sessions: 4503bdcc-d428-4e05-ade6-966360bf5543
The platform gives a signed identity token to each Chainloop sandbox that it spawns. The new sandboxidentity package defines the token and how to sign and verify it. The token is a JWT signed with a new key for each sandbox. The key has a short-lived certificate from the control plane CA, and the certificate chain is in the x5c header. The CA marks the certificate only for a caller with the dedicated permission (R-002). The token names the organization, the project, the workflow, the sandbox run and the Chainloop token of the sandbox, and it holds the commit verification with the full commit hash (R-003). It has no expiry of its own (D-013). Verify checks the chain against the trusted root of the control plane, the mark, the organization, the project and the workflow, that the time of signing is inside the certificate lifetime (D-007), and the signature. The Issuer helper lets the platform issue a token in one call. The signing request gets the sandbox_identity_issuer flag, and the attestation init request gets the sandbox_identity token. The verifier exports ValidateCertificate, so the identity and the attestation use the same certificate check. Assisted-by: Claude Code Signed-off-by: Javier Rodriguez <javier@chainloop.dev> Chainloop-Trace-Sessions: 4503bdcc-d428-4e05-ade6-966360bf5543
The control plane now issues the certificates that sign a sandbox identity, and verifies and enforces the identity. - A new policy, PolicySandboxIdentityIssue, is the dedicated permission. No role holds it, and only a system API token can hold it (R-002, D-004). The signing service marks the certificate only for a caller that holds it. The CA writes the mark in the organizational unit of the subject (D-005). A CA that does not write the subject itself gives no marked certificate, and a mark that was not requested gives no certificate. - At attestation init, the control plane verifies the identity of a CHAINLOOP_SANDBOX run against its own CA, for the attested workflow, and only from the Chainloop token that the identity names (R-003, D-012). It records the identity on the workflow run, and tells the CLI the result. A token that does not verify is the same as no token (D-011). - When it stores an attestation of the sandbox runner, the control plane rejects an authenticated runner without a verified identity, and a commit verification that the identity does not hold (R-006, D-010). Other runners keep their behavior. Assisted-by: Claude Code Signed-off-by: Javier Rodriguez <javier@chainloop.dev> Chainloop-Trace-Sessions: 4503bdcc-d428-4e05-ade6-966360bf5543
…tity The CHAINLOOP_SANDBOX runner reads the identity token from CHAINLOOP_SANDBOX_IDENTITY. At attestation init, the CLI sends the token to the control plane. When the control plane verified it, the CLI also verifies it against the trusted root of the control plane, in the same way as attestation verify. Only then is the runner authenticated (R-004), and only then does it return the commit verification of the identity, for the full hash of the attested commit (R-005). A token that does not verify, or a control plane that does not know the identity, gives a runner that is not authenticated and has no platform data. The CLI logs the reason (D-011). This replaces the unsigned hand-off in CHAINLOOP_SANDBOX_COMMIT_VERIFICATION and its "unavailable" status for an unreadable value. No released CLI reads that value. Assisted-by: Claude Code Signed-off-by: Javier Rodriguez <javier@chainloop.dev> Chainloop-Trace-Sessions: 4503bdcc-d428-4e05-ade6-966360bf5543
2ee9aae to
f907159
Compare
Init loads the state that it wrote, and the load makes a new runner. A new CHAINLOOP_SANDBOX runner has no verified identity, so the runner info that init resolves next recorded authenticated=false, while git.head had the commit verification of the identity (R-004). Init now keeps its own runner when the runner can carry a verified identity. The other runners do not change. Assisted-by: Claude Code Signed-off-by: Javier Rodriguez <javier@chainloop.dev> Chainloop-Trace-Sessions: 4503bdcc-d428-4e05-ade6-966360bf5543
…t verification The git.head subject also records the signature algorithm, and the renderer takes it from the commit verification first. When the identity holds a signature algorithm, the control plane now also requires the attestation to record the same one (R-006). When the identity has no algorithm, the attestation takes it from the signature of the commit, so the control plane does not compare it. Assisted-by: Claude Code Signed-off-by: Javier Rodriguez <javier@chainloop.dev> Chainloop-Trace-Sessions: 4503bdcc-d428-4e05-ade6-966360bf5543
Context: PFM-7678. Design: Spec PFM-7678: Verified identity for Chainloop sandboxes, merged in chainloop-dev/platform#6875. It replaces D-009 of the commit verification spec (chainloop-dev/platform#6821). The platform side is chainloop-dev/platform#6861, which waits for a release with this change.
Today any CLI run can claim to be a Chainloop sandbox. So the platform cannot give a sandbox data that the attestation can trust. With this change, the platform gives each sandbox a signed identity token, and the CLI and the control plane verify it.
sandboxidentitypackage defines it. It is a JWT signed with a new key for each sandbox. Thex5cheader holds a short-lived certificate from the control plane CA. The token names the organization, the project, the workflow, the sandbox run and the Chainloop token of the sandbox. It holds the commit verification of the attested commit. It has no expiry of its own: the control plane accepts it only with the Chainloop token that it names, and that token expires.sandbox_identity:create. A caller with this permission gets a certificate with a mark in its subject. No role holds the permission, and only a system API token can hold it. A CA that cannot write the mark gives no certificate.CHAINLOOP_SANDBOXrunner reads the token from theCHAINLOOP_SANDBOX_IDENTITYenvironment value. The CLI verifies the token against the trusted root of the control plane, in the same way as theattestation verifycommand. Only then is the runner authenticated, and only then doesgit.headrecord the commit verification. A token that does not verify gives a runner that is not authenticated and has no platform data. The CLI logs the reason.The attestation format does not change, and the other runners do not change.
Notes on the implementation:
SandboxIdentityResult, which the spec does not name. The CLI authenticates the runner only when the control plane verified the identity and the verification of the CLI also passes. So the CLI and the control plane cannot disagree about the identity.signature.algorithmofgit.head.This replaces the unsigned
CHAINLOOP_SANDBOX_COMMIT_VERIFICATIONvalue of the first version of this PR. No released CLI reads that value.CommitVerificationkeeps its JSON tags and status names, because the token carries the commit verification in that format.AI disclosure: AI assisted with this change.
Assisted-by: Claude Code