Skip to content

pstack: add maintain-formal-verification-skill - #716

Merged
cursor[bot] merged 7 commits into
mainfrom
lauren/maintain-formal-verification-skill-d5ef
Oct 19, 2026
Merged

cursor[bot] merged 7 commits into
mainfrom
lauren/maintain-formal-verification-skill-d5ef

Conversation

@poteto

@poteto poteto commented Oct 18, 2026 •

Copy link
Copy Markdown
Collaborator

What

Adds /maintain-formal-verification-skill, the companion to /create-formal-verification-skill. The create skill writes a project-local formal verify skill one time. The maintain skill keeps that verify skill in line with the code after changes that did not recheck it, such as merges, refactors, and edits made without the verify skill.

One pass:

  1. Finds the project's formal verify skills by what they do, not by name. The generated skill has no fixed name, and /setup-pstack (from pstack: offer formal verification in setup, sync help and guide #715) finds it in the same way.
  2. Runs each check as it is.
  3. Reads every change to the covered code since the recorded sync commit, including code that moved and new code that touches the guarded state.
  4. Updates the model, harness, or properties from the code, not from the result. A property changes only when evidence shows that the promise changed on purpose.
  5. Reruns each check. On a pass, it breaks the new code to show that the check can still fail. On a failure, it replays the counterexample against the code to decide whether the failure is drift in the verification or a bug in the code.
  6. Updates the verify skill's coverage record and moves its sync commit forward. It does not move the sync commit for a blocked check, and it says so in the report.

Each check ends as in sync, updated, bug, or blocked. The skill does not edit product code, and it does not weaken a property to make a check pass.

Change to create-formal-verification-skill

The generated skill now records which files and functions each check covers, and the commit where the verification last matched them. The maintain skill reads that record to know what to diff. The naming does not change.

Docs

  • README: adds a row for the new skill.
  • poteto-help: adds a routing-table row, and adds the skill to the formal verification state check and recipe.
  • Guide page 6: adds one paragraph at the end of the formal verification section.
  • Guide page 9: names the skill in the sentence on special cases, and points the "covers both kinds" link at the formal verification section on page 6 instead of the app-verification section.

node scripts/check-pstack-docs.mjs reports 0 problems, node scripts/validate-plugins.mjs passes, and node scripts/generate-pstack-harness-trees.mjs --check reports the harness tree up to date.

Eval

The eval was preregistered. It had 54 sealed offline runs: 3 cases × 3 arms × 3 models (Claude Opus 5.5, GPT-5.6 Sol, Grok 4.7, all at xhigh) × 2 replicates. A blind Gemini 4 Argon High judge scored each run.

  • Cases. Real upstream histories with a formal suite that drifted:
    • x/sync/semaphore with TLA+, where the stale model still passes.
    • The same history with a missing wakeup added.
    • cachetools with a Python model check, where the stale check fails on an intended change.
  • Arms. The verify skill without a coverage record (B0). The verify skill with the record that this PR adds to the create skill (B1). B1 plus this skill (M).

Results:

  • Correctness: no real gain. Correct runs of 18: B0 16, B1 14, M 16. M is 2 runs above B1, which meets the preregistered ship rule. But most failures were runs that found a real, older bug in the semaphore code that the case key did not include, or runs that added checker trace files to .gitignore. With those two corrections, all three arms score 18 of 18. On these cases, the models already keep the verification in sync without the skill.
  • The skill makes runs prove that the updated check can still fail. In runs where the final check passed, every M report showed a deliberate break and the check that caught it (11 of 11). B0 did this in 5 of 10 runs, and B1 in 4 of 10. M reports also separated drift from bugs in 18 of 18 runs.
  • The record makes runs update the sync commit. B1 and M updated the sync commit in 18 of 18 runs. B0 did so in 4 of 18.
  • No harm found. No M run changed product source, weakened the NoStrandedWaiter violation, or called the intended cachetools change a bug. One M run edited .gitignore, and one made its breaks in the product file and then restored the file.

Follow-up: blocked check. Bugbot found that step 6 moved the sync commit even for a blocked check, which the runs above never produced. The fix adds one sentence to step 6 and corrects a step number in step 2. A second preregistered test (one case where one check cannot run, old text against new text, the same three models, 3 runs each, 18 sealed runs, blind judge) gave: new text 9/9 runs kept the blocked check's sync commit and reported it as blocked, and old text 0/9. With the old text, 7 runs moved the shared sync commit to HEAD.

This PR does not bump the pstack version.

Open in Web Open in Cursor 

Note

Low Risk
Documentation and agent skill definitions only; no runtime product code, auth, or data paths change.

Reviewed by Cursor Bugbot for commit 011ef53. Bugbot is set up for automated code reviews on this repo. Configure here.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 4078be5. Configure here.

Comment thread pstack/skills/maintain-formal-verification-skill/SKILL.md Outdated
Comment thread pstack/skills/maintain-formal-verification-skill/SKILL.md Outdated
Comment thread pstack/skills/maintain-formal-verification-skill/SKILL.md
cursoragent and others added 5 commits October 18, 2026 22:00
create-formal-verification-skill now writes, into the generated verify skill,
which files and functions each check covers and the commit at which the
verification was last confirmed to describe them, so a later agent can see
when covered code has changed since.

Co-authored-by: lauren <poteto@users.noreply.github.com>
Upkeep loop for a formal verify skill: read the commits that touched covered
code since the recorded sync commit, update the model, harness, or properties
from the code, rerun, prove each updated check can still fail, and report each
check as in sync, updated, bug, or blocked. Listed in the README and in the
poteto-help routing table.

Co-authored-by: lauren <poteto@users.noreply.github.com>
create-formal-verification-skill gives the generated skill no fixed name, and
setup finds it by its description. The maintain skill now does the same.

Co-authored-by: lauren <poteto@users.noreply.github.com>
Adds the skill to the spots that already cover formal verification: the
verify-and-ship section, the special-cases sentence on guide page 9, and the
poteto-help state check and recipe.

Co-authored-by: lauren <poteto@users.noreply.github.com>
Co-authored-by: lauren <poteto@users.noreply.github.com>
@cursor
cursor Bot force-pushed the lauren/maintain-formal-verification-skill-d5ef branch from 4078be5 to c50858d Compare October 18, 2026 22:01
@cursor
cursor Bot changed the base branch from lauren/setup-pstack-formal-verification-docs-3e39 to main October 18, 2026 22:01
cursoragent and others added 2 commits October 19, 2026 03:38
maintain-formal-verification-skill set the sync commit to the checked commit
for every check, including a blocked one. A blocked check was not confirmed to
describe the code, so moving its sync commit hides the drift from the next
pass. Step 6 now leaves it in place and says so in the report. Step 2 now
points at step 5, where a failure is judged.

Co-authored-by: lauren <poteto@users.noreply.github.com>
…l-verification-skill-d5ef

Co-authored-by: lauren <poteto@users.noreply.github.com>
@poteto
poteto marked this pull request as ready for review October 19, 2026 05:06
@cursor
cursor Bot merged commit 7b353f7 into main Oct 19, 2026
2 checks passed
caelaxie added a commit to pstack-ports/devin-pstack that referenced this pull request Oct 20, 2026
<!-- CURSOR_AGENT_PR_BODY_BEGIN -->
synced to
[`d457567`](cursor/plugins@d457567).
later commits on upstream `main` through
[`ddf824a`](cursor/plugins@ddf824a)
do not touch `pstack/`. version is `0.15.28` in `plugin.json`,
`.devin-plugin/plugin.json`, and the readme marker.

ported:
-
[`6b78e9f`](cursor/plugins@6b78e9f)
([#721](cursor/plugins#721))
create-formal-verification-skill: tla+ is the default for event-order
questions in any language, and when no verifier runs on the language the
agent says so and tests the real code with generated inputs.
-
[`7b353f7`](cursor/plugins@7b353f7)
([#716](cursor/plugins#716)) add
maintain-formal-verification-skill. create-formal-verification-skill
records covered code and the sync commit. poteto-help routing table,
guide pages 6 and 9, and recipes mention it. slash commands keep the
`/pstack:` prefix. `disable-model-invocation: true` stays `triggers:
[user]`.
-
[`d457567`](cursor/plugins@d457567)
([#724](cursor/plugins#724)) guide page 6
matches the create-formal-verification-skill tool choice, and the
version bumps to `0.15.28`. the same bump is in
`.devin-plugin/plugin.json` and the readme marker, which is where this
repo records the version.

skipped:
- `harness/pstack/claude-code/` copies in
[#721](cursor/plugins#721),
[#716](cursor/plugins#716), and
[#724](cursor/plugins#724). this port does not
carry the generated claude code tree.
- [#724](cursor/plugins#724)
`pstack/.cursor-plugin/plugin.json`. cursor packaging. the version bump
lands in `.devin-plugin/plugin.json` instead.
- [#716](cursor/plugins#716) upstream readme
skill-table row for `/maintain-formal-verification-skill`. this port's
readme is a pointer, not a copy of the upstream catalog. the version
marker is bumped. the skill is indexed in poteto-help, the guide, and
recipes.

<!-- CURSOR_AGENT_PR_BODY_END -->

<div><a
href="https://cursor.com/agents/bc-d0ddf2c8-7c38-5482-a7c0-b374846fcae6?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-web-light.png"><img
alt="Open in Web" width="114" height="28"
src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a>&nbsp;<a
href="https://cursor.com/background-agent?bcId=bc-d0ddf2c8-7c38-5482-a7c0-b374846fcae6&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img
alt="Open in Cursor" width="131" height="28"
src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a>&nbsp;</div>

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Kaitian Xie <caelaxie@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants