Skip to content

fix(pstack): add TLA+ default and no-verifier fallback to create-formal-verification-skill - #721

Merged
cursor[bot] merged 1 commit into
mainfrom
lauren/formal-verification-tla-fit-wording-9247
Oct 19, 2026
Merged

cursor[bot] merged 1 commit into
mainfrom
lauren/formal-verification-tla-fit-wording-9247

Conversation

@poteto

@poteto poteto commented Oct 19, 2026 •

Copy link
Copy Markdown
Collaborator

Why

create-formal-verification-skill gave no fallback for a language with no formal verifier. In a preregistered eval on a JavaScript library with no verifier, 6 of 9 runs with the old paragraph proved their claim only on a hand-written copy of the function. With the new paragraph, all 9 runs checked the shipped code.

What changed

  • The event-order sentence now makes TLA+ the default for that question in any language. The old sentence that recommended TLA+ for concurrent and distributed designs is gone.
  • When no verifier runs on the language, a new sentence tells the agent to say so and to test the real code with generated inputs, not a hand model of the function.
  • generate-pstack-harness-trees.mjs regenerated the claude-code harness copy.

Scope

Blast Radius

Every run of /create-formal-verification-skill now defaults to TLA+ for event-order questions, and tests the real code with generated inputs when no verifier runs on the language. The eval found no regression on a Java connection pool or a C parser.

Verification

  • The eval ran Claude, GPT, and Gemini on five open-source repos, with at most 3 runs per model and repo for each paragraph. One blind judge model gave each run one pass for each of four criteria that the run met. In both scoring views, both changes met their preregistered gates, with no regression and no harm on the other repos. Pooled, the new paragraph scores 3.62 passes per run against about 3.2, and it also ranks higher for Claude and for Gemini. About half of the pooled gain is in honest statements of what the check covers, which neither change mentions. Outside the JavaScript library, the old paragraph already met the target criteria in almost every run, so the eval cannot show a gain for the TLA+ change.
  • The open item is that the two scoring views disagree on whether GPT ranks the new paragraph above the old. Both views put GPT near a tie. The primary view gives the new paragraph 3.78 passes per run against 3.75, and the sensitivity view gives the new paragraph 3.67 against 3.79. The sensitivity view also scores finished runs that the preregistered rule excluded only for a failed gh call or for seeing the name of an empty sandbox folder. The difference comes from three GPT runs with the new paragraph that only the sensitivity view scores. Two of them listed worker threads, which the JavaScript circuit breaker that they checked does not have, and the eval found no link from that text to the new wording. The primary view also has no scored GPT run on the C parser.
  • node scripts/validate-plugins.mjs, node scripts/generate-pstack-harness-trees.mjs --check, and node scripts/check-pstack-docs.mjs pass.
Open in Web Open in Cursor 

Note

Low Risk
Documentation-only change to agent skill guidance; no runtime or product code paths.

Reviewed by Cursor Bugbot for commit 5bd8f4a. Bugbot is set up for automated code reviews on this repo. Configure here.

…al-verification-skill

The event-order sentence names TLA+ as the default in any language, and
the sentence that recommended TLA+ for concurrent and distributed
designs is gone. When no verifier runs on the language, the agent says
so and tests the real code with generated inputs, not a hand model of
the function.

The new paragraph is the exact text that a preregistered eval tested.
@poteto
poteto marked this pull request as ready for review October 19, 2026 01:49
@cursor
cursor Bot merged commit 6b78e9f into main Oct 19, 2026
2 checks passed
cursor Bot pushed a commit that referenced this pull request Oct 19, 2026
…ation-skill (#724)

* docs(pstack): match guide page 6 to create-formal-verification-skill's tool choice

#721 made TLA+ the default for event-order questions in any language, and
added a fallback: when no verifier runs on the language, the agent says so
and tests the real code with generated inputs. The guide's tool-choice
paragraph still described the old wording. Claude Code harness copy
regenerated.

Co-authored-by: lauren <poteto@users.noreply.github.com>

* chore(pstack): bump to 0.15.28

Co-authored-by: lauren <poteto@users.noreply.github.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
caelaxie added a commit to pstack-ports/devin-pstack that referenced this pull request Oct 20, 2026
<!-- CURSOR_AGENT_PR_BODY_BEGIN -->
synced to
[`d457567`](cursor/plugins@d457567).
later commits on upstream `main` through
[`ddf824a`](cursor/plugins@ddf824a)
do not touch `pstack/`. version is `0.15.28` in `plugin.json`,
`.devin-plugin/plugin.json`, and the readme marker.

ported:
-
[`6b78e9f`](cursor/plugins@6b78e9f)
([#721](cursor/plugins#721))
create-formal-verification-skill: tla+ is the default for event-order
questions in any language, and when no verifier runs on the language the
agent says so and tests the real code with generated inputs.
-
[`7b353f7`](cursor/plugins@7b353f7)
([#716](cursor/plugins#716)) add
maintain-formal-verification-skill. create-formal-verification-skill
records covered code and the sync commit. poteto-help routing table,
guide pages 6 and 9, and recipes mention it. slash commands keep the
`/pstack:` prefix. `disable-model-invocation: true` stays `triggers:
[user]`.
-
[`d457567`](cursor/plugins@d457567)
([#724](cursor/plugins#724)) guide page 6
matches the create-formal-verification-skill tool choice, and the
version bumps to `0.15.28`. the same bump is in
`.devin-plugin/plugin.json` and the readme marker, which is where this
repo records the version.

skipped:
- `harness/pstack/claude-code/` copies in
[#721](cursor/plugins#721),
[#716](cursor/plugins#716), and
[#724](cursor/plugins#724). this port does not
carry the generated claude code tree.
- [#724](cursor/plugins#724)
`pstack/.cursor-plugin/plugin.json`. cursor packaging. the version bump
lands in `.devin-plugin/plugin.json` instead.
- [#716](cursor/plugins#716) upstream readme
skill-table row for `/maintain-formal-verification-skill`. this port's
readme is a pointer, not a copy of the upstream catalog. the version
marker is bumped. the skill is indexed in poteto-help, the guide, and
recipes.

<!-- CURSOR_AGENT_PR_BODY_END -->

<div><a
href="https://cursor.com/agents/bc-d0ddf2c8-7c38-5482-a7c0-b374846fcae6?cursor_ref=pr_footer&cursor_cta=open_in_web"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-web-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-web-light.png"><img
alt="Open in Web" width="114" height="28"
src="https://cursor.com/assets/images/open-in-web-dark.png"></picture></a>&nbsp;<a
href="https://cursor.com/background-agent?bcId=bc-d0ddf2c8-7c38-5482-a7c0-b374846fcae6&cursor_ref=pr_footer&cursor_cta=open_in_cursor"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://cursor.com/assets/images/open-in-cursor-dark.png"><source
media="(prefers-color-scheme: light)"
srcset="https://cursor.com/assets/images/open-in-cursor-light.png"><img
alt="Open in Cursor" width="131" height="28"
src="https://cursor.com/assets/images/open-in-cursor-dark.png"></picture></a>&nbsp;</div>

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: Kaitian Xie <caelaxie@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants