Replace the differential ledgers with a committed snapshot (#664)

docs/design/decisions.md

@@ -1417,6 +1417,28 @@ Declined:
 - A word-boundary prefix (2026-09-05, #508's option 3) — answers the two Bob Jones rows and not `'Andrews, M.D.'`, and adds a tokenising rule to a string comparison.
 - Refusing two rules in one ledger from sharing a prefix (2026-09-05) — a property of the ledger, not of a pin; the ledger's rule families are deliberate (`fix(comma-family)`, `fix(#296)`), and the resolver already makes the only case that matters loud.
 
+### differential-ledger, retired for a snapshot (2026-10-13, #664)
+
+The ledgers' last arc. They were built to show that 2.0 changed nothing against 1.4 except fixes the release log lists, and they did; as a standing check every later behavior change had to be classified once per ledger, and the guards policing the ledgers outgrew the ledgers.
+
+Decisions that landed:
+
+- 2026-10-13 #664 — **a committed snapshot replaces the classified gate.** `tools/differential/snapshot.jsonl` records every corpus name's reading (the seven role fields, ambiguity kinds, `initials()`, and the facade where it differs), one line per (name, order); `tests/v2/test_snapshot.py` fails on drift and prints the moves grouped by changed field. A change that moves readings re-records, and the PR diff of the snapshot is the list of names it moved. Nothing is classified: what a reviewer reads in the diff is what a ledger rule used to say in a regex. The six `expected_since_*.toml` ledgers and `tests/v2/test_ledger_guards.py` are deleted, and with them the contract/radar tiers, so the CJK corpus is one file again and the rules corpus harvests tolerated rules' examples too.
+- 2026-10-13 #664 — **`compare.py` stays, unclassified.** It keeps the released-wheel runner and its checks against comparing one tree with itself (the version tell, the tree-side path check, the stripped `PYTHONPATH`; mechanisms.md#VERSION-TELL, #GENERATED-SCRIPT-OUTSIDE-THE-WORKTREE), prints the same grouped report as the snapshot and exits 0. It answers "how did release X read these names" for a release tagged before the snapshot existed; after that, `snapshot.py --since <tag>` is the cheaper answer.
+- 2026-10-13 #664 — **the CLDR agreement ratchet stays** (#654). It answers a different question, agreement with CLDR's stated fields rather than stability.
+
+Measured 2026-10-12, before the change, over the 40 first-parent commits ending at 04cc0814: 5,236 of 25,226 changed lines (21%) were ledger, `compare.py` or harness-test edits, and 21 of the 40 commits touched them. The share is uneven rather than steady: across the 29 of those commits that touch `nameparser/` it runs from 0% to 82%, median 11%, above 30% on 11 of them (#655: 236 of 448). Run against the code before and after #655, the snapshot reports five moved names, against that PR's 236 lines of ledger and guard edits; merging master afterwards, it reported #663's one new corpus name where #663 had added the same ten lines to each of six ledgers. RECOMPUTE (shares): `git log --first-parent -40 --format=%h 04cc0814`, and for each commit sum `git show --numstat` over every path except `*.jsonl` and `cldr_agreement.json`, the harness share over paths matching `expected_since|compare\.py|ledger_guards|test_differential|_differential_fixtures`; a commit "touches `nameparser/`" when any path starts with it.
+
+Accepted costs:
+
+- No forced explanation per moved name. A reviewer can approve a large snapshot diff without reading it; the grouping by changed fields is the mitigation.
+- No bound on what a change may reach. `_CORPUS_CLAIMS`, the dormancy checks and the contest pins existed only to police ledgers; mechanisms.md#LEDGER-RULE-SEPARATION and #CROSS-RULE-OUTCOME-PINS are retired with them.
+- The 1.4 compatibility promise is a question asked when needed (`compare.py --baseline 1.4.0`), not a gate at every release. The facade is deleted in 3.0.
+
+Declined:
+
+- Trimming the ledgers rather than retiring them (2026-10-13). The radar tier (#468, 2026-09-01) was that attempt: it stopped new corpora from failing the run, but every contract name still needed a rule in every ledger, and the cost grew by one ledger per release.
+
 ### comma-suffix-arc — #291/#296/#316 (2026-07-26 → 2026-08-01)
 
 #291 was filed 2026-07-26 out of the 2.0 vocabulary cleanup, with

docs/design/rules.md

@@ -15,7 +15,7 @@ An `annotation` names a policy, locale (`[ru]`), or extras gate (`[ja+segmenter]
 
 One class of rule stands outside the normative reading and declares it on a `tolerated: <reason>` line beside its pointer line. Such a rule is ABOUT an input the writing systems it speaks about do not produce — a shape no convention writes, read best-effort because parsing is total over strings — and it describes what the parser does with that input instead of promising it. Its statement is still precedence-bearing and its examples are still executable and still run, but they illustrate rather than promise, and the behavior is changeable without notice. W3 is the only such rule today (the 2026-09-01 comma demotion).
 
-The marker's unit is the whole rule, because that is the unit `tools/differential/build_rules_corpus.py` skips: it harvests no example from a marked rule, so none of them is enforced against released baselines. Two obligations follow, and neither is automatic. A clause that reaches past the tolerated shape does not belong in a marked rule — say it in a normative one. And the skip only STOPS enforcing a name; keeping it watched is a separate act, discharged here by giving each demoted example a `tolerated` row in `tests/v2/cases.py`, which projects it onto the differential's radar tier.
+The marker's unit is the whole rule, so a clause that reaches past the tolerated shape does not belong in a marked rule — say it in a normative one. The marker changes what the rule promises, not what is watched: a marked rule's examples are in `tools/differential/corpus_rules.jsonl` like any other, so the snapshot shows when their reading moves (#664).
 
 ## Not in scope
 
@@ -1313,8 +1313,7 @@ S2. Rationale: generational suffixes and credentials are recognized
     after the name. That reading is older than the company clause,
     so it is witnessed by tests/v2/cases.py's
     a_split_degree_in_front_is_out_of_the_walk rather than by a line
-    here, which would bring the name into the corpus that enforces
-    it at released baselines for a reading this clause did not make.
+    here, which would promise a reading this clause did not make.
     A title standing between the credential and the member, and a
     member a particle chain (P2) had taken, were two more such shapes
     until the credential's run (#602) took both in: the rows
@@ -1396,13 +1395,11 @@ S3. Rationale: credentials are often written run together with
     writes no abbreviations with a period that only a Latin
     convention writes — `John Smith 田.中.` is the shape — and a
     composed form no writing system produces is tolerated input
-    rather than contract. A normative rule cannot hold an example of
-    it without putting the string into the corpus that enforces it
-    at released baselines, so the witnesses are the tolerated row
+    rather than contract, which an example of a normative rule would
+    promise. So the witnesses are the tolerated row
     tests/v2/cases.py's
-    an_initialless_script_glued_into_periods_is_not_this_shape,
-    which keeps the name on the differential's radar tier, and the
-    CJK assertions in tests/v2/pipeline/test_vocab.py's
+    an_initialless_script_glued_into_periods_is_not_this_shape and
+    the CJK assertions in tests/v2/pipeline/test_vocab.py's
     test_period_joined_vocab_retires_the_single_character_chunk.
     W3 states the same precedence for the same reason.
     history: decisions.md#S2 · interacts: S2, C1, H2, W3 · implemented: nameparser/_pipeline/_vocab.py
@@ -2063,8 +2060,8 @@ C1. Rationale: a credential run after the comma means the name is in
     Accepted: the further-comma qualifier carries no example line of
     its own. It discriminates PAIRS and spans both branches, so
     exemplifying it means a with-comma partner for each — every one
-    of them a name entering the rules corpus for behavior that has
-    not moved since v1. Its executable witness is instead the pair
+    of them a promise about behavior that has not moved since v1.
+    Its executable witness is instead the pair
     already standing in the v1-style bank,
     tests/test_suffixes.py's
     test_roman_numeral_i_after_single_initial_lastname_comma_format
@@ -2137,8 +2134,8 @@ C2. Rationale: text beyond the recognized comma parts should be
       "John Smith, MD, XYZ"  unlisted_caps_suffixes-everywhere  →  ambiguities=("comma-structure",)
     Accepted: the no-name-reading clause's REPORT half carries no
     example line of its own. What it moves is a report with no field
-    beside it, and an example line would enter the rules corpus for
-    that report alone; its title half is witnessed by the two `John
+    beside it, and an example line would promise that report alone;
+    its title half is witnessed by the two `John
     Smith, Jr., van der Berg Prof.` lines above, and its report half's
     executable witnesses are the case rows
     a_credential_run_after_the_comma_reports_no_chain_fork and
@@ -2405,7 +2402,7 @@ W3. Rationale: a family name declared by a comma is the writer's
       "田中さん 様."              →  suffix="さん 様."
       "김민준씨."                 →  suffix="씨."
       "田中. 太郎"                →  family="田中."
-    tolerated: native CJK writing has neither a family-comma convention nor an edge full stop of any width on a name word, so the five comma lines above and the three period lines under them illustrate current behavior — changeable without notice — rather than promise it; the line carrying neither, beside them, is W1's claim, which is normative. All eight stay watched at every released baseline on the differential's radar tier (tools/differential/corpus_cjk_tolerated.jsonl, projected from the `tolerated` rows of tests/v2/cases.py) instead of its contract tier, and those rows pin them at HEAD.
+    tolerated: native CJK writing has neither a family-comma convention nor an edge full stop of any width on a name word, so the five comma lines above and the three period lines under them illustrate current behavior — changeable without notice — rather than promise it; the line carrying neither, beside them, is W1's claim, which is normative. All eight stay watched by the snapshot (tools/differential/snapshot.jsonl, over corpus_rules.jsonl and corpus_cjk.jsonl), and the `tolerated` rows of tests/v2/cases.py pin them at HEAD.
     history: decisions.md#W3 · interacts: W1, W2, C1, H2 · implemented: nameparser/_pipeline/_script_segment.py
 
 W4. Rationale: Chinese, Japanese and Korean all write the family

docs/migrate.rst

@@ -113,8 +113,9 @@ warned removals:
      - Gone; empty fields are always ``''``
 
 Finally, if you want to see the difference on your own data rather than
-ours, ``tools/differential/`` in the source repository diffs 1.4 and
-2.0 parses over a corpus of names you supply. It is development
+ours, ``tools/differential/compare.py`` in the source repository diffs
+any released version's parses against the source tree over the corpora
+beside it, or over a file of your own names (``--corpus``). It is development
 tooling, not part of the installed package — see its README.
 
 Attribute map

nameparser/_facade.py

@@ -709,8 +709,8 @@ def all_particle_guard(got: list[str], words: list[str]) -> list[str]:
             # which compare the seven role fields only (Case carries
             # no initials column); the covering test is
             # tests/test_initials.py::test_initials_middle_name_all_prefixes,
-            # and since #484 the differential compares initials() on
-            # both surfaces for names whose roles agree. _split_last
+            # and tools/differential/snapshot.jsonl records initials()
+            # on both surfaces for every corpus name. _split_last
             # already applies the same guard to the base, which is why
             # last_base was never empty here.
             return [_render._first_letter(w) for w in words]

nameparser/_pipeline/_script_segment.py

@@ -255,8 +255,8 @@ def _is_post_nominal(state: ParseState, i: int) -> bool:
 # classification rather than cut into 박사 + 님, which is what the
 # same input gives when the step is removed. '선생님' is post-nominal
 # entire with no name word behind it, so the scan yields no site at
-# all and the token stays whole. Both are contract-tier corpus names
-# and rules.md#W2 example lines; decisions.md#cjk-comma-demotion
+# all and the token stays whole. Both are corpus names and
+# rules.md#W2 example lines; decisions.md#cjk-comma-demotion
 # carries the forced-predicate measurement behind them.
 def _peel_site(state: ParseState, flat: Sequence[int],
                tails: frozenset[str]) -> tuple[int, int] | None:

nameparser/config/_invariants.py

@@ -30,9 +30,8 @@
 hygiene depend on the parser, and the question worth asking is not
 "is this entry pre-folded" but "does ``Lexicon`` store it unchanged".
 That is one assertion over ``_PHRASE_FIELDS`` in
-``tests/v2/test_ledger_guards.py``, the suite that already imports the
-parser's fold for exactly this purpose, and it is where an entry
-written ``'z. domu'`` is caught.
+``tests/v2/test_lexicon.py``, and it is where an entry written
+``'z. domu'`` is caught.
 """
 from __future__ import annotations
 

tests/test_initials.py

@@ -292,22 +292,9 @@ def test_initials_follow_the_one_case_fork_on_both_letters(self) -> None:
         # 'y' outside the marked set is still the connective.
         hn = HumanName("maria y lopez")
         self.m(hn.initials(), "m. y. l.", hn)
-        # These two are among #528's four movers, and the #528 ledger
-        # rule's own name_regex (a literal alternation, in
-        # tools/differential/expected_since_1.4.0.toml) covers all four
-        # names by itself -- but only "john e jones" also has a row in
-        # _CROSS_RULE_WINNERS (tests/v2/test_ledger_guards.py) and
-        # _RECORDED_DIFFS (tools/differential/compare.py): those two
-        # Python dicts, keyed by the LEDGER FILENAME rather than
-        # anything stored in the TOML, record that
-        # "fix(initials-per-word) a connective run initials each word"
-        # also reaches "john e jones" through its lowercase ' e ' and
-        # contests the name with #528's rule. "jones, john e" has no
-        # row in either: that connective-run rule's operative half is a
-        # `(?=\s)` lookahead requiring whitespace AFTER the letter, and
-        # a letter at the end of the string can never satisfy it -- so
-        # the comma form is a distinct shape from the space-written
-        # form even though both give the same initials() answer.
+        # These two are among #528's four movers, written once with
+        # the letter mid-name and once at the end after a comma: both
+        # give the same initials() answer by different routes.
         hn = HumanName("john e jones")
         self.m(hn.initials(), "j. e. j.", hn)
         hn = HumanName("jones, john e")