Self-test
PDS Stack’s gates are reasoning-based. A prompt edit that quietly hollows one out compiles fine, breaks no test, and produces no error — the only way to notice is to run a whole feature cycle again and compare.
So that is what _stack-test-pulse/ is: a throwaway product driven end to end through every
active gate, in an isolated sandbox, with every agent signal transcribed verbatim and scored.
Six runs so far, plus a targeted validation cycle for 4.1.x. The transcripts, the scores and the findings are all in the repo, including
the runs where the stack failed.
The runs
| Run | Date | Target | Result |
|---|---|---|---|
| 1 | 2026-08-20 | Does the machinery work at all? | ~89/100 · 14/14 mechanisms fired |
| 2 | 2026-08-20 | The REJECT → correction → re-eval loop | ~91/100 · rejected at 11, shipped at 19 |
| 3 | 2026-08-28 | The conductor, the guardrails, the proof rule | ~86/100 · 5 findings |
| 4 | 2026-08-28 | Confirming run 3’s fixes on a fresh install | ~92/100 · all confirmed |
| 5 | 2026-08-28 | The junior interview, driven end to end | ~89/100 · 6 findings |
| 6 | 2026-09-27 | V5: one phase per session, measured in tokens | ~87/100 · 8 findings · 51.5M read · $25 |
| 4.1.x | 2026-09-27 | The 4.1 fixes, on a fresh install, Sketch lane | 2.1M read · 0 ADR-014 targets exceeded |
Why the score went down
Run 3 targeted everything that did not exist when runs 1 and 2 scored ~91: /pds, the aesthetic
gate as a shipped file, the git guardrails, and the proof rule.
Three axes went up. The spec gate refused a commit against an unvalidated spec on its first real use, closing a weakness that had stood since run 1. ANALYZER caught a genuine defect the builder had self-passed — and proved it by running code rather than by re-reading the spec:
1 commit in the repo, filter "fix" → 0 groups → the UI renders "No commits yet"The total fell anyway, because two rules had shipped without ever being executed once.
The conductor did not cross its own gate. /pds STEP 1 looked for [À COMPLÉTER]
placeholders; the shipped English templates contained none in roadmap.md or
design_guide.md. Two of three empty context files passed as complete — the exact failure the
conductor exists to prevent.
The proof rule was not runnable. It said to run assertions “with whatever the project can
already execute”. In a moduleResolution: bundler project there is no such thing: extensionless
imports break Node’s loader, .ts extensions need a compiler flag that would leak into product
source, and node:assert needs a dependency the rule forbids. Three attempts failed before one
worked.
Run 4 confirmed both fixes on a fresh install: the bootstrap now blocks on all three context files, and the documented proof command runs verbatim with nothing installed. A sixth finding surfaced while fixing them — the verdict band publicly named “SHIPPED WITH NOTES” describes a feature that is explicitly not committed. The thresholds are untouched; every surface now states the commit rule next to the band, so the name cannot be read as an outcome it does not produce.
Both original findings were fixed, and a fourth check was added to the release pipeline: a detection rule must match the files it targets. Existence checks could never have caught the first one — both files shipped, both were packaged, they simply did not agree with each other.
The finding that needed a real build
Run 5 drove a full cycle in junior mode, from three empty context files to a verdict. The interview works — five beginner answers produced a usable product vision and roadmap. Two things around it did not, and one of them is a different kind of problem from everything found before.
Two approved gates produced a violation of the frozen contract. An acceptance criterion, validated and therefore frozen, forbade any colour literal outside a token map. The Quality Brief, approved four steps later, authorised an accent on one count. BOB obeyed the brief and breached the spec — with both gates behaving exactly as specified. Nothing in the flow compared them.
Every earlier finding was a rule pointing at nothing, or a guard with the wrong scope — the kind of thing careful reading can catch. This one is structural: the brief sits downstream of the freeze and was allowed to contradict it. BOB now checks the brief against the spec’s visual criteria and refuses to emit one that conflicts, routing the clash back to RAY, because changing a frozen criterion is a new cycle by the stack’s own rule.
It took building a feature to surface. No amount of re-reading the prompts would have produced it.
Run 6 — the first measured V5 cycle
Two Standard features, each phase a separate headless session, every session measured with
npm run tokens: 12 sessions, 51.5M tokens read, $25.
The half of V5 that is conversation discipline worked. The main conversation fell from ~75 % of the spend (up to 669k of context, before V5) to 10 %, never above 62k. Direction came before scope twice and nothing downstream contradicted it — run 5’s two-gates-disagreeing finding did not recur.
The half that was a mechanical claim did not. bob-build was 78 % of the spend.
The image guard never ran. It lived in bob-build’s frontmatter, and a frontmatter hook does
not fire when the agent runs as a subagent — reproduced in isolation with a hook that only
touches a file. A full-size screenshot was read back. The same hook in .claude/settings.json
blocked the Read inside the same subagent.
The install did not ship what its own CLAUDE.md commands. No memory/, no scripts/, no
memory:index — while the generated CLAUDE.md requires memory/identity.md first and
npm run memory:index after every delivery. check-parity was green because its reference pass
did not cover those paths.
The other findings: bob-build hit its 80-turn cap four times in fifteen runs; BOB wrote
components/ui/ by hand; one phase ran in the same conversation as the previous one; gate output
ran longer than output: short; a correction round undid two waived rules. Full list in
_stack-test-pulse/SCORING.md.
4.1.x — the fixes, checked one by one
4.1 fixed the blocker and the three majors, each with a targeted test rather than a full re-run:
the image guard moved to .claude/settings.json scoped to bob-build; the installer ships
memory/, scripts/ and their npm commands; RAY cuts specs into lots of about 50 turns with a
fresh builder per lot; a hand edit in components/ui/ is a blocker. 4.1.1 made check-parity
execute the image guard on three cases instead of searching for its text.
Then one Sketch cycle on a fresh install, measured:
| 4.1.x validation | ADR-014 target | |
|---|---|---|
| Tokens read, whole cycle | 2.1M | — |
| Main conversation, worst context | 98k | < 150k ✓ |
bob-build | 19 turns · 43k of context | < 50 turns per lot ✓ |
| Targets exceeded | 0 | — |
One Sketch cycle is not run 6’s two Standard features — the totals do not compare. What does: the builder stayed small, and nothing crossed a ceiling.
What the runs are good for
The axis scores are self-assessed. They are useful as a delta between runs, not as an absolute grade: “defect-catching power went 17 → 19 → 20” is a real signal; “the stack is 89/100” is a number about a small sample. A drop on any axis is the finding. The total is a headline.
When it goes stale
baseline.json records the hashes of the gate-defining files as of the last run. The release
check compares them and says so when they have moved:
⚠ 3 gate file(s) changed since the last pulse run — benchmark is staleIt warns and never blocks — no CI job can drive an agent through a feature cycle for you. The warning only tells you the benchmark no longer describes the current stack.
Running it yourself
The full procedure is in _stack-test-pulse/README.md. In short: a sandbox outside the repo, one
T2 feature driven end to end with no help past any gate, then a second feature built with a
deliberate corner cut so ANALYZER has something to reject. Transcribe the real signals, score the
axes, and write down what got worse as plainly as what got better.
A run that only confirms the stack is fine has told you nothing. Runs 3 and 6 are in the repo precisely because each found shipped rules that had never been executed.