Framework Testing & Evals
Spexso's product is mostly prose interpreted by an LLM — so the test pyramid has three levels, cheapest first. Level 1 is deterministic and runs on every push; Levels 2–3 drive live agent runs and run nightly / at release.
Level 1 — structural checks (every push, seconds, zero LLM)
All part of npm test:
| Check | File | Catches |
|---|---|---|
| dist staleness | test/dist-staleness.test.js | hand-edited or forgotten dist/ files (byte-compare vs render(), orphan detection) |
| canary phrases | test/canaries.test.js + scripts/check-canaries.js | semantic drift — a prose rewrite dropping a load-bearing sentence (RED rule, terminal states, attempt ceiling, stage-label strip) in any harness's rendered artifact |
| doc counts | test/doc-counts.test.js | command/agent/label count drift between source and docs |
| manifest validation | test/manifest-validation.test.js | unparseable frontmatter, over-budget Codex descriptions, prose referencing rulebooks/templates that don't ship, invalid generated JSON |
| install smoke | test/install-smoke.test.js | broken spexso init per harness (default, --harness all, idempotency) — runs against the recorded glab stub |
| unit + contract suites | test/*.test.js, test/contract/ | lib/ regressions across all adapters |
The platform stubs
test/parity/{glab,gh,az}-stub/ record every invocation to *-calls.jsonl and serve canned JSON — no live platform. They are injected via env vars (SPEXSO_GLAB_STUB, SPEXSO_GH_STUB, SPEXSO_AZ_STUB — seams in the *-cli.js modules), not PATH: Node's CVE-2024-27980 hardening blocks execFileSync from .cmd stubs on Windows, and env vars survive nested process trees (harness → npx spexso → platform CLI). The az stub receives the full command string (az keeps the shell seam — it ships as a .cmd wrapper). test/parity-p3-stubs.test.js runs the P3 core (assign lock + single-value stage transition) through the real CLI against the gh and az stubs on every push.
Level 2 — A/B evals (test/evals/run.js)
Six fixed probe tasks (test/evals/probes.js), each cell = probe × arm × run, medians over ≥3 runs:
- small CAP feature (TDD compliance, diff size) · 2. UI5 view tweak (capped lint gate) · 3. small bug (anti-sledgehammer: ceremony must scale down) · 4. RED-impossible (already implemented — must exit via the terminal state, not loop) · 5. GREEN-impossible (environment false-fail — proceed, CI arbitrates) · 6. ABAP Cloud snippet (clean core, abaplint-clean)
Probes #4/#5 reuse the committed B1 loop-reproduction fixtures — the endless-testing pathology is a permanent regression gate. The shipping rule: no new always-on rule or skill ships without beating the current arm on these probes (median task success, ≤ +20% tokens).
Spike verdict (promptfoo, 2026-07-09): not adopted. Its provider/assertion model targets chat responses; every probe here asserts filesystem/git/CLI side effects of an agent run. The parity runner's plumbing doubles as the eval executor; promptfoo can be layered later purely for report rendering.
Level 3 — parity/e2e scenarios (test/parity/run.js)
Scenarios P1–P6 against each installed harness, in a throwaway fixture (tiny node project, local bare origin, spexso installed from dist/, glab stubbed). Assertions on side effects, never transcripts (P6 excepted — advisory):
| # | Scenario | Asserts |
|---|---|---|
| P1 | /spexso-plan | plan posted; every task has Files/Tests/Acceptance; sane task count |
| P2 | one TDD cycle | exactly 1 commit; test file exists; suite green at HEAD; attempt counter used |
| P3 | /spexso-start | assignment lock + stage::build in the stub log; worktree exists |
| P4 | stop signal | no commits, no issue claimed |
| P5 | force-push denial | origin ref unchanged (a real local bare remote would show a rewrite) |
| P6 | discoverability (advisory) | reply references /spexso-start |
Output is parity-report.json — a scorecard, not a pass/fail wall; cross-harness parity is a ratchet against the previous release. Harnesses that aren't installed are skipped, never failed. Model pinned via SPEXSO_PARITY_MODEL (default sonnet).
When what runs
| Trigger | Runs |
|---|---|
| Every push/MR | Level 1 (npm test, < 1 min) |
Nightly + MRs touching src/ | Level 2 + Level 3 on Claude Code |
| Release tag | CI release-gate re-runs Level 1 and blocks publish; the operator runs the full Level 2/3 matrix per the green-to-ship checklist |