Factory (Ralph loop on a Hetzner server)

The factory builds a whole milestone without a developer at the keyboard. One command on your PC makes sure a Hetzner server exists, puts the project on it and starts a loop that runs /spexso-marathon --milestone X --max 1 in a fresh claude -p context per issue — the "Ralph" pattern — and merges each issue through the merge train into targetBranch. The server deletes itself after 60 minutes without work.

Phase 1: humans write the specs (/spexso-kickstart, /spexso-brainstorm); the factory only builds stage::spec issues. Autonomous speccing is a later phase.

One-time setup

1. Hetzner: create a dedicated project (e.g. spexso-factory) with nothing else in it, an API token for it, and add your SSH key to it. The token also lands on the server (for self-deletion), so it must not be able to touch anything else.
2. GitLab bot: the same bot user as auto-pick — Developer on the product, allowed to merge into targetBranch, token scopes api, write_repository, read_api.
3. Claude: run claude setup-token on your PC; it prints a one-year token for your subscription. Or use an API key — see Licence.
4. Your PC: hcloud and ssh installed, and these environment variables set:

Variable	Value
HCLOUD_TOKEN	token of the factory Hetzner project
CLAUDE_CODE_OAUTH_TOKEN or ANTHROPIC_API_KEY	exactly one of the two
SPEXSO_BOT_TOKEN	GitLab bot token
SPEXSO_BOT_NAME, SPEXSO_BOT_EMAIL	commit identity of the bot
SPEXSO_FACTORY_WEBHOOK (optional)	webhook for live notifications — see Following a run

Optional ~/.spexso/factory.json: { "serverName", "serverType", "location", "image", "sshKey", "marketplaces", "plugins", "mcpServers", "testMaxWorkers" } (defaults spexso-factory, ["cx43", "cax31"], ["fsn1", "nbg1", "hel1"], ubuntu-24.04, the only key in the project; the tool lists below). Each key replaces its default.

Plugins and MCP servers. A headless server never installs what a repo's .claude/settings.json declares, so every start installs them at user scope (claude plugin install --scope user, claude mcp add-json --scope user). Defaults: superpowers, ui5, cds-mcp, and the sap-skills sap-cap-capire, sapui5, sapui5-linter, sapui5-cli, sap-fiori-tools, sap-btp-developer-guide, sap-btp-best-practices, sap-btp-connectivity, and ponytail (from DietrichGebert/ponytail: its hooks put every session and subagent in "lazy senior dev" mode, the shortest diff that works); MCP servers sap-docs (HTTP) and fiori-mcp (npx @sap-ux/fiori-mcp-server). A plugin that fails to install is a warning, not a stop. The agents reach them through their tools: lists — see Agents. Server type. serverType and location are each one value or a list, tried in order, every location for a type before the next type: the default cx43 (8 shared vCPU, 16 GB, x86) gives Claude Code, npm ci, a test run and a headless browser room at the same time. When no EU location has a cx43 left (or Hetzner rejects it), start falls back to cax31 (8 vCPU, 16 GB, ARM; cloud-init installs the glab package for the server's architecture). The cx and cax types exist only in fsn1, nbg1 and hel1; where the server runs does not matter to the loop. When none of the types is free, start retries every minute until one is (Ctrl-C stops). --type cx43 asks for that type only, with no backup: start waits until Hetzner has one. cloud-init adds a 4 GB /swapfile, so a peak that once ended in an OOM kill on a cx33 now swaps instead. Both apply only to a newly created server. Every start sets TEST_MAX_WORKERS for the loop from the RAM the server really got: one Jest/Vitest worker per 4 GB, at most one per core (cx33 → 2, cx43 → 4). Set testMaxWorkers in factory.json to override it. The product's test config has to read the variable (maxWorkers: process.env.TEST_MAX_WORKERS). Hetzner renames types now and then — check hcloud server-type list.

The product must depend on a Spexso version that ships spexso factory: the server runs the product's own npx spexso.

Commands (from the project folder on your PC)

npx spexso factory start --milestone "Verlofbeheer" [--max-iterations 50] [--type cx43]
npx spexso factory status          # loops, last outcome, rate-limit wait
npx spexso factory logs            # follow the loop's journal: Claude's text, one line per tool call, refused permissions
npx spexso factory stop            # after the current iteration
npx spexso factory stop --now      # immediately; the issue in progress is reset

npx spexso factory start --milestone "Verlofbeheer" --test   # the test server, next to the build server
npx spexso factory status --test   # status / logs / stop take --test for the test server

start creates the server when it is missing (cloud-init: scaffold/factory/cloud-init.yml, a few minutes), writes the env file over SSH (tokens are never in user-data and never stored on your PC), clones or pulls the project into ~/projects/<group>-<project>, runs npm ci (with a server-side ~/.npmrc that authenticates @flexso with the bot token, so the product needs no .npmrc of its own), a claude -p smoke test, and starts the systemd user unit spexso-factory@<group>-<project>. Several projects can run side by side on one server. Running start again for the same project stops its loop first, refreshes the checkout and restarts it.

Before it touches Hetzner, start compares the quality gate in .gitlab-ci.yml (TEST_COMMAND, each &&-part of STATIC_CHECKS) with testCommand / staticChecks in .spexso/config.json and prints a quality gate drift warning for every CI command the build loop does not run. A type check that only CI runs is found only after a push and a full pipeline, and that stops the line. A staticChecks entry with {files} counts in its project-wide form.

The loop (spexso factory run, on the server)

Per iteration: factory stop file → git pull --ff-only (plus npm ci when the pull changed package-lock.json, so a project that upgrades spexso mid-run gets the CLI its pulled commands expect; the loop process itself keeps its code until the next start) → preflight (no tokens) → claude -p "/spexso-marathon --milestone X --max 1" → classify. An issue with a needs::* label is never picked: it waits on a human. When the sweep finds an open MR to finish, that iteration runs /spexso-finish --train-merge <id> --mr <mr> instead. claude -p runs with CLAUDE_CODE_PRINT_BG_WAIT_CEILING_MS=0: the marathon's issue runner is a background subagent, and print mode would otherwise stop it 10 minutes after the orchestrator's last turn, halfway through a large issue.

An iteration is done when its work has landed. A red gate in the merge train is backpressure: --train-merge runs npm ci in the worktree, waits for the MR pipeline, and gives the builder repair rounds as long as they make progress (at most 5) before it hands off (Finish). Stop the line: while a milestone issue sits at stage::review with a needs::* label and an open MR on spexso/issue-<id>, the preflight starts no new issue. The loop does not exit then: it waits, and checks every 10 minutes whether the MR was merged or closed, or lost its needs::* label. Then it resumes by itself (and treats the MR as new, so a relabelled one gets a fresh train-merge instead of a hand-off). The status issue, the webhook and factory status (heldBy in state.json) say what unblocks the line: !50 (#3): merge or close it, or remove needs::conflict-resolution. After 24 hours of waiting the loop exits stuck. The marathon inside the iteration runs the same check (npx spexso train guard). Building on, on a base without that work, is how conflicts pile up.

The sweep finds an issue's MR on its own branch (spexso/issue-<id>, or its lead's). When none is open or merged there, an open MR into the target branch whose title or description closes the issue counts too: a replacement MR a human opened on another branch is merged by the train like the original.

Claude runs with --permission-mode bypassPermissions: nobody is at the keyboard to approve a command the permission check cannot judge (pipes, variables, cd + git), and the server is single-purpose and disposable. What the agent can reach is bounded by the tokens on the server — the bot's GitLab token and the dedicated Hetzner project — which is why both must be scoped to the factory alone.

Exit reason	Code	When
done	0	nothing ready and nothing stuck; prints how many stage::opportunity issues still need a spec
stuck	2	an open MR held by needs::* (line stopped) for 24 hours, or nothing ready while issues with needs::* or blocked specs remain. The summary names the stuck issues and, separately, the specs that only wait on an open blocker (blocked: #34 by #33)
auth	3	the Claude credential was refused — renew the token
breaker	4	three errors in a row
max, stopped	0	--max-iterations reached, or factory stop

Bundles

Small issues each paying for their own worktree, review and MR pipeline is overkill. With bundle.budget in .spexso/config.json, one iteration builds a bundle: ready issues in picker order, as long as their weights fit the budget.

"bundle": { "budget": 10, "sizeScores": { "xs": 2, "s": 3, "m": 4, "l": 5, "xl": 10 } }

With these numbers, one bundle holds for example XS + XS + S (7), or M + L (9). An XL fills it alone. sizeScores are the defaults and can be left out; budget: 0 (the default) builds one issue per iteration. Bundling needs parallel.budget 1 and is never used by --autopick. The choice is code, not prompt: npx spexso window switches to bundle mode itself when the config asks for it, and marks its output with "bundle": true. A marathon orchestrator therefore cannot skip the bundle by stopping after the plain window.

• One branch: the first issue is the lead. The others ("companions") are locked with npx spexso bundle join, which adds a hidden <!-- spexso-bundle lead=… --> line to their description. Every member is planned and built in turn in the lead's worktree, then reviewed once, against every member's spec.
• One MR: it closes every member, and the merge train runs once. A companion that fails its build is reset out of the branch and goes back to stage::spec (bundle drop), to be built on its own. A failing lead fails the bundle before any companion starts.
• Per issue after the merge: each member gets stage::ship, needs::manual-test, its own checklist, demo note and spec archive. The test server therefore tests them one by one.
• Sweep: it finds a companion's MR through its lead's branch. Only the lead triggers the train-merge, and a held bundle MR stops the line just like a single one.
• Status issue: the Work column lists the issues an iteration actually built, a bundle with all its members.

A rate limit (subscription window) does not count as an iteration: the loop waits until the resetsAt of a rejected rate-limit event plus one minute, or 20 minutes when there is none, and retries. A retry while still limited fails on its first request, before anything is touched.

State lives in .spexso/factory/ (it ignores itself): state.json and per iteration the raw stream (<time>.jsonl) and the readable journal (<time>.log). The factory has its own stop file there, because the marathon deletes .spexso/stop when it honours it.

The sweep

Before every iteration the loop looks at milestone issues assigned to the bot at stage::build or stage::review without a needs:: label — what an interrupted iteration leaves behind:

Situation	Action
Last outcome rate_limited or interrupted, no MR	reset: worktree and branch spexso/issue-<id> removed, unassigned, back to stage::spec
Last outcome error or unknown, or ok without an MR	hand-off: needs::attention + note, unassigned. Commits the iteration made are pushed to spexso/issue-<id> first, and the note says so: the work survives the server
Open MR	merge: the worktree is restored from origin/spexso/issue-<id> when this server does not have it (the old one deleted itself), and the next iteration runs /spexso-finish --train-merge — rebase onto targetBranch, full suite and MR pipeline with repair rounds, merge. A conflict or a suite still red labels the issue as usual
Open MR, but the previous iteration already tried to merge it and was not cut short by a rate limit or stop	hand-off naming the MR
Open MR, worktree cannot be restored	hand-off naming the MR
Merged MR	nothing

The bot identity is what keeps the sweep away from a developer's own issues: never run the server with your personal GitLab token.

Following a run

The server deletes itself, so what matters leaves it while the loop runs. Every report is best effort: a failed upload or webhook never stops the loop.

• Status issue. Per milestone one issue [factory] <milestone> (type::tracking, no milestone, so nothing picks it up as work). Its description is the live state: running iteration n on which issue since when, rate-limit wait, or the exit reason, plus a table of this run's iterations. Every iteration adds a comment with its outcome and duration, with the journal and the gzipped stream-json attached as uploads. Those stay in GitLab after the server is gone. A run that ends done closes the issue; the next run opens a new one.
• Notifications. With SPEXSO_FACTORY_WEBHOOK set on your PC, start puts it in the server's env file and the loop posts start, every iteration, rate limits, hand-offs and the stop. The URL picks the format: Teams (a Workflows webhook on logic.azure.com, powerplatform.com or webhook.office.com) gets an adaptive card, ntfy plain text with a title (https://ntfy.sh/<secret-topic> is enough for your phone), anything else {"text": …} (Slack, Mattermost, Google Chat).
• Diagnosis. On a hand-off, and when the loop stops stuck or breaker, one short claude -p without tools reads the last iteration's journal and writes what happened, at which step it went wrong and the most likely fix. A hand-off's diagnosis lands on the handed-off issue, with the logs attached; a stop's diagnosis on the status issue. Without a journal (a fresh server) or a Claude answer (rate limit) the comment says so.

Test server (factory start --test)

A second server that runs next to the build server and does what a tester does after a merge: it tests every issue in the milestone that waits for a manual test, in a headless browser. The two meet only in GitLab. The test server files bugs as stage::spec issues in the milestone; the build server builds them; after the merge a bug gets needs::manual-test and the test server tests it in turn.

Its own server. --test creates spexso-factory-test (your serverName + -test) with its own self-deletion, so a test run never takes CPU or RAM from a build. It uses the same server types as the build server (cx43, then cax31): the browser is Playwright's own Chromium, which exists for x86 and ARM, not Google Chrome, which has no Linux ARM build. start --test installs Chromium's system libraries as root (once per server) and Chromium itself for the factory user, with a fixed link ~/.local/bin/spexso-chromium (the folder name differs per revision and architecture). It adds testPlugins (frontend-design, sap-browser-automation) and testMcpServers (@playwright/mcp with --browser chromium and chrome-devtools-mcp with --executablePath on that link, both headless and pinned) to the usual plugins and MCP servers, and writes FACTORY_MODE=test to the loop's conf. The systemd unit is the same; factory run reads the mode. A server created before this needs no new cloud-init.

The app runs on the server. The test server starts the app from its own checkout of the latest main, with demo.startCommand, and waits for demo.healthCheck (or demo.localUrl). start --test refuses without both keys, and they must be committed on the target branch: the server reads the config from its checkout. Mock users come from the project's CAP config (cds.requires.auth.users): no credentials on the server, no MFA. The app restarts for every issue, so each test starts from clean demo data.

One issue per iteration (a Ralph loop). Every iteration starts fresh:

1. git pull on main, and npm ci only when package-lock.json changed.
2. Ask GitLab again for the candidates: issues in the milestone with needs::manual-test (open and closed: a project that closes on merge keeps the label) and no auto-test-* label. Lowest id first. An issue merged or relabelled a minute ago is in the next iteration.
3. Start the app; a fresh claude -p "/spexso-autotest <id> --base-url <url> --out <dir>" tests it against the issue's latest checklist and demo scenario (the server writes them to <dir>/issue.md) and writes result.json; stop the app.
4. The server applies the verdict:

Verdict	What happens
pass	auto-test-success + a comment with one row per checklist item and the tested main SHA
fail	auto-test-fail + a comment with the screenshot. All new failures of the issue go into one new issue type::bug, stage::spec, in the same milestone, with steps to reproduce, one EARS AC per failure and the screenshot, linked to the failed issue: one build and one CI run instead of one per failure. A failure that a known bug already describes files nothing new: the issue is linked to that bug and waits for it (see below)
skipped	auto-test-skipped: nothing on the checklist can be checked in a browser (e.g. only "a test pins X")
blocked	no label: the test itself is wrong (outdated scenario, drifted selector, login, unreset data) or the run failed. Retried once in the same run, then listed in the summary

Retest. The next run skips every issue with an auto-test-* label. Remove the label to have an issue tested again. A bug carries a marker <!-- spexso-autotest parent=<id> depth=<n> -->: when it passes, the test server removes auto-test-fail from its parent, so the parent is tested again in the next iteration, up to the original. A fix that fails files a new bug, at most 3 deep; the fourth failure files nothing and is listed under "needs a human".

Known bugs. Before each test the server lists the milestone's unresolved bugs (type::bug, open or closed-but-untested, without auto-test-success) in issue.md. When the skill finds a failure one of them already describes, it names that bug in duplicateOf instead of writing it into bug.md. The server only accepts ids from that list. It links the issue to the bug and leaves a note <!-- spexso-autotest waiter=<id> --> on the bug; when the bug passes its own auto-test, every waiting issue loses auto-test-fail and is tested again, just like the parent.

When it stops. No candidate left: while the milestone still has build work (open stage::spec/build/review issues without a needs::* label), it checks again every 10 minutes; for at most 6 hours without anything to test. Then it posts the summary to SPEXSO_FACTORY_WEBHOOK (✅ passed, ❌ failed with their bugs, 🔁 tested again, ⏭ skipped, ⚠ blocked, 🙋 needs a human) and ends done; the server deletes itself 60 minutes later. An app that does not start ends the run stuck, and the server stays up 24 hours for a look.

Self-deletion

A user timer checks every 5 minutes. When no spexso-factory@* unit is active (a loop waiting on a rate limit is active) and nobody is logged in for 60 minutes — or 24 hours when a loop stopped on a problem (stuck, auth, breaker in its state.json), so you can still log in — the server deletes itself through the Hetzner API with the token from its env file. If start failed before writing that file, delete the server by hand: hcloud server delete spexso-factory. The timer script comes from cloud-init, so a server created before the 24-hour rule keeps the 60 minutes until it is recreated.

Licence

Anthropic's terms describe subscriptions as meant for ordinary, individual use; products or services should use an API key. A personal loop on your own subscription is fine to try; for structural client work switch to ANTHROPIC_API_KEY — the loop then passes --max-budget-usd (factory.maxBudgetUsd in .spexso/config.json, default 10) per iteration.