publish-pipeline

Project: publish-pipeline — automated docs release

Status: live (2026-10) — end-to-end run verified. Replaces the manual release runbook with a dispatched Gitea Actions workflow plus a host-side git relay. The human gate is unchanged: operator review of the sanitized diff, then dispatch.

Related: dev-vm, infra-wiki, wiki, gitea, diag-pipe (same trust boundary, shared redaction state).

Summary

A docs release used to be a ten-step manual runbook across three machines. It is now: Devin merges to the devvm mirrors and tags a publish epoch; the operator reviews the diff and dispatches publish.yml with that epoch; a dedicated runner LXC does the rest — un-redact into gitea, rebaseline the mirror, rebuild + deploy both wikis, and stamp a new epoch on the way out. A watcher on the host relays refs between the isolated devvm and the runner over git transports only.

The epoch tag is both the input barrier (“don’t start until this exact cross-repo state arrived”) and the completion signal (“the new tag on every devvm mirror = the run landed”). A missing output epoch is the failure indication — no separate health check needed.

Motivation

The manual flow was ~10 ordered steps on the operator machine (sync-in, sync-out, wiki build+deploy×2, backups, state persistence, per-machine clones and vault refresh). Complex, correct, and begging for automation — every step was mechanical except the diff review. The constraint that shaped the design: devvm must not gain connectivity to anything, so the runner can’t fetch the mirrors directly and the host can’t be mounted into the runner.

Architecture

The relay (host-ferry)

devvm mirrors ──ssh fetch──▶ /srv/mirrors/*.git ──ssh push──▶ runner /srv/mirrors
devvm mirrors ◀──ssh push─── /srv/mirrors/*.git ◀──ssh fetch── runner /srv/mirrors
runner ──https+PAT──▶ gitea (vnet0)   runner ──ssh──▶ wiki (vnet0)

host-ferry.sh runs on the Proxmox host as a systemd watcher (watch mode — ls-remote polls both endpoints every ~5s, relays only on real ref changes). Both transports are host-initiated — the host reaches devvm (devin alias) and the runner (ci alias) because guest→host rules only drop guest-initiated traffic; host OUTPUT into either network is fine. No firewall exceptions, no shared filesystem, no bind mounts.

Per pass, per repo:

1. fetch --prune devvm — the store’s refs/{heads,tags} track devvm (canonical; devvm is authoritative).
2. fetch runner — the runner’s refs land in refs/tracking/runner/* (a parallel namespace; runner state never shares canonical refs).
3. Deliver differing tracked refs to devvm in one git push --atomic — heads fast-forward-only; if any tracked head is behind or diverged the whole repo’s delivery is skipped (a stale run’s baseline/epoch tags must never land pointing at orphaned commits). New branches always deliver.
4. fetch --prune devvm again, then converge the runner to canonical — force heads+tags, delete runner refs canonical lacks. Between runs the runner’s copies are pure slave replicas.

Git transport everywhere means every observable state is a consistent one — that’s why this beat a shared volume (filesystem copies can tear a live packfile; fetch/push are transactional per ref).

The publish-lock ref

A whole run is a transaction boundary git can’t express (many ops across many repos), so the run marks it with a ref: refs/ferry/publish-lock on the runner’s copy of infra-docs, set after the epoch wait and deleted on exit. While present the ferry skips all runner-side ops; devvm↔store traffic continues. A stale ref self-heals — the next run sets and clears it.

Publish epochs

ci/epoch.sh hashes the refs/heads/main sha of every repo in EPOCH_REPOS into one value and tags each repo’s main with publish-epoch-<hash> (annotated; the manifest is the tag message).

• Input: Devin tags post-merge; dispatch passes the hash as expect_epoch. The run polls every 5s until the tag exists on all epoch repos on the runner’s mirrors — dispatch is fully timing-independent.
• Output: on a clean run only, the runner recomputes the epoch over its mutated heads and re-tags; the tags ride the ferry back after the lock clears.
• Check: ci/epoch.sh status ~/mirrors <repos> on devvm — all repos agreeing on one epoch = last run completed and durable.

Runner container

LXC 116 gitea-runner (10.0.1.36) — unprivileged Debian, needs features: nesting=1 for the containerized Quartz build/wrangler deploy. Runs gitea-runner daemon under systemd (registered label gitea-runner:host, host executor — no podman-in-podman). Root is fine inside an unprivileged CT; the gitea runner and the git-transport user share /srv/mirrors on the same identity.

Trust model

Same boundary as the docs mirror and diag-pipe: the runner never reaches devvm or the host — it talks only to gitea, the wiki deploy target, and public egress (image pulls, Pages upload). The ferry is the choke point: a compromised runner can only write its own repo copies; the only thing crossing is refs the ferry chose to relay, and heads can never be force-moved backward on devvm.

The human gate stays upstream: branch review, then dispatch is the approval click. Automation covers transport, bookkeeping, redaction — same division as diag-pipe.

Failure semantics

• Runner crashes mid-run → lock ref may linger (harmless), output epoch never stamps → epoch.sh status shows the old epoch → fix and redispatch.
• devvm mirror moved during a run (operational violation — devvm’s mirrors are read-only for Devin while a run is in flight) → the ferry’s divergence abort drops that repo’s output → output epoch never completes → manual state repair.
• Wiki build/deploy fails → phase returns nonzero, epoch not stamped, wiki-deployed-sha not recorded — rerun is not skipped next time.

Shakedown notes (things that actually broke)

• LXC needs features: nesting=1 in 116.conf for docker-in-LXC — pct set writes it but the container needs a real stop/start.
• A gitea bot user with “Disable Sign In” set can’t auth git over https at all — PAT included.
• GITEA_* is a reserved prefix for actions variables and secrets — hence VAR_GITEA_* vars and BOT_TOKEN/API_TOKEN.
• git fetch auto-follows tags into refs/tags/* regardless of refspec destination — remote.*.tagopt=--no-tags on the runner remote is load-bearing, or runner tags leak into canonical.
• git clone --mirror leaves remote.*.mirror=true behind — a normal push under it becomes a mirror push and can revert the remote. ensure_remotes strips it every pass.
• Phase functions invoked as phase || run_failed=1 run with errexit suppressed — internal failures must return 1 explicitly or a failed deploy still stamps the epoch.
• The runner caches STATE_DIR between runs; a clone made while the state repo was empty has no branches — fetch-before-checkout heals it.
• A wrong-but-writable WIKI_PATH deploys green: rsync --delete creates the target, exits 0, wiki-deployed-sha records a deploy the webroot never saw. ci.example.env now ships the real webroot, but verify ci.env’s WIKI_PATH matches the wiki Caddyfile root — and after any such fix, force_wiki=true to override the falsely recorded sha.