Disaster recovery — rebuild order
Rebuild-from-scratch path after catastrophic host loss. Assumes: Hetzner rescue/provisioning available, Storage Box intact, this vault available.
Phase 0 — what survives
| Asset | Location | State |
|---|---|---|
| This documentation | gitea repo (+ obsidian syncs) | current |
| Guest backups | backups LV on host (/mnt/backups/dump) | scheduled Mon/Thu, keep-last=5 — but same-RAID1, no offsite copy |
| Media/downloads | Storage Box | durable |
| Everything else | dead host | rebuild |
Phase 1 — host
- Provision the server / install Proxmox VE via Hetzner’s server setup.
- Run the PVE post-install script.
- Network: apply the SDN/bridge config — network-bootstrap
produces topology (
vmbr0+vnet0, SNAT,10.0.1.0/24). - Storage: re-add the Storage Box CIFS pool in the Proxmox UI, then
add the
optionsline to/etc/pve/storage.cfg— storage. - Tailscale: install on the host → management access returns immediately (Proxmox UI over tailnet only — never public).
Phase 2 — infrastructure services (in order)
| Order | Guest | Why first |
|---|---|---|
| 1 | technitiumdns (101) | DNS needed by everything’s resolution |
| 2 | caddy (100) | ingress; needs tailscale + --accept-dns=false + Cloudflare token |
| 3 | lldap (107) | directory for the IdP |
| 4 | pocketid (103) | SSO before consumer services |
Deploy each via its helper-script (see inventory for IDs/IPs), then:
- Rebuild the Technitium
«MYDOMAIN»zone from the table in dns. - Install Tailscale on
caddy,technitiumdns,gitea— tailscale. - Restore the Caddyfile — caddyfile.
- Re-create Pocket-ID admin + LLDAP connection; re-register OIDC clients (gitea).
Phase 3 — remaining services
In any order: gitea (104), apache-couchdb (105), wastebin (102).
- For guests with backups: prefer restoring from the
backupsLV (/mnt/backups/dump) over rebuilding. - Re-mount the Storage Box per guest type — storage.
- Re-create CouchDB databases/users per vault — obsidian-livesync.
- Gitea: re-add embedded-SSH keys, set advertised SSH host to the tailnet name — gitea.
Phase 3b — isolated dev subnet (optional, agent environment)
The vnet1 pair is self-contained and doesn’t gate other services:
- Re-create
vnet1(SDN zone + subnet10.0.2.0/24, SNAT on) and install/etc/network/if-up.d/vnet-rules— script and rationale in topology. - Rebuild
vpngw(LXC 111) — mullvad app + forwarding/killswitch rules +/dev/net/tunpassthrough +nesting=1, all in dev-vm. - Rebuild/restore
devvm(QEMU 112) — dev-vm-bootstrap for a fresh build; credentials are provisioned by hand.
Phase 4 — verify
- Tailnet client resolves
*.«MYDOMAIN»via split DNS. - Each service URL serves over https with a valid cert.
-
git clone gitea@gitea.«TAILNET_DOMAIN»:«SSH_USER»/<repo>.git. - OIDC login works on gitea.
- Snapshots restored / data intact.
- Cross-tailnet users re-share devices + Split DNS — dns.
See also
- lockout-recovery — locked out of the PVE firewall specifically.