dev-vm

Project: dev VM on the Proxmox host

Status: complete — devvm is the agent host; migration verified end-to-end (sessions, auth, repos, sync pipeline). Old VM retained as fallback until retired.

Move the Devin/agent development environment off the desktop workstation (which has a hardware fault and can’t stay up for days) onto the Proxmox server as an always-on guest — while preserving the rule that the agent environment cannot see identifying infrastructure details.

Design

Isolated subnet

A second SDN vnet — vnet1, 10.0.2.0/24 — hosts two guests:

• vpngw (LXC 111, 10.0.2.2) — a minimal VPN gateway. Its only outbound traffic is the Mullvad tunnel; it forwards devvm’s traffic into it.
• devvm (QEMU 112, 10.0.2.10) — default route via 10.0.2.2.

devvm 10.0.2.10 ──vnet1──▶ vpngw 10.0.2.2 ──wg0-mullvad──▶ Mullvad ──▶ internet
                              │            (WG over TCP — udp2tcp)
                              └── tunnel endpoint traffic only, via host SNAT

The tunnel transport is udp2tcp (WireGuard wrapped in TCP, via the mullvad app): the provider edge firewall drops all inbound UDP including replies, so raw WireGuard can never handshake (see topology).

Result: devvm’s egress IP is a Mullvad exit IP — «PUBLIC_IP» never appears in its path (e.g. curl ifconfig.me from devvm must never return «PUBLIC_IP»; the build checklist verifies this).

Host rules on vnet1

pve-firewall is enabled at the datacenter level, but host INPUT is effectively unfiltered (no host.fw; PVEFW-INPUT passes unmatched traffic — tailnet→host ssh works with no IN rule). So all vnet1 isolation lives in the mangle table — evaluated before the filter table, immune to PVEFW state and policies.

Rules are applied by /etc/network/if-up.d/vnet-rules — not by post-up lines in interfaces.d/sdn, which is SDN-owned and wipes hand-added lines on Apply. (vnet1’s SNAT + CT-zone lines are in the sdn file — SDN generates those itself when the subnet has SNAT enabled; only the mangle isolation ruleset below lives in the hook.) Full hook script in topology:

# host-local guard (PREROUTING sees packets before the INPUT decision)
-i vnet1 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
-i vnet1 -m addrtype --dst-type LOCAL -j DROP

# transit policy (FORWARD)
-o vnet1 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
-i vnet1 -o vnet1 -j ACCEPT
! -i vnet1 -o vnet1 -m conntrack ! --ctstate ESTABLISHED,RELATED -j DROP
-i vnet1 -d 10.0.0.0/8     -j DROP
-i vnet1 -d 172.16.0.0/12  -j DROP
-i vnet1 -d 192.168.0.0/16 -j DROP
-i vnet1 -d 100.64.0.0/10  -j DROP
-i vnet1 -d 169.254.0.0/16 -j DROP
-i vnet1 -o vmbr0 ! -s 10.0.2.2 -j DROP

Why each rule matters:

• --dst-type LOCAL drop — blocks new connections to every host-local address: 10.0.2.1, «PUBLIC_IP» (PVE UI :8006 — pveproxy binds 0.0.0.0), the host’s tailnet IP. devvm→host is same-subnet L2, never transits vpngw — this is the only guard. The ESTABLISHED accept before it preserves the ssh jump path (host→VM replies are ESTABLISHED).
• ! -i vnet1 -o vnet1 ! ESTABLISHED -j DROP — nothing (LAN guests, internet) may initiate connections into vnet1. Host→guest ssh is unaffected: it’s host OUTPUT, not FORWARD. The ! -i is load-bearing — bridged packets present vnet1 as both in/out devices, so a -o vnet1-only version also drops devvm’s internet-bound traffic headed for vpngw (confirmed live: it killed the installer’s connectivity).
• -i vnet1 -o vnet1 -j ACCEPT — all intra-bridge traffic is allowed. Must not be /24-scoped: packets bridged to vpngw carry their final destination, and Mullvad’s in-tunnel DNS (10.64.0.1) is inside 10.0.0.0/8 — a subnet-scoped accept would wrongly drop them (confirmed live). LAN-private enforcement for devvm is vpngw’s own -i eth0 ! -o wg-mullvad killswitch; the private drops below still cover routed-out bypass attempts (-o not vnet1).
• 100.64.0.0/10 — tailnet CGNAT; without it the VM could reach tailnet devices if the host routes them.
• ! -s 10.0.2.2 -o vmbr0 -j DROP — the second half of the killswitch: devvm cannot bypass vpngw by repointing its default route at 10.0.2.1. devvm’s legitimate traffic is L2 to vpngw and never leaves via vmbr0, so permitting only 10.0.2.2 to egress is complete by construction.
• No IPv6 is configured on vnet1; if that changes, equivalent ip6tables rules are required.

vpngw rules

Debian 13 LXC (unprivileged), mullvad app + iptables, net.ipv4.ip_forward=1 — persisted in /etc/sysctl.d/90-vpngw.conf (a bare sysctl -w does not survive reboot):

iptables -A FORWARD -i eth0 -o wg0-mullvad -j ACCEPT
iptables -A FORWARD -i wg0-mullvad -o eth0 -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
# killswitch: devvm traffic may only ever leave via the tunnel
iptables -A FORWARD -i eth0 ! -o wg0-mullvad -j DROP
iptables -t nat -A POSTROUTING -o wg0-mullvad -j MASQUERADE
# MSS clamp: tunnel MTU is 1326 — without this, TCP through the tunnel
# stalls on the first large transfer
iptables -t mangle -A FORWARD -o wg0-mullvad -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

Persisted via iptables-persistent (netfilter-persistent save after changes). The ! -o wg0-mullvad drop means: tunnel down → devvm traffic dies, no fallback. The gw itself needs nothing but the tunnel endpoint — and the host’s vnet1 drops confine the gw exactly like any other vnet1 guest.

Container config — unprivileged LXC has no /dev/net/tun by default; mullvad’s userspace tunnel needs it. In /etc/pve/lxc/111.conf on the host:

features: nesting=1
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net/tun dev/net/tun none bind,create=file

nesting=1 is required on Debian 13 (systemd v256+). Without it the container can’t create mount namespaces, and the whole class of sandboxed/credentialed systemd units fails at boot — systemd-sysctl dies with status=243/CREDENTIALS, journald and tmpfiles-setup fail, the tmpfs mounts (tmp.mount, run-lock, dev-mqueue) fail. Signature observed live: the gw looks healthy and its own tunnel works (curl ifconfig.me returns the exit IP), but sysctl.d is never applied so ip_forward silently stays 0 after every reboot — devvm loses all connectivity with zero log evidence (journald is dead too). Check with systemctl --failed inside the CT.

mullvad settings: udp2tcp transport (required — see topology), lockdown-mode on, auto-reconnect on. Lockdown also drops inbound LAN traffic — so pinging 10.0.2.2 from devvm fails by design (pct exec/pct enter is the management path; nothing should ever initiate a connection to vpngw anyway).

Exit location matters for throughput. udp2tcp wraps the tunnel in a TCP connection to the relay — TCP-over-TCP throughput craters with outer-leg RTT. A transatlantic us exit measured sub-200kB/s from the host; se (single-hop, ~10ms) restored normal speeds. If a distant exit is ever wanted again, use multihop with a near entry — but note not every entry relay listens on udp2tcp (fi failed with EphemeralPeerNegotiationTimeout).

vpngw DNS — its own resolver needs TCP to cross the edge: resolv.conf with public nameservers + options use-vc (glibc does TCP/53; replies pass via the edge’s ACK rule). Only needed for the Mullvad API at connect time.

devvm DNS/MTU: static IP, DNS 10.64.0.1 (Mullvad’s in-tunnel resolver — they hijack/filter port 53 to other resolvers as an anti-leak measure, so 1.1.1.1 lookups can appear to time out), and interface MTU 1200 in netplan (must stay at or below vpngw’s udp2tcp tunnel MTU of 1326 — itself not WG’s usual 1420). Note: the live link ran 1280 for a while after the config was lowered — a netplan apply or reboot is what actually re-syncs it.

Access path

operator ──tailnet──▶ «HOSTNAME» ──ssh──▶ devvm 10.0.2.10

• The dev VM is not tailnet-joined — tailscale status would expose peer names and «TAILNET_DOMAIN».
• Operator ssh config: ProxyJump «HOSTNAME» (the host can reach 10.0.2.x; the isolation only blocks VM-initiated traffic). Aliases: devin → agent user devin@10.0.2.10 (workspace, mirrors, devin CLI); devvm → admin user devvm@10.0.2.10 (sudo). Mirror remotes and repo clones over ssh must use the devin alias — ~/mirrors lives in the agent user’s home.
• Fallback console: web-UI noVNC console (qm terminal needs a serial port — qm set 112 --serial0 socket + kernel console=ttyS0 if ever wanted).

Guest specs (as built)

devvm:

Type	QEMU (full env parity: podman, kernel control)
VMID	112
Name	devvm
OS	Ubuntu Server 26.04.1 — same release as the workstation VM (resolute), exact tool parity
vCPU	4 (host is i7-7700, 4c/8t) — verified from guest 2026-10
RAM	8 GB (≈7.3 GiB visible) — verified from guest 2026-10
Disk	48 GB virtio — root LV is 23G of the 46G VG, ~23G unallocated headroom
IP	10.0.2.10/24, gw 10.0.2.2 (vpngw), MTU 1200, DNS 10.64.0.1
Firmware	SeaBIOS — do not switch to OVMF: virtiofs+OVMF deadloops on PVE 9/QEMU 11 (2026-10-08-ovmf-virtiofs-deadloop)

vpngw:

Type	LXC (unprivileged; needs /dev/net/tun passthrough — see rules section)
CTID	111
Name	vpngw
OS	Debian 13 minimal
vCPU / RAM / disk	1 / 512 MB / 4 GB — it only runs mullvad-daemon + iptables
IP	10.0.2.2/24, gw 10.0.2.1
Notes	Mullvad account: uses one device slot (max 5). Account number/credentials live only on the guest — never committed. Created directly (pct create), not via helper script — helper scripts hang on this subnet (they run apt before you can fix resolv.conf).

SDN IPAM vs static IPs — resolved 2026-09

Attaching 112’s NIC to vnet1 auto-reserved the next free IPAM entry (.1 gw, .2 vpngw, .3 this MAC) while the guest is statically .10 via netplan — the panel showed the reservation, not reality. Resolved: the DHCP range was dropped from vnet1 entirely (static is enforced — it’s the secure subnet) and 112’s IPAM entry corrected to .10. Nothing ever consulted the reservation — the mangle rules key off interfaces + the ! -s 10.0.2.2 guard.

Bootstrap

dev-vm-bootstrap — replicates the agent environment: devin user, openssh-server, git, uv, podman, devin CLI, dotfiles. Credentials (ssh keys, devin auth) are provisioned manually at build time — never in the repo.

Build checklist

• Create vnet1 in the SDN UI (zone + vnet, subnet 10.0.2.0/24 gw 10.0.2.1, SNAT enabled); install /etc/network/if-up.d/ vnet-rules (script in topology), chmod +x; run IFACE=vnet1 /etc/network/if-up.d/vnet-rules; verify iptables -t mangle -L PREROUTING -n / -L FORWARD -n show the vnet1 rules and iptables -t nat -S POSTROUTING shows both SNATs.
• Host firewall state confirmed: no host.fw, INPUT jumps to ts-input + PVEFW-INPUT, unmatched traffic accepted — recorded in topology. (Design is PVEFW-agnostic via mangle.)
• Create vpngw (LXC 111): pct create from Debian 13 template, static 10.0.2.2/24, gw 10.0.2.1. Add /dev/net/tun passthrough lines and features: nesting=1 to /etc/pve/lxc/111.conf, restart. resolv.conf → public nameservers + options use-vc (TCP DNS through the edge).
• Install mullvad app .deb; account login; transport udp2tcp; lockdown + auto-reconnect on; connect. ip_forward=1 via /etc/sysctl.d/90-vpngw.conf (not sysctl -w alone), the forwarding/killswitch/clamp rules above, iptables-persistent + netfilter-persistent save. Verify curl ifconfig.me on the gw shows a Mullvad IP, and devvm-style traffic dies when disconnected.
• Create QEMU 112 (qm create + Ubuntu 26.04.1 ISO), install via web-UI console: static 10.0.2.10/24, gw 10.0.2.2, DNS 10.64.0.1, disk 48 GB. (qm terminal needs a serial port — use the console.)
• Post-install: netplan mtu: 1200; verify isolation from the VM: no ping/ssh to 10.0.1.x, 10.0.2.1, «PUBLIC_IP»; internet works and curl ifconfig.me returns a Mullvad IP, never «PUBLIC_IP».
• Killswitch test: mullvad disconnect on vpngw → devvm loses all connectivity (no silent fallback).
• Run dev-vm-bootstrap.
• Provision credentials: devin auth login, operator pubkey into both users’ authorized_keys. No gh — account suspended; remotes are local bare repos, no keypair needed on the guest.
• Operator ssh config ProxyJump entry; test end-to-end.
• Add both guests to backup/snapshot set (TODO tracks backup automation).
• Update inventory (remove “planned” markers).

Cutover checklist

• git clone mirrors work from devvm.
• A devin session on devvm can run the doc pipeline repo.
• Desktop VM stays until devvm is proven; then retire. (devvm proven — sessions/auth/pipeline all verified. Old VM kept running as fallback; retire at leisure. Its ~/mirrors copy is now stale — never sync against it.)

Non-docs repos (cogmind-coach, pub-devops-repo-l3-main): ferried to gitea via operator clone-over-ssh (git clone devin:repos/<name>). One-time by design — no ongoing sync; re-ferry manually if needed. Their devvm origins still point at the dead github remotes (tombstones).

Open questions

• Resource allocation final numbers (vCPU/RAM/disk) — resolved 2026-10 from the guest: 4 vCPU, 8 GB RAM, 48 GB disk (see specs).