Project: VPS ingress relay (rathole)

Status: design decided, not yet deployed (2026-10). Related: topology (edge invariants), tailscale (tailnet membership model), dns (service records), caddy (TLS termination).

Replace the DERP-only tailnet path for HTTP ingress with a dedicated relay: tailnet clients establish direct WireGuard UDP to a nearby VPS, which forwards raw TCP into a persistent outbound-initiated TCP tunnel held open by the server side. The transatlantic leg becomes UDP; the TCP-carrying-TCP hop is confined to a clean ~1ms same-backbone link.

Problem

«PUBLIC_IP» sits behind the Hetzner Robot firewall, which is stateless: the tcp ack→ephemeral rule passes TCP replies, but UDP has no reply flag, so all inbound UDP is dropped — including replies to outbound UDP. Consequence (topology): tailscaled can never hold a direct UDP session — every byte of tailnet ingress rides DERP (shared, TCP-encapsulated), which is the throughput bottleneck for media streaming.

The invariant that produces this is deliberate: with a stateless edge, any UDP accept admits unsolicited datagrams, so “no inbound UDP, ever” is the only “no unsolicited inbound” posture available.

Options considered

Option	Verdict
Tailscale peer relay (GA ≥1.86)	Best transport (UDP datagram forwarding, no TCP nesting), but the relay→server leg needs inbound UDP replies — dead at the edge. Possible only with a source-scoped edge rule (`udp, src «VPS_RELAY_IP»/32, dport 32768–65535`) + host `INPUT -i vmbr0 -p udp -m conntrack ! --ctstate ESTABLISHED,RELATED -j DROP`, which restores kernel-level established-enforcement one layer down. Softens the letter of the invariant; deferred — kept as fallback design. Also only serves own-tailnet clients.
Self-hosted derper	Works with zero edge changes (both legs are outbound-initiated TCP, pass the `tcp ack` rule). But keeps TCP nesting end-to-end, and — critically — caddy’s home DERP moves to the custom region, which foreign-tailnet clients can’t resolve without adding the same `derpMap` to their tailnet policies.
VPS L4 forward + reverse TCP tunnel (chosen)	Client→VPS is a real direct WireGuard session (VPS has a public endpoint); VPS→server is server-initiated TCP — passes existing edge rules unchanged. TLS passes through to caddy untouched. Works for foreign-tailnet viewers via the existing sharing pattern (share the VPS node, repoint DNS).

Design

tailnet client ──WG UDP, direct──▶ VPS ──rathole──▶ caddy:443 ──▶ services
        (VPS has public endpoint; client initiates)
                                   ▲
                    persistent outbound TCP tunnel,
                    initiated server-side — passes the
                    tcp-ack edge rule; zero edge changes

VPS: cheap VM near the DC (same backbone as «PUBLIC_IP»); tailscaled + rathole server + sshd. Joined to the tailnet as a dedicated node; shared to foreign tailnets exactly like caddy is today. Public surface: tailscaled UDP + sshd + rathole control port firewall-scoped to src «PUBLIC_IP». Service listener binds the VPS’s tailnet IP only — nothing service-related is public.
Server side: rathole client inside the caddy LXC, targeting 127.0.0.1:443, dialing «VPS_PUBLIC_IP»:7000 outbound. Fresh outbound data channel per client connection (rathole’s anti-HoL design — parallel streams don’t block each other).
DNS: repoint the «MYDOMAIN» service A records in technitium from «CADDY_TAILNET_IP» to «VPS_TAILNET_IP». One :443 forward upgrades the entire service catalog — caddy still routes by SNI. Foreign tailnets get the fast path automatically via their existing split-DNS + the shared node.
TLS: end-to-end client→caddy, untouched — the VPS forwards ciphertext (SNI visible, nothing more). Dumber than DERP in trust terms.

rathole config sketch

VPS (server.toml — rathole’s “server” is the side that listens):

[server]
bind_addr = "0.0.0.0:7000"
# + noise transport (protects the token on the wire)
 
[server.services.https]
token = "«RATHOLE_TOKEN»"
bind_addr = "«VPS_TAILNET_IP»:443"

caddy LXC (client.toml):

[client]
remote_addr = "«VPS_PUBLIC_IP»:7000"
retry_interval = 1
# + noise transport
 
[client.services.https]
token = "«RATHOLE_TOKEN»"
local_addr = "127.0.0.1:443"

systemd: Restart=always on the client; After=tailscaled on the VPS unit (bind is to the tailnet IP).

VPS firewall

The VPS is the relaxed end — real public IP, ordinary stateful filter. Public surface is tailscaled + (optionally) sshd only:

Port	Interface	Source	Purpose
`udp/41641`	public	`0.0.0.0/0`	tailscaled WireGuard transport — internet-facing by design, crypto-gated
`tcp/7000`	public	`«PUBLIC_IP»/32` only	rathole control + all per-connection data channels
`tcp/443`	`tailscale0`	tailnet (ACL-gated)	rathole forward listener — never public
`tcp/22`	optional	—	sshd; tailnet-only ssh + provider console preferred

table inet filter {
  chain input {
    type filter hook input priority 0; policy drop;
    ct state established,related accept
    iif lo accept
    udp dport 41641 accept
    ip saddr «PUBLIC_IP» tcp dport 7000 accept
    iif "tailscale0" tcp dport 443 accept
  }
}

Egress: default-allow (TCP 443 to control plane/DERP fallback, UDP 3478 STUN, DNS, package mirrors). Tailnet ACL additionally denies all VPS egress — enforced by tailscaled independently of this table.

Network/security notes

Edge invariant preserved verbatim — still zero inbound UDP, still no new-inbound of any kind; every server-side leg is outbound initiated. Unsolicited traffic still dies at transport/kernel layer, never reaching an application.
VPS is the only new internet-facing thing, and it sees only ciphertext. ACL it to nothing: it never initiates tailnet traffic — grant is members → «VPS_HOSTNAME»:443 only, deny its egress to all else.
Client-IP masking accepted: caddy sees connections from the VPS tailnet IP. Doesn’t matter — auth is per-user via OIDC. If attribution is ever wanted, switch to frp (proxyProtocolVersion v2)
- Caddy proxy_protocol listener wrapper.
Residual overhead: server leg is TCP-in-TCP per stream, but on a ~1ms clean link the nesting penalty is negligible — the congestion that made DERP painful lived on the long-haul leg, now UDP.
Churn: media apps open many short conns (trickplay, API, websockets); each costs a fresh outbound data channel. Harmless at this scale.

Deployment checklist

Provision VPS near DC; join tailnet (dedicated tag), share to foreign tailnets; lock firewall («PUBLIC_IP»-scoped rathole port, tailscaled UDP, sshd).
rathole server + client units deployed; tunnel up.
Tailnet ACL: members → «VPS_HOSTNAME»:443; deny VPS egress.
Repoint «MYDOMAIN» A records in technitium to «VPS_TAILNET_IP».
Verify: tailscale status shows direct to VPS from a client; curl -v https://wastebin.«MYDOMAIN» returns caddy’s cert unchanged; throughput vs DERP baseline (iperf3/large-file pull).
Docs on cutover: rewrite the “no inbound UDP, ever” warning block context in topology (invariant unchanged — add relay path), update tailscale membership table, dns records, catalog.

Open questions

Whether the peer-relay fallback design (scoped edge UDP rule + host conntrack gate) is worth revisiting later — strictly better transport, but adds the foreign-tailnet grant problem back.
Revert path: repoint DNS back to «CADDY_TAILNET_IP» — nothing else breaks; keep old records commented in technitium for instant fallback.

PVE Infra Docs

Explorer

vps-relay