Project: observability — platform metrics + alerting
Status: design — drafted 2026-10-16, pending review, then an implementation session. Scoped platform-wide, not tied to any one service stack. Operator chose the full metrics path (Prometheus + Grafana) over lightweight checks. Related: inventory, catalog, storage, dns, diag-pipe.
Why
Recent incidents were each discovered by the operator noticing a symptom, then diagnosed by hand. The failure classes span the whole platform, not one service:
| Incident | Symptom noticed | Detector that should have fired |
|---|---|---|
| Storage Box CIFS session drop | virtiofs guests wedged mid-IO | storage mount canary + I/O-stall alert |
| virtiofs stale mount (devvm, 2026-10-16) | share mounted but listing empty | virtiofsd presence + dir-listing probe |
| guest deadloop (2026-10-08-ovmf-virtiofs-deadloop) | VM unbootable on restart | guest liveness/heartbeat metric |
Goal: detect these classes automatically, plus the usual baseline — guest/service liveness, disk headroom, backup freshness.
Design
Placement
New lightweight LXC (mon, next free VMID/IP per inventory) —
independent of any single service stack, so monitoring keeps working
while other guests are being serviced. Debian 13 + rootful podman +
podman-compose, same recipe as the other podman guests. Scraping is
plain LAN HTTP — no new exposure; everything internal per network
rules.
Components
| Piece | Covers |
|---|---|
prometheus | scrape + rule eval + alerts |
grafana | dashboards; fronted via Caddy (mon.«MYDOMAIN») like the rest |
pve-exporter | VM/CT up-down, cpu/mem/storage per guest, node health — platform-wide coverage in one exporter (needs a read-only PVE API token) |
node-exporter | per-guest disk/mem/cpu — roll out guest-by-guest, not blanket v1 |
blackbox prober | HTTP liveness across the Caddy-fronted service inventory (catalog/dns already enumerates the sites) + TLS expiry |
| per-service exporters | apps that expose /metrics natively or via a sidecar — add as they land |
| textfile-collector probes | cron jobs emitting incident-specific detectors |
Per-service candidates as they come online: gitea (native metrics endpoint), the wiki deploy job, generic systemd-unit health.
Incident-derived detectors
Cron jobs on mon (or the relevant host) emitting textfile metrics —
each maps to a real failure above:
- storage mount canary — host-side stat/open probe on a canary
path under the Storage Box share +
dmesgscan for freshCIFS/Send errorlines → metric. Catches the Oct-1 wedge class before anyone hits play. - virtiofsd health — per-guest virtiofsd process presence + a directory-listing probe on each virtiofs consumer. Catches the mounted-but-empty class seen live on devvm.
- guest liveness — pve-exporter
pve_upper VMID + optional qemu-guest-agent heartbeat → catches deadloop/hung guests. - backup freshness — age of newest successful vzdump per guest vs the Mon/Thu schedule → catches silent backup rot.
Alerting path
Alerts only need to reach the operator: Grafana built-in alerting to a local target (self-hosted ntfy, or a webhook the operator picks). No external egress required. Alertmanager is a one-line addition later if paging-grade is wanted.
Open questions
- VMID/IP allocation for
mon— next free per inventory. - Grafana behind Caddy (
mon.«MYDOMAIN») per convention — assumed yes. - Read-only PVE API token for pve-exporter — token lives in the
monLXC only. - Retention + sizing: 15–30 d of metrics at default interval — a few GiB; trivial for an LXC.
- node-exporter rollout order — start with
mon+ PVE host + the storage-touching guests, extend later. - Which Caddy-fronted sites get blackbox probes v1 — generate from catalog?
Non-goals
- Logs aggregation (Loki) — the observed incidents were metric-shaped; revisit if gaps appear that metrics can’t see.
- Tracing / APM — irrelevant at this scale.