Skip to content

18 — Observability (metrics, logs, alerts)

One pane of glass for host health, service up/down, appliance metrics, and firewall-conformance drift, with push alerts to phones. Built on the existing AlmaLinux + podman-quadlet + Ansible conventions and the default-deny zone model.

Architecture (push model)

Every fleet host runs Grafana Alloy (pinned binary + systemd) that scrapes local node metrics and tails journald, and pushes to an obs VM in SERVERS (10.20.20.50) that initiates nothing — a pure sink. Appliances (which can't run Alloy) push via their native line protocols to the same sink.

  • obs VM = 4 podman quadlets (roles/obs): VictoriaMetrics (:8428, metrics TSDB; InfluxDB ingest on /write, Graphite on :2003), Loki (:3100, logs), Grafana (:3000, UI), Alloy syslog-gateway (:514 → Loki).
  • Fleet agent (roles/alloy, all_linux, serial: 1): node metrics → VictoriaMetrics remote_write; journald → Loki. The VPS edges run it too (edge.yml, since 2026-07-10): they push to obs over the WG tunnel — obs's /32 is routed in edge_peer_allowed_ips and the origin masquerades the flow (origin_tunnel_masq_dests), so it reaches OPNsense as ordinary DMZ→obs traffic on the existing port-scoped 8428/3100 allows (no new firewall rules). Per-edge attribution = the Alloy host label (obs sees the origin's IP as the source).
  • ntfy is a separate standalone bus (roles/ntfy), not part of obs.
  • Grafana = mon.${domain} (wg-admin only, via the mgmt-proxy). Auth = local admin + Authelia OIDC SSO (fleet-admins → Admin, else Viewer; local admin kept as break-glass). See roles/obs (the OAuth2 client is declared in the Authelia config).

Apply: ansible-playbook -i inventory.ini playbooks/site.yml -l obs then --tags alloy for the fleet agent. Dashboards (Node Exporter Full, Fleet logs) and alerting (ntfy contact point + rules) are provisioned as code under roles/obs/files/dashboards and roles/obs/templates/grafana-alerting.yml.j2.

Dashboard: Site visits (server-side) (site-visits.json). Public-site visit counts for sites that ship no analytics JS — derived from the origin Caddy access log (access_log: true per vhost, runbook 09 §9), which reaches Loki over the journald pipeline already described here. No new collector. The dashboard groups by request_host, so it is generic: opting another vhost in makes it appear with no edit. Client IPs arrive pre-masked to /24 and carry no UA/Referer/cookie — the counting logic (what is a "page view" vs an asset vs a download) lives in the dashboard, not on the TLS terminator. Two horizons: the raw lines expire with obs_loki_retention (30 d, they are request records), while the count is promoted to a metric — alloy_access_log_metrics: true makes that host's Alloy parse its own Caddy access lines into site_http_requests_total{vhost,class} (closed label set; never the path — bots would make it unbounded), remote-written here and kept for obs_vm_retention (13 mo). So the long-horizon artifact is an aggregate, not a request record. Only that one series is kept from Alloy's self-scrape; its internal metrics are dropped before remote_write.

Firewall: DMZ → SERVERS observability ports (LIVE)

DMZ hosts run agents too, so the obs build added one allow rule group on the DMZ (opt3) interface, between the Authelia allow (origin Caddy → :9091) and the Block DMZ → RFC1918 rule:

Action Proto Source Destination Port Purpose
pass TCP DMZ net 10.20.20.50 (obs) 8428 Alloy → VictoriaMetrics
pass TCP DMZ net 10.20.20.50 (obs) 3100 Alloy → Loki
pass TCP DMZ net 10.20.20.60 (ntfy) 8090 DMZ publishers → ntfy

The rule is port-scoped (DMZ still cannot reach obs SSH / other SERVERS ports). Regression-tested in tests/fw-conformance/conformance.py. MGMT→SERVERS and intra-SERVERS were already allowed, so Proxmox/OPNsense need no new rule.

Appliance feeds

Proxmox — DONE (InfluxDB metric server)

Configured cluster-wide; PVE (MGMT) → obs (SERVERS) is already allowed:

pvesh create cluster/metrics/server/obs \
  --type influxdb --server 10.20.20.50 --port 8428 --influxdbproto http \
  --max-body-size 1000000

VictoriaMetrics ingests the InfluxDB line protocol on :8428/write (no auth, v1). Verify: blockstat_*, cpustat_*, memory_* series appear in VM.

OPNsense — telegraf metrics (DONE) + remote syslog (DONE)

Metrics — DONE. os-telegraf is installed and enabled with the InfluxDB output → obs and the network + pf inputs on (plus the cpu/mem/disk/system defaults). Equivalent GUI path: System → Firmware → Plugins → install os-telegraf → Services → Telegraf → General: Enabled; Output → InfluxDB URL http://10.20.20.50:8428, database opnsense (VM ignores it), no auth; Inputs enable network + pf. Metrics arrive tagged host=opnsense.${domain} — cpu_*, mem_*, net_bytes_* (per zone VLAN), pf_entries/pf_match (state table). The firewall edits applied: <telegraf> general/enabled, output/influx_enable|influx_url|influx_database, input/network|pf, then configctl template reload OPNsense/Telegraf && configctl telegraf restart.

Logs — DONE (2026-07-12, via API). One remote-logging destination: 10.20.20.50:514, transport tcp4, rfc5424 = 1 — the obs Alloy syslog-gateway accepts RFC5424 over TCP (octet-counted) only; the GUI default (UDP/RFC3164) is silently rejected (expecting a version value, see runbook 24). Applied over the API with the fw-automation token (~/opnsense-api-token): POST /api/syslog/settings/addDestination (enabled=1, transport=tcp4, hostname/port as above, rfc5424=1) then POST /api/syslog/service/reconfigure. Rebuild note: this is runtime config.xml state — re-run those two API calls (or GUI: System → Settings → Logging / Targets, transport TCP, RFC5424 checked) after an OPNsense rebuild. Verify in Loki: {job="syslog", host="opnsense.${domain}"} — includes filterlog (per-rule pass/block lines).

OPNsense → obs is directly-connected (the firewall owns 10.20.20.1), not Proton, so no extra allow rule is needed.

fileserver — no appliance feed. The fileserver is a normal AlmaLinux 9 fleet host: metrics + logs are shipped by the fleet alloy agent (the same role as every other Linux box), so no appliance-specific Graphite/syslog config is needed.

Alerts

grafana-alerting.yml provisions a single ntfy webhook contact point (http://10.20.20.60:8090/lab-alerts, internal SERVERS path), a root policy routing everything to it (+ a lab-heartbeat sub-route, see below), and these rules (all modelled on each other; runbook 27 is the canonical detail for the failure/ delivery/resource ones):

  • fleet-host-down — lag(node_load1[20m]) > 300s (NoData → Alerting: a fully-down host stops pushing; [20m] so it keeps its $labels.host for 5–20m before NoData).
  • fleet-unit-failed — node_systemd_unit_state{state="failed"} == 1 for 2m; catches any crashed service or podman-quadlet container. Needs the alloy systemd collector.
  • fleet-host-mem / -disk / -cpu — real resource pressure: memory via MemAvailable (>90%, not Proxmox's cache view), disk >90% (ext4/xfs/btrfs), CPU >90% utilisation (NOT load1 — LXCs report the host's loadavg). Per-host, sustained for:.
  • fleet-ssh-cert-expiring — < {{ obs_alert_cert_days_left }} (14) days left; host certs auto-renew on ctrl, so firing means the signer/tang1 is broken (runbook 26).
  • fleet-user-cert-expiring — control-plane SSH user certs < {{ obs_alert_user_cert_days_left }} (3) days (or missing): the almalinux control cert + optionally the claude-agent cert. These are YubiKey-signed and do NOT auto-renew, so firing = re-sign before automation/agent access lapses. Emitted by the common cert-expiry emitter via ssh_extra_certs on ctrl (runbooks 26/28). Config-drift itself is watched by the ctrl lab-drift-check timer (rb15 §6).
  • fleet-site-down — public-site blackbox probe_success == 0 (runbook 27).
  • fleet-zfs-degraded — node_zfs_zpool_state{degraded|faulted|…} (belt-and-suspenders to the ZED zedlets); fleet-zfs-capacity — any pool >85% full (zfs_pool_capacity_ratio); fleet-nfs-down — fileserver :2049 unreachable.
  • fleet-smart-health-failed (critical) / fleet-smart-prefail (warning) — SMART disk health, the early warning BEFORE a pool degrades. smart_device_health_passed == bool 0 (SATA overall-fail or NVMe critical-warning → replace the disk) and pre-fail counters (smart_current_pending_sectors/_offline_uncorrectable/_reallocated_sector_count > 0, NVMe _media_errors > 0 or _percentage_used > 90). Collected by the smart_metrics role on fileserver + bizon (textfile → Alloy) and by pve-zfs-push.sh for pve (runbook 27).

    pve's vmpool/rpool capacity + SMART isn't scraped (pve is MGMT; SERVERS→MGMT is blocked, so no obs→pve:9100 scrape). provision/pve-zfs-push.sh + a 5-min timer push zfs_pool_* and smart_* from pve to VM (MGMT→SERVERS), so pve's pools + disks show on the zfs.json dashboard + the capacity/SMART alerts. pve isn't in Ansible — re-deploy the script + timer (and smartmontools+python3) on a pve rebuild (runbook 24/25).

  • fleet-alert-heartbeat — always-firing vector(1), routed to the lab-heartbeat topic every ~15m; the ctrl dead-man's-switch alerts if that topic goes silent (catches "Grafana up but not delivering" — the 2026-06-28 blind spot). See runbook 27.

(deleteRules prunes two superseded rules — fleet-host-swap, fleet-disk-full. Keep this list in sync with roles/obs/templates/grafana-alerting.yml.j2 when rules change.)

⚠️ Grafana alerting is provisioned as code — an unparseable rule/template makes Grafana fail to start (a monitoring outage). Model every new rule on the working ones in grafana-alerting.yml.j2 and API-validate before deploying (runbook 27 §1).

Failure alerting (services/containers/VMs down) — runbook 27. The systemd-unit rule above + the alloy systemd collector + a dead-man's-switch on ctrl (Grafana can't alert on its own death — ctrl curls the obs health endpoints and ntfys directly) make up the "anything crashed" feature. See runbook 27 for the design + the deploy/validate/rollback checklist.

Subscribe a phone to https://ntfy.${domain}/lab-alerts (Tier-2, WG-only since 2026-07-02 — reachable over always-on WireGuard, not the public edge) to receive pushes. Add rules for backup-failed and Proton-tunnel health over time.

Deferred — firewall-conformance as a metric

The plan calls for a systemd timer running conformance.py --json from a MGMT vantage, writing a Prometheus textfile (fw_conformance_pass{check=…}) into Alloy's textfile-collector dir → a Grafana panel + alert on == 0. This needs a persistent MGMT host (mgmt-proxy) with a fleet SSH identity to probe every zone — gated on the SSH-CA rollout (runbook follow-up). Until then run the harness from the control node on demand (python3 tests/fw-conformance/conformance.py).