Runbook 24 — `tang1` NBDE/Tang keyserver¶

Goal¶

Stand up tang1, a single-function Tang keyserver, so pve's (future) encrypted ZFS vmpool auto-unlocks at boot via Clevis + Tang (NBDE — Network-Bound Disk Encryption): the decryption key is reconstructed from a network handshake with tang1, lives only in RAM, and is unavailable if the box is stolen or taken off the LAN (fail-closed). This is milestone 1 of the vmpool encryption project — nothing downstream (create vmpool, migrate VMs, raw zfs send replication) proceeds until tang1 is proven here.

Context¶

What tang1 is: the lab's first physical, non-VM infra host — a Raspberry Pi 3 running Raspberry Pi OS Trixie (Debian 13) arm64 Lite, deliberately minimal (only sshd + tangd).
Why MGMT, not SERVERS (VLAN 20): OPNsense runs on pve and does the inter-VLAN routing, so a keyserver behind a VLAN that only exists after OPNsense boots would be a circular boot dependency. MGMT routing is up early, and pve itself is on MGMT (10.20.10.5), so the unlock path pve → tang1:80 stays inside one L2 segment.
Why an appliance, not a fleet Ansible host: the end-state is a read-only overlay (immutable root), which fights recurring config management. So tang1 is treated like OPNsense/HAOS — codified by a one-shot script (provision/tang1-setup.sh) + this runbook, and kept out of site.yml / all_linux. The only Ansible touch is an external blackbox probe from ctrl.
Read-only overlay is DEFERRED to the final milestone (after the whole vmpool migration is proven). tang1 stays writable during buildout so it's easy to iterate.

Prereqs - Pi 3 flashed with the pinned image (~/tang1-flash/raspios-lite-arm64.img.xz, verified against image.sha256) and on MGMT ethernet. - A vantage with MGMT access (direct ethernet to MGMT, or laptop on the MGMT subnet — note WiFi that lands on 10.1.10.x is firewalled off MGMT). - The bootstrap SSH key ~/tang1-flash/bootstrap_ed25519 (its pubkey was baked into the flash). - OPNsense root API key (KeePass lab.kdbx) for the DHCP static mapping.

Facts to confirm for your environment - tang1 MAC: b8:27:eb:98:cb:c3 (Pi Foundation OUI) → static lease 10.20.10.11. - Tang port + key dir on your package: systemctl cat tangd.socket and dpkg -L tang (this runbook assumes Debian's defaults: port 80, keys in /var/lib/tang).

1. Flash the SD (Raspberry Pi Imager — user-run)¶

Re-image so SSH + the tangadm user + the bootstrap key are present (a plain dd of the raw image leaves SSH disabled — that was the original "pings but port 22 closed" symptom).

In Raspberry Pi Imager → OS customization: - General: hostname tang1; username tangadm; set locale/timezone; Wireless LAN OFF. - Services: Enable SSH → "Allow public-key authentication only"; paste the contents of ~/tang1-flash/bootstrap_ed25519.pub. - OS source: Use custom → the verified local raspios-lite-arm64.img.xz. - Storage: the correct SD device (confirm size/model; on the build laptop it was the 16 GB "SC16G" at /dev/mmcblk0 — not the system NVMe).

Boot on MGMT. First connect (pin the host key):

ssh -i ~/tang1-flash/bootstrap_ed25519 tangadm@10.20.10.100   # DHCP IP on first boot

2. Static IP `10.20.10.11` (boot-critical — NOT DHCP-dependent)¶

Why static, not DHCP: tang1 is on pve's boot-time unlock path, but DHCP is served by Kea on OPNsense, which is a VM on pve. If tang1 waited for DHCP at boot, a cold start would deadlock (pve needs tang1 → tang1 needs OPNsense → OPNsense is a locked-away VM on pve). A static IP breaks it: pve and tang1 share the MGMT L2 segment, so pve reaches .11 directly with every VM down.

Set it in NetworkManager (Pi OS Trixie default):

nmcli con mod "Wired connection 1" ipv4.method manual \
  ipv4.addresses 10.20.10.11/24 ipv4.gateway 10.20.10.1 ipv4.dns 10.20.10.1 ipv4.dns-search ${domain}
nmcli con up "Wired connection 1"   # run detached (systemd-run --no-block) — it bounces the link

Also keep a Kea reservation (MAC b8:27:eb:98:cb:c3 → .11) as a dormant backup via the OPNsense API (key:secret from lab.kdbx; POST /api/kea/dhcpv4/add_reservation then /api/kea/service/reconfigure): with a static IP tang1 never asks DHCP, but if it ever reverts (e.g. a re-flash) the reservation still lands it on .11. Reboot and confirm ipv4.method=manual and the address shows global (not dynamic). Re-pin the host key for .11.

3. Provision (the one-shot script)¶

From a repo checkout on the laptop/ctrl, run the idempotent setup over SSH:

ssh -i ~/tang1-flash/bootstrap_ed25519 tangadm@10.20.10.11 \
    'sudo bash -s' < provision/tang1-setup.sh

The script (see its header for the full list) installs tang+clevis+firewalld, ensures the Tang keys exist, enables tangd.socket, sets firewalld to default-deny inbound except 22/tcp + 80/tcp, installs fleet SSH-CA trust (cert principals admin / admin-mgmt → tangadm) + sshd hardening, and prints a summary (thumbprint, listeners, firewall). It is safe to re-run.

After this, the CA cert path works (ssh tangadm@10.20.10.11 from a host with a current admin-mgmt cert), and the bootstrap key is demoted to break-glass (it still works, because AuthorizedPrincipalsFile governs only certificate principals, not plain authorized_keys).

4. Prove-out (milestone-1 acceptance)¶

Round-trip (on any clevis client — pve is the real one; tang1 itself works for a smoke test):

echo -n "nbde-smoke-test" | clevis encrypt tang '{"url":"http://10.20.10.11"}' > /tmp/c.jwe
# first use prompts to trust the advertised thumbprint — confirm it matches `tang-show-keys 80`
clevis decrypt < /tmp/c.jwe          # -> nbde-smoke-test

Fail-closed (this is the whole point — no Tang, no unlock):

sudo systemctl stop tangd.socket
clevis decrypt < /tmp/c.jwe ; echo "exit=$?"   # MUST fail (non-zero, no plaintext)
sudo systemctl start tangd.socket
clevis decrypt < /tmp/c.jwe                     # succeeds again

Boot-path reachability — from pve (10.20.10.5):

curl -fsS http://10.20.10.11/adv >/dev/null && echo "pve -> tang1:80 OK"

5. Monitoring (external blackbox probe)¶

ctrl (the MGMT prober) already runs prometheus.exporter.blackbox. The target is declared in ansible/group_vars/control/main.yml (alloy_blackbox_targets):

  - { name: tang1-nbde, address: "10.20.10.11:80" }

Apply it:

ansible-playbook playbooks/site.yml --tags alloy --limit ctrl

Confirm probe_success{name="tang1-nbde"} = 1 in VictoriaMetrics / the Boot & Service Readiness Grafana dashboard. (No on-box metrics agent — keeps the Pi minimal.)

Central access audit (logs). tang1-setup.sh §5b forwards tang1's syslog to the obs Alloy syslog-gateway (10.20.20.50:514 → Loki). tang is the chokepoint for every NBDE decrypt in the lab (pve vmpool unlock, ctrl control-key load, host-CA signing — runbook 26), and each recovery is a socket-activated tangd@<…-PEERIP:port>.service instance, so the peer IP + operation of every key-unlock are centrally queryable — independent of any app-level alert (catches a rogue clevis decrypt). Query in Grafana Explore: {job="syslog"} |= "tangd". (No alert on raw /rec — legitimate unlocks are frequent.) This is syslog forwarding (rsyslog), not a full agent — the box stays single-function. Verify after provisioning that the installed tangd logs the operation/key-id at the verbosity you want.

⚠️ Format gotcha (found 2026-07-12, runbook 42 C5 — the shipping was dead in practice). The obs gateway (loki.source.syslog) accepts RFC5424 with octet-counted TCP framing ONLY; legacy RFC3164 (rsyslog's default) is rejected with expecting a version value and ships nothing, silently. §5b now sets template="RSYSLOG_SyslogProtocol23Format" + TCP_Framing="octet-counted" — keep both if you touch the forward. Applied to the live tang1 the same day (its build predated §5b entirely: rsyslog wasn't even installed); verified flowing ({job="syslog"} in Loki, host/app labels from the relabel added to the gateway).

6. Key escrow & credential hygiene (do not skip)¶

The Tang signing/exchange keys (/var/lib/tang/*.jwk) are crown jewels: once the read-only overlay lands, losing them = unrecoverable unlock. - Back up /var/lib/tang/*.jwk to KeePass lab.kdbx + an offline copy. - Record in KeePass: the bootstrap_ed25519 private key, the tangadm password (if set), and the tang1 host-key fingerprint.

Validation¶

[ ] ssh tangadm@10.20.10.11 works via a CA cert (bootstrap key not required).
[ ] ss -lntup on tang1 shows only :22 and :80 listeners (nothing else).
[ ] firewall-cmd --list-ports = 22/tcp 80/tcp; rules + tangd survive a reboot.
[ ] clevis round-trip returns the plaintext; fail-closed test fails with tangd down and recovers when it's back.
[ ] pve (10.20.10.5) reaches http://10.20.10.11/adv.
[ ] probe_success{name="tang1-nbde"} = 1 in Grafana.
[ ] Tang keys + bootstrap key + host fingerprint recorded in KeePass + offline.

Next milestone¶

Done since: the Samsung 960G NVMe was reclaimed, vmpool → vmpool/enc (aes-256-gcm) → vmpool/enc/vmdata built + Tang-bound + VMs migrated (runbook 25). Off-box replication is also live, but took a different shape than the raw-zfs send -w-to-supernas sketch here: non-raw syncoid replication direct to bizon, and the on-box HDD (rpool) backup tier has been retired — see the runbook 40 Redesign (2026-07-25). A manual passphrase fallback stays on the dataset so a dead tang1 can never brick pve's boot; tang1 remains the sole keyserver by decision — a second Tang was declined in favour of key escrow (decision record below). See the vmpool-encryption-migration memory.

tang2 — decision record (2026-07-10): key escrow instead of a second Tang¶

A tang-on-bizon bridge (sss t:1 for the ctrl-side pins) was built and retired the same day — the operator chose plaintext key escrow in KeePass over a second unlock server (simpler; one fewer trust-bearing service; same recovery property). What the analysis established, kept for the future dedicated tang2 Pi at 10.20.10.12: - Only an MGMT-L2 device can serve pve's boot-unlock. At pve boot OPNsense (a VM on pve) is down, so no inter-VLAN routing exists — a SERVERS-zone Tang can never cover that path. The vmpool pin stays tang1-only; its break-glass is the passphrase + escrowed raw key (rb25), which is exactly the model extended here. - Escrow model for the ctrl-side sealed material: every .jwe dies with tang1's /var/lib/tang/*.jwk, so each needs a tang-independent recovery: instance material — plaintext tar in KeePass + offline (rb33). host-CA key — escrow its plaintext in KeePass (decrypt the .jwe on ctrl while tang1 is healthy: clevis decrypt < /etc/lab/host-ca.jwe > staged; attach to KeePass; shred staged); until then a dead tang1 freezes autosign and the YubiKey host-CA break-glass (rb26) covers signing. agent key — no escrow needed: fail-closed is the design; the operator re-issues.

References¶

NBDE / Clevis + Tang: clevis(1), clevis-encrypt-tang(1), tang(8), tang-show-keys(1)
Design: ../CLAUDE.md (Platform/provisioning & backups); host-firewall model: 23-host-firewall.md

Runbook 24 — tang1 NBDE/Tang keyserver¶