Runbook 25 — Encrypted `vmpool` + Tang self-heal unlock (NBDE)¶

Goal¶

Run the whole VM/LXC fleet from a fast SSD vmpool that is ZFS-native-encrypted (aes-256-gcm) and auto-unlocks at boot via Clevis + Tang (NBDE) against tang1 (runbook 24). The dataset key lives only in RAM, is reconstructed from a network handshake with tang1, and is unavailable if pve is stolen or taken off the LAN (fail-closed) — yet the unlock is self-healing (it retries until tang1 is reachable) and has a manual passphrase break-glass so a dead tang1 can never permanently brick the boot.

This is milestone 2 of the vmpool encryption project (milestone 1 = tang1, runbook 24).

Context¶

Why: the old fleet ran on a slow 3 TB spinning HDD (rpool/data); a full pve reboot caused a ~30-min boot storm (load 30+, 84 % iowait). Moving guests to the SSD vmpool fixes that and encrypts them at rest. The reclaimed disk is the Samsung 894 GiB NVMe (PCI 05:00.0) that used to hold the decommissioned next box. (The 1 TB media/home NVMe is a separate Crucial @ 10:00.0 passed through to the fileserver VM — not touched here; its own encryption is a later milestone.)
Tiering: vmpool/enc holds all guest system disks + the Nextcloud data zvol + speed- sensitive working data (Postgres, VictoriaMetrics/Loki, CI). Bulk media/home stay on the NFS fileserver. Keep vmpool lean → fast boots.
Single keyserver by decision: everything fail-closed depends on tang1 alone. The manual passphrase fallback (below) is mandatory; a second tang2 geo node was declined in favour of plaintext key escrow (rb24 decision record, 2026-07-10), so tang1 stays the sole keyserver and escrow — not a second unlock server — is the recovery path.

Prereqs - tang1 live + proven (runbook 24): 10.20.10.11, thumbprint 51kHWdOjLCSdlp_RT9xPULyDMOlqKLoYr0N3gCu18Lw, reachable from pve on :80. - The reclaim target disk is disposable and unmounted (verify by-id, no live data).

1. Reclaim the NVMe¶

Confirm the device by by-id (never nvmeXn1, which can renumber), tear down any old LVM/ZFS remnants, and wipe:

ls -l /dev/disk/by-id/ | grep -i samsung           # -> nvme-SAMSUNG_MZ1L2960HCJR-00A07_S665NN0TB11768
zpool status                                        # ensure no pool is on it (old 'superserver' gone)
vgchange -an ubuntu-vg 2>/dev/null || true
wipefs -a /dev/disk/by-id/nvme-SAMSUNG_MZ1L2960HCJR-00A07_S665NN0TB11768
sgdisk --zap-all /dev/disk/by-id/nvme-SAMSUNG_MZ1L2960HCJR-00A07_S665NN0TB11768
blkdiscard /dev/disk/by-id/nvme-SAMSUNG_MZ1L2960HCJR-00A07_S665NN0TB11768   # clean SSD

2. Create the pool + encrypted dataset¶

zpool create -o ashift=12 -O compression=zstd -O atime=off \
  vmpool /dev/disk/by-id/nvme-SAMSUNG_MZ1L2960HCJR-00A07_S665NN0TB11768

Generate + escrow the raw key BEFORE any data rides on it (§3), then:

# feed the 32-byte raw key on stdin (keylocation=prompt reads it interactively)
zfs create -o encryption=aes-256-gcm -o keyformat=raw -o keylocation=prompt vmpool/enc
zfs create vmpool/enc/vmdata

Add it as Proxmox storage — sparse 1 so thin guest disks don't over-reserve (the fleet is ~163 GiB actual but ~1.1 TiB provisioned):

pvesm add zfspool vmpool-enc --pool vmpool/enc/vmdata --content images,rootdir --sparse 1

3. Escrow the key three ways (do not skip — this is the DR backbone)¶

The raw key is the crown jewel: lose it and tang1 and the pool is unrecoverable. 1. Ansible vault — vault_zfs_pve_vmpool_enc_key in ansible/group_vars/all/vmpool-enc-key.vault.yml. The *.vault.yml suffix is gitignored (the repo is secret-free — encrypted-or-not, no key file is committed), so this is a local encrypted copy, not an in-git one. 2. KeePass lab.kdbx — the base64 of the raw key (primary human break-glass). 3. Offline — a copy off the lab entirely.

Round-trip-verify all three (sha256 of the decoded key == the live key) before putting data on it.

4. Bind the key to Tang (Clevis)¶

Wrap the raw key with tang1 into a JWE that is useless without tang1 — safe to leave on the unencrypted rpool:

clevis encrypt tang '{"url":"http://10.20.10.11"}' < /path/to/rawkey > /etc/zfs/vmpool-enc.jwe
clevis decrypt < /etc/zfs/vmpool-enc.jwe | sha256sum     # == sha256 of the raw key
chmod 600 /etc/zfs/vmpool-enc.jwe

vmpool auto-imports at boot (it's in zpool.cache); only the dataset key needs unlocking.

5. Self-healing boot unlock (timer-driven, fail-closed)¶

A single boot-time one-shot is brittle: at boot network-online.target can fire before the MGMT path actually passes traffic (esp. under a boot storm), so the first unlock races and gives up. Instead a timer drives an idempotent unlock that retries forever — so tang1 returning at any time (rebooted, briefly unplugged, powered on later) unlocks the pool and starts its guests, while staying fail-closed (off-LAN → tang1 never reachable → never unlocks).

/usr/local/sbin/vmpool-unlock.sh (mode 700):

#!/usr/bin/env bash
# Idempotent NBDE unlock for vmpool/enc, driven by vmpool-unlock.timer + at boot.
# Self-healing; fail-closed. Break-glass: base64 -d <<<"<KeePass key>" | zfs load-key vmpool/enc
set -uo pipefail
JWE=/etc/zfs/vmpool-enc.jwe; TANG=http://10.20.10.11
[ -f "$JWE" ] || exit 0
[ "$(zfs get -H -o value keystatus vmpool/enc 2>/dev/null)" = available ] && exit 0   # no-op if unlocked
curl -fsS -m3 "$TANG/adv" >/dev/null 2>&1 || { echo "tang1 unreachable - staying locked, will retry"; exit 0; }
clevis decrypt < "$JWE" | zfs load-key vmpool/enc
zfs mount vmpool/enc 2>/dev/null || true
zfs mount vmpool/enc/vmdata 2>/dev/null || true
systemctl --no-block start pve-guests.service 2>/dev/null || true   # start guests once the pool is up
echo "vmpool/enc unlocked via tang1 + guests kicked"

/etc/systemd/system/vmpool-unlock.service (one-shot; not ordered Before=pve-guests — so a slow/absent tang1 never blocks the rest of boot):

[Unit]
Description=Unlock vmpool/enc via Clevis+Tang (NBDE; fail-closed, self-healing)
After=network-online.target zfs-import.target
Wants=network-online.target
ConditionPathExists=/etc/zfs/vmpool-enc.jwe
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/vmpool-unlock.sh

/etc/systemd/system/vmpool-unlock.service.d/nolimit.conf (a one-shot can't use Restart=; the timer drives retries, so disable the start rate-limit that would otherwise block frequent triggers):

[Unit]
StartLimitIntervalSec=0

/etc/systemd/system/vmpool-unlock.timer:

[Unit]
Description=Retry vmpool/enc NBDE unlock every 30s until tang1 reachable (self-heal)
[Timer]
OnBootSec=5
OnCalendar=*:*:0/30
AccuracySec=2s
Persistent=true
[Install]
WantedBy=timers.target

Enable the timer (not the service directly):

systemctl daemon-reload
systemctl enable --now vmpool-unlock.timer

Gotcha: a stale RemainAfterExit=yes instance stuck active will stop the timer re-arming (systemctl list-timers shows NEXT: -). If that happens: systemctl stop vmpool-unlock.service && systemctl reset-failed vmpool-unlock.{service,timer} && systemctl restart vmpool-unlock.timer.

6. Prove-out (acceptance)¶

# auto-unlock works
systemctl start vmpool-unlock.service; zfs get -H -o value keystatus vmpool/enc   # available

# self-heal: lock it, let the timer recover within a tick
zfs unmount vmpool/enc/vmdata; zfs unmount vmpool/enc; zfs unload-key vmpool/enc
sleep 35; zfs get -H -o value keystatus vmpool/enc                                # available again

# fail-closed: tang down -> stays locked (the whole point)
ssh tangadm@10.20.10.11 sudo systemctl stop tangd.socket
zfs unload-key vmpool/enc; sleep 35; zfs get -H -o value keystatus vmpool/enc     # unavailable
ssh tangadm@10.20.10.11 sudo systemctl start tangd.socket                         # then it self-heals

# manual break-glass (no tang at all): load the escrowed key by hand
base64 -d <<<"<base64 key from KeePass>" | zfs load-key vmpool/enc

Then reboot pve and confirm vmpool/enc is available + mounted on its own, and guests start.

7. Provision guests onto `vmpool-enc`¶

provision/lab-provision.sh targets STORAGE=vmpool-enc by default and supports both guest types via the optional TYPE column in provision/vms.conf (vm | lxc). See runbook 01 for the full provisioning flow and the VM-vs-LXC taxonomy; in short: - VM (qm clone the AlmaLinux 9 template): own-kernel / passthrough / appliance / vTPM guests — OPNsense, Home Assistant, tvheadend, fileserver, FreeIPA, Nextcloud, ctrl, and runner (runs arbitrary CI builds via rootful podman → needs a VM kernel boundary, not nested LXC). - LXC (pct create from the AlmaLinux 9 LXC template, unprivileged, nesting=1): the lighter native single-app guests (Caddy proxies, bind, jump) and the podman/docker quadlet hosts. The stock CT lacks the sshd / almalinux user / CA-trust the VM cloud-image shipped, so up_one_lxc reproduces them (openssh-server + python3, the almalinux user, user-CA + TrustedUserCAKeys, pre-seeded auth_principals so ctrl's control cert connects on the first run, the bootstrap key, NOPASSWD sudo). After that an LXC is identical to a VM from Ansible's view. (common skips the VM-only qemu-guest-agent on containers; size LXCs ≥ ~1 GiB RAM so dnf transactions don't OOM.)

Migrate an existing VM's disks onto the encrypted pool with:

qm disk move <vmid> <disk> vmpool-enc --delete      # live for a running VM

Disaster recovery¶

Scenario	What happens	Recovery
`tang1` briefly down (reboot/unplug)	pool stays locked; guests on it don't start	Nothing — the timer self-heals within 30 s of `tang1` returning, then starts the guests.
`tang1` dead / lost	pool won't auto-unlock	Break-glass now: `base64 -d <<<"<KeePass key>" \\| zfs load-key vmpool/enc` → pool up immediately. Rebuild `tang1`: flash a new Pi, restore `/var/lib/tang/*.jwk` from KeePass+offline, run `provision/tang1-setup.sh` — the existing `vmpool-enc.jwe` binding works unchanged (same keys).
pve stolen / off-LAN	`tang1` unreachable → key never reconstructed	By design — the data stays encrypted. (Correct posture, not a failure.)
Lost the raw key too	unrecoverable	Prevented by the three-way escrow (§3). Verify it periodically.
`vmpool-enc.jwe` corrupted/lost	auto-unlock breaks	Re-create it from the escrowed raw key: `clevis encrypt tang '{"url":"http://10.20.10.11"}' < rawkey > /etc/zfs/vmpool-enc.jwe`.

Key locations (single source of truth): raw key → KeePass lab.kdbx (primary) + offline + the gitignored vmpool-enc-key.vault.yml (vault_zfs_pve_vmpool_enc_key). Tang keys → /var/lib/tang/*.jwk on tang1, backed up in KeePass + offline (runbook 24 §6).

Validation¶

[ ] vmpool/enc is aes-256-gcm; vmpool-enc Proxmox storage is sparse.
[ ] Raw key escrowed + round-trip-verified in all three places.
[ ] clevis decrypt < /etc/zfs/vmpool-enc.jwe reproduces the key (== sha256).
[ ] vmpool-unlock.timer enabled; systemctl list-timers shows a future NEXT.
[ ] Prove-out passes: auto-unlock, self-heal, fail-closed, manual break-glass.
[ ] pve reboots with vmpool/enc unlocking on its own + guests starting.
[ ] A guest (VM and LXC) provisions onto vmpool-enc and its disk is encrypted.

Next milestone¶

Encrypt the fileserver/supernas pool (media + home) the same way — ZFS-native + Tang, the unlock running inside the fileserver VM (it holds live data → zfs send to a new encrypted dataset). Off-box replication is already live but took a different shape than the raw-zfs send -w-to-supernas sketch here: non-raw syncoid replication direct to bizon, and the HDD (rpool) backup tier has already been retired (see the runbook 40 Redesign, 2026-07-25). A second tang2 keyserver was declined in favour of plaintext key escrow (rb24 decision record) — fail-closed hinges on tang1 by choice, with the passphrase + escrowed raw key as break-glass.

References¶

NBDE / Clevis + Tang: clevis(1), clevis-encrypt-tang(1), zfs-load-key(8); tang1 = runbook 24.
Provisioning + VM/LXC taxonomy: 01-provision.md; design: ../CLAUDE.md.

Runbook 25 — Encrypted vmpool + Tang self-heal unlock (NBDE)¶