46 — Rotating the lab's keys (dependency-ordered)¶

Status: written 2026-07-18. The hygiene-rotation program: what can be rotated, in what order, and what cannot.

Is this even possible?¶

Yes — but not as one operation, and not in an arbitrary order. Three honest caveats up front, because they shape everything below:

There is no "rotate everything" button, and building one would be a mistake. The lab's secrets form a dependency graph. Several are required in order to rotate others. Rotating in the wrong order destroys the credential you need for the next step.
Some things are restore-not-rotate. tang1's keys are the clearest case: rotating them invalidates seven Clevis-sealed blobs simultaneously and irreversibly, one of which unlocks the pool every VM lives on. The design deliberately escrows them instead.
A few cannot be rotated in place at all. The YubiKey CA keys are generated on-device and non-exportable — a "rotation" is really provision a new CA, publish it, distrust the old, which is rb45 §6/§7.

Everything else rotates cleanly, provided you go leaves-first and make-before-break.

This runbook is hygiene, not incident response. It is ordered for safety — verify between steps, no time pressure. A suspected-compromise rotation has the opposite ordering (blast radius first, accept breakage) and is not written yet.

Related: 45 the tokens · 20 the SSH CA · 24 Tang · 25 vmpool · 43 the secrets store · 15 ctrl + the TPM seal · 47 the lab.kdbx presence-unlock.

1 — The five roots¶

Nothing in the lab unlocks these. Everything else is derived from them.

Root	Lives	Backed by
YubiKey PIV 9c/9d CA keys	on-chip, non-exportable	nothing — by design (rb45 §1.4)
YubiKey FIDO2 `lab.kdbx`-arm credentials (rb47)	on-chip hmac-secret, per-token non-exportable; the `identity-{A,B}.txt` files are required to use them	`identity-{A,B}.txt` + `master.age` escrowed off `atomic` (rb47)
`lab.kdbx` master password	the operator	offline copy in the safe; re-encrypted into `master.age` (rb47)
*tang1 `/var/lib/tang/.jwk`**	the tang1 Pi	escrowed into `lab.kdbx` + offline (rb24)
ctrl's vTPM	Proxmox vTPM state on pve	nothing — non-migratable, dies with the VM

The graph runs roughly:

YubiKey / KeePass  →  tang1  →  the .jwe blobs  →  keys  →  vault  →  repos

Rotate downstream-first, so every upstream break-glass stays intact while you work.

2 — The load-bearing edges¶

Three facts determine the whole ordering. Internalise these and the rest follows.

The KeePass password alone unlocks the entire Ansible vault. lab.kdbx holds an Ansible Vault entry whose password is the vault passphrase; scripts/secrets.sh and scripts/vault-pass.sh both read it. So lab.kdbx is not "one more secret store" — it is the recovery root for Tier 2 as well.

Tang gates the load, not the session. load-control-key.sh notes that an already- loaded key survives tang1 going down. You can therefore break a .jwe, not notice, and discover it at the next ssh-add -D or reboot. Always validate a key rotation in a fresh agent.

vault-pass.sh's plaintext tiers win over the TPM. $LAB_VAULT_PASS and $LAB_VAULT_PASS_FILE are checked before the TPM-sealed credential. A stale variable in the environment silently shadows the real path — and will keep returning the old password after a rotation.

3 — Circular dependencies (and how each is broken)¶

These are real, documented, and already have prescribed breaks. Know them before rotating.

Loop	The break
lab-secrets ⇄ deploy key — the repo holds `lab.kdbx`; `lab.kdbx` escrows the deploy key; the deploy key fetches the repo	out-of-band escrow of the key (rb43)
lab.kdbx ⇄ vault passphrase ⇄ TPM cred — the DB holds the passphrase you seal; the DB arrives via a sync needing that vault	manual `scp` of `lab.kdbx` on rebuild (rb15)
tang1 ⇄ lab.kdbx — tang1's `.jwk` are escrowed in the DB; the DB may arrive via a tang-sealed key	escrow to offline media too, not only KeePass
pve/vmpool ⇄ tang1 ⇄ OPNsense — vmpool needs tang1; tang1 on DHCP would need OPNsense; OPNsense is a VM on vmpool	tang1 has a static IP on MGMT (rb24)
ctrl ⇄ vault ⇄ ctrl — the converge installs the vault-cred plumbing, but needs the vault	manual bootstrap (rb15)
host-CA self-update ⇄ signing key — the updater verifies `main` against `allowed_signers` from the tree it is about to replace	add the new signer line first, in a commit signed by the old key

Note ctrl's vTPM lives on pve, on the pool tang1 unlocks — so the TPM path itself sits behind the tang gate at cold start. The TPM is not an independent recovery route from a vmpool failure.

4 — Phase 0: pre-flight (do not skip)¶

Rotation is safe only if the escrows you are about to rely on actually work. Verify, don't assume:

All three vmpool escrows round-trip (rb25) — vault copy, KeePass copy, offline copy.
clevis decrypt succeeds for every .jwe — id_ctrl, id_agent, id_agent_task, the host-CA key, instance material, id_lab_secrets (if sealed), and the vmpool binding.
The host-CA raw key is escrowed. rb24 still reads as though this is outstanding; confirm against lab.kdbx before trusting either statement. If it is genuinely missing, do it now — while tang1 is healthy — because a dead tang1 otherwise freezes host-cert renewal down to the YubiKey break-glass.
systemd-creds decrypt --name=lab-vault /etc/lab/vault.cred - succeeds in a fresh almalinux login shell. Not under sudo — sudo strips LAB_VAULT_CRED.
Out-of-band copies of id_lab_secrets and lab.kdbx on removable media.
Clear the environment — no stale LAB_VAULT_PASS / LAB_VAULT_PASS_FILE (§2).

5 — The rotation order¶

Seven phases, leaves first. Complete and verify each before starting the next.

Phase 1 — pure leaves (nothing depends on them)¶

Cheapest and safest; do these first so a mistake costs nothing.

Forgejo Actions secrets, registry push tokens, every ntfy token.
The CI vault password — a wholly separate scope (ansible/ci/), so rotate it before the admin vault to rehearse the motion cheaply. Update the runner's LAB_VAULT_PASS.
The runner registration token — ephemeral; delete the runner state and re-register.
id_agent / id_agent_task — the cheapest keys in the lab. New keypair, new passphrase, new .jwe, new KeePass entry, then re-sign with the YubiKey (rb28). No escrow needed; fail-closed is the design.

Phase 2 — the Ansible vault password¶

Order within this phase is what matters:

Update the Ansible Vault entry in lab.kdbx first.
ansible-vault rekey every *.vault.yml in the lab-secrets clone; commit and push.
Re-seal both /etc/lab/vault.cred and /etc/lab/vault-agent.cred — same plaintext, two seals. Forgetting the agent one half-breaks the lab silently.
Verify a converge from a fresh login shell.

Phase 3 — the `lab-secrets` deploy key¶

Strict make-before-break: generate the new key → add its pubkey in Forgejo while the old one still works → prove a pull with the new key → remove the old → re-escrow to KeePass → only then re-seal its .pass.jwe, and only if the Tang-aware loader is already deployed.

Phase 4 — `id_ctrl`¶

Prefer a passphrase-only rotation: the pubkey and its cert are unchanged, so nothing downstream moves. A full keypair rotation additionally needs a new YubiKey-signed cert and touches the fleet's auth_principals — a much larger change.

Validate in a fresh agent (§2).

Phase 5 — the online host CA (rb26)¶

Make-before-break, and the window is set by cert lifetime:

Generate the new CA key in /dev/shm; seal it to tang1.
Add its pubkey as an additional line in ssh_host_ca.pub — alongside the existing online CA and both YubiKey 9d lines. Trust is additive.
Converge --tags sshca fleet-wide and push to the appliances by hand.
Let host certs re-sign under the new key and roll through their validity window.
Only then remove the old line.

Never remove the old CA pubkey before every host has converged with the new one.

Phase 6 — tang1's `.jwk` (avoid; only if genuinely compromised)¶

This invalidates all seven .jwe blobs at once. The design treats tang keys as restore-not-rotate — a fresh Pi with the same .jwk files makes every existing blob work unchanged, which is precisely why they are escrowed rather than rotated.

If you must:

Break-glass load everything into RAM first — ssh-add the keys, zfs load-key the pool — so a mistake is not immediately fatal.
Rotate the tang keys.
Immediately re-encrypt all seven blobs from their escrowed plaintexts.
Re-escrow the new .jwk to KeePass and offline.
Reboot pve to prove vmpool self-heals unattended. An untested tang rotation is not a completed one.

Phase 7 — the roots¶

KeePass master password — change it, re-cut master.age against the new value so the rb47 presence-unlock keeps working, verify re-open (typed) and an arm-kdbx from both tokens, then commit and push the new lab.kdbx to lab-secrets. If you skip the push, the next converge overwrites your rotated DB with the old one (§6, L9). lab.kdbx is password-only — there is no OTP factor to rekey (rb45 §3.5).
The rb47 FIDO2 arm credentials — per-token and non-exportable, so a "rotation" is generate a fresh identity on the token, then re-cut master.age to the new recipient set (rb47 §Build) — never a shared-secret reprogram. Additive: master.age can carry old and new recipients during the changeover.
The YubiKey CA keys — one token at a time; the fleet trusts both, so the other keeps working throughout. rb45 §6/§7.

6 — Lockout catalogue¶

Each of these is an ordering mistake, not a command mistake. They are listed worst-first.

L1 — Rotate tang1's keys before re-encrypting the blobs, then reboot pve. The worst ordering in the lab. All seven blobs die at once. pve cannot unlock vmpool, so OPNsense, ctrl and forgejo never boot; ctrl's vTPM is on that pool, so the vault credential is also unreachable; and lab.kdbx — which holds the recovery for all of it — lives on ctrl. Terminal unless the offline copies are good. This is why Phase 0 exists.

L2 — Re-seal the TPM credential before updating the Ansible Vault KeePass entry. The new passphrase then exists only in that terminal's scrollback. Every vault-pass.sh fallback tier reads KeePass, so a stale entry returns the old password everywhere.

L3 — Delete the old deploy key before proving the new one pulls. Survivable only via the cached clone; on a fresh ctrl there is no cache and the sync is fatal. Combined with L2 it is unrecoverable.

L4 — Tang-seal id_lab_secrets before deploying the Tang-aware loader. The old loader uses the key directly, now passphrase-protected, with no interactive prompt available — it runs as a root oneshot.

L5 — Remove the old host-CA pubkey before the fleet converges. Every host's certificate becomes untrusted at once, and the autosigner cheerfully re-signs with a CA nobody trusts. Recoverable, but if id_ctrl was rotated in the same window and tang1 is down, you have no fleet reach to fix it.

L6 — Rotate the lab.kdbx master password without re-cutting master.age. The DB is password-only; the rb47 presence-unlock (arm-kdbx) ships the password out of master.age. Change the password and forget to re-encrypt master.age to it → the agent's arm path serves the old password and every kdbx-cli read fails, while a typed break-glass open still works (masking the breakage until you next rely on the agent). Re-cut master.age in the same sitting (rb45 §7.6, rb47). (The historical L6 — reprogramming one token's OTP slot but not the other, which permanently locked out the un-reprogrammed token under the old AND-combined composite key — no longer applies: the OTP factor is retired and there is no "key A or key B" composite to desync.)

L7 — Rebuild ctrl without preserving tpmstate0. Both sealed credentials die. Recoverable only from the KeePass entry — so anyone who also hit L2 is now unrecoverable. Note the seal has no PCR policy, so ordinary kernel and firmware updates are safe; it is specifically vTPM state loss that kills it.

L8 — Rotate the operator signing key and populate allowed_signers in one step. The host-CA updater verifies against the allowed_signers in the tree it is replacing. Land a commit signed by the new key while the tree lists only the old → verification fails → it stays pinned, quietly, forever. Fail-safe, so signing continues and you discover it weeks later. Add the new signer line first, in a commit signed by the old key.

L9 — Let a stale clone overwrite a freshly rotated lab.kdbx. The loader unconditionally installs lab.kdbx over the checkout copy on every boot and converge. Rotate locally, forget to push, reboot → the old DB is restored over your rotation. Not fatal on its own, but it silently reverts a rotation you believe succeeded.

L10 — Rotate id_ctrl while relying on an already-loaded agent key. The old key persists in the running agent, so everything looks fine until the next ssh-add -D or reboot — at which point fleet automation is dead. Validate in a fresh agent.

7 — Special cases¶

7.1 — Cannot be rotated in place¶

Thing	Why	Instead
YubiKey 9c/9d CA keys	generated on-device, non-exportable	provision a new CA, publish, distrust the old — rb45 §6/§7
tang1 `.jwk` (in practice)	invalidates seven blobs at once	escrow and restore; treat rotation as a compromise-only action
Proton WireGuard keys	vendor-issued	request fresh configs (rb05)
FreeIPA CA / DM / admin	no documented procedure anywhere in the repo; renewal is notoriously fragile	plan separately; FreeIPA is `t4_critical`, so it goes last in any phase order

7.2 — Re-sealing is not re-keying¶

Worth separating, because it saves an enormous amount of work: the online host CA's key can be re-sealed to a rebuilt tang1 by regenerating its .jwe from the escrowed raw key. The public key is unchanged, so no fleet re-trust is needed (rb26).

Only a genuinely new CA key triggers the additive-trust dance in Phase 5. Ask which one you actually need before starting.

7.3 — DNSSEC: a single CSK, and currently unanchored¶

The zones use a single CSK (alg-13, dnssec-policy default, inline-signing on the hidden primary) — there is no KSK/ZSK split to roll independently, and dnssec-policy default schedules no rollovers at all.

More importantly: the zones are signed but not yet anchored — no DS at any parent (rb17). So today a key change is consequence-free. Once a DS is published it becomes total-outage class, and two invariants bind:

The DS must be regenerated and published before the old key retires — make-before-break.
Never restore zones from a backup predating a key change.

Re-take the DNSSEC escrow after any key rollover or zone add (rb09). Note the repo documents no TTL or propagation arithmetic; the only timing figures are the 5-day RRSIG runway alert and a 48-hour DS soak.

7.4 — Secrets with coupled or hazardous rotation¶

Secret	Hazard
Authelia `storage_encryption_key`	not freely rotatable — it decrypts stored data at rest, including enrolled TOTP secrets. Rotating it without a migration discards them. Becomes load-bearing the moment 2FA is enabled (rb45 §5)
OIDC client secrets	provider and client must converge together, or that login breaks (rb43)
TSIG keys	BIND supports multiple named keys, so this is naturally make-before-break: add key2 → converge the primary → switch each consumer → remove key1. `vault_bind_tsig_acme` has five consumers; enumerate them all first
`vault_ldap_bind_password`	one `svc-ldap` credential shared by Authelia and Jellyfin — both converge together
origin-caddy WG identity	deliberately pinned, not rotated — it is escrowed precisely so a rebuild keeps the same identity. A fresh key silently breaks every edge tunnel (this cost hours once). Rotating means a coordinated two-sided change across origin and every edge
ntfy topic URLs	three steps, and the last is human: converge, then re-subscribe the phone. Converging without re-subscribing opens a silent missed-alert window
Seeded admin passwords in `lab.kdbx`	point-in-time copies of vault values — rotating the vault value does not update KeePass. Update both or the break-glass record goes stale

8 — Known gaps (as of writing)¶

Surfaced while mapping the graph; each is independent of any rotation you actually perform:

scripts/allowed_signers ships empty (comments only), so the commit-signature gate on the host-CA self-updater is inert — verification always fails, the signer stays pinned and keeps signing. Fail-safe, but it is not providing the integrity check it appears to. It becomes load-bearing the moment real signers land — see L8.
Most secrets have no documented rotation procedure, only a generation step. The exceptions: host certs (rotate-host-certs.sh), the lab-secrets deploy key, OIDC client secrets, the DNSSEC escrow, YubiKey PINs, and fleet root passwords.
Shared-secret fan-outs must be rotated atomically across all consumers: vault_bind_tsig_acme (five Caddy planes) and vault_ldap_bind_password (one svc-ldap credential shared by Authelia and Jellyfin).
rb24's host-CA escrow status is ambiguous — it reads as outstanding; verify against lab.kdbx and correct whichever is stale.
Alert-threshold drift: runbooks/27-failure-alerting.md passes obs_alert_cert_days_left=7, contradicting the 14 in roles/obs/defaults — and rb26 records that exact 7→14 correction as already made. rb27 looks like un-fixed drift from the same edit.
Appliance cert trust is unmonitored. push-appliance-ssh.sh pushes CA trust to Proxmox and OPNsense, but neither runs Alloy, so they are outside the ssh_host_cert_days_left metric entirely. A rotation that silently fails to land on the appliances would not alert.

8.1 — Where to write next¶

The procedures that do not exist anywhere and would need authoring before a real rotation: ansible-vault rekey (two scopes + TPM re-seal + re-encrypting lab-secrets), zfs change-key with escrow refresh (two pools, and two distinct key domains on the wd18tb pool), Tang key rotation with its full JWE fan-out, staged TSIG rotation, FreeIPA CA/DM/admin/svc-ldap, and Forgejo runner + Grafana tokens.

Phasing: any fleet-wide rotation should consume group_vars/all/tiers.yml via scripts/tier.sh — the repo's declared single source of truth for "the order I touch things" (t0_workstations → t1_canary → … → t4_critical). Do not invent a new order.

Designing a rotation alert: set the alert threshold strictly below the steady-state renewal floor by at least one full cycle. The host-cert rule is the model — 28-day validity, 21-day renewal floor, 14-day alert, so firing means a whole renewal cycle was missed rather than a cert merely aging.

Validation¶

[ ] Phase 0 completed: every escrow verified to round-trip, not merely present.
[ ] After each phase, the affected credential works in a fresh shell / fresh agent.
[ ] After Phase 2, a converge succeeds from a clean login shell with no LAB_VAULT_* set.
[ ] After Phase 5, every host carries a cert from the new CA before the old line goes.
[ ] After Phase 7, both tokens independently arm lab.kdbx (rb47) and sign a cert.
[ ] The rotated lab.kdbx is committed and pushed to lab-secrets (else L9).
[ ] A cold reboot of pve unlocks vmpool unattended (the only real proof for Tang work).