Runbook 15 — Dedicated Ansible control node¶

Goal: stand up a dedicated control-node VM (ctrl, AlmaLinux 9, MGMT 10.20.10.8) that runs all of Ansible — site.yml, update.yml, drift checks — and holds the control-plane secrets (lab.kdbx + the vault password, with the vault password TPM-sealed, never in plaintext). ctrl authenticates to the whole fleet with a CA-signed control cert — not a raw key — so there is no per-host key to distribute. YubiKey-gated CA signing lives on the admin laptop; ctrl and the admin laptop coordinate through git.

Why dedicated: the control node holds the crown of the config plane (fleet access + vault). Co-locating that with a general-purpose daily-driver (personal data, a browser, a roaming laptop) is the worst custody spot for it. A single-purpose EL9 VM in MGMT is common-managed, observable, Proxmox-backed-up, and destroy-and-rebuild like the rest of the fleet — no snowflake.

Why a cert, not a raw key: the fleet already trusts the SSH user CA (TrustedUserCAKeys), so a single CA-signed cert lets ctrl reach every host with nothing registered per-host. A cert also expires and is KRL-revocable — a raw authorized_keys entry is neither. So ctrl's fleet credential is a cert, and fleet_transport_pubkeys carries only root@pve (the pre-cert provisioning bootstrap).

Why MGMT: the control node must reach every zone, and per the inter-zone policy only MGMT may reach all zones. A control node in SERVERS or DMZ could not reach the others.

When to run: once, early — the control node is the thing that runs runbook 16 and everything after. The first apply is launched from your admin laptop (where the YubiKey + repo live); then ctrl self-manages.

Facts to confirm: Proxmox VM storage name (local-zfs below); your admin machine has the repo + lab.kdbx + vault access + the YubiKey to bootstrap from.

1. Provision the VM¶

# on the Proxmox host (pve), from the provision/ dir
./lab-provision.sh up ctrl                 # clones the AlmaLinux 9 template (vms.conf), starts it
# add a vTPM so the vault password can be sealed to it — a cold stop/start attaches the
# device (lab-provision.sh can't express it, like fileserver's PCI / tvheadend's USB):
qm stop 9128
qm set 9128 --tpmstate0 local-zfs:1,version=v2.0
qm start 9128

2. Bootstrap from your admin machine¶

ctrl is in all_linux + the [control] group, so the baseline + control role apply to it like any host. From the admin laptop (your YubiKey cert reaches the fresh VM over the provisioning path), repo root / ansible/:

ansible-playbook playbooks/site.yml -l ctrl

This runs common (SSH-CA trust, IPA enroll — MGMT can reach IPA, alloy, hardening, dnf-automatic) and the control role (pinned ansible-core + versionlock, git, keepassxc-cli, the /etc/lab secrets dir, and /etc/profile.d/lab-vault-cred.sh exporting LAB_VAULT_CRED). After this, ctrl shows up in Grafana automatically.

3. Give `ctrl` its fleet credential + secrets (one-time, out of band)¶

These are credential-bearing and not in git. ctrl runs Ansible as the almalinux user, so everything here lives under ~almalinux.

Root & sudo on ctrl. almalinux has sudo on ctrl (NOPASSWD drop-in /etc/sudoers.d/10-almalinux-ctrl, codified by the control role, control_restore_almalinux_sudo). It's needed two ways: ctrl is ansible_connection=local, so it self-manages via become; and it's the JIT ctrl-root path — an operator-signed admin-ctrl task cert → ssh almalinux@ctrl → sudo -i (runbook 28). That sudo is gated upstream by which cert principals may reach almalinux@ctrl (admin/admin-ctrl only — the agent's standing admin-mgmt cannot). Root SSH stays OFF (PermitRootLogin no, common hardening); root on ctrl is otherwise console-only (Proxmox console + the KeePass root password).

Break-glass recovery if almalinux sudo is ever lost (which also breaks ctrl self-management): two routes. (a) Console — Proxmox console → root (KeePass pw, or single-user rd.break) → recreate /etc/sudoers.d/10-almalinux-ctrl. (b) Offline from pve (no ctrl password needed — used 2026-06-28): scripts/fix-ctrl-access.sh run on pve as root, launched detached (systemd-run --no-block --collect --unit=fix-ctrl bash /root/fix-ctrl-access.sh) — it graceful-stops ctrl, mounts its root partition offline (the vmpool-enc pool is Tang-unlocked on pve), restores the sudoers drop-in + touch /.autorelabel, and restarts ctrl (always-restart trap). Launch detached because the operator/agent session may run on ctrl and dies when it stops. The control role keeps the drop-in reproducible afterwards.

a. The control cert — ctrl's key signed by the user CA, with the zone + crown-jewel principals so it reaches every host as almalinux:

# on ctrl: generate the control key. Initially no passphrase — it is protected by living on
# the TPM-backed, single-purpose box and by the cert's expiry/revocability. Once tang1 exists
# (runbook 24) you can additionally Tang-gate it for a network-bound fail-closed kill-switch
# on fleet reach — see §3d below.
ssh-keygen -t ed25519 -N '' -C ctrl@<instance> -f /home/almalinux/.ssh/id_ctrl

# sign it on the admin laptop (YubiKey): scripts/yubikey-tool.sh -> U (Sign a USER cert), answering:
#   user key to sign : id_ctrl.pub   (copy it over first)
#   principals       : admin-mgmt,admin-servers,admin-dmz,admin-freeipa,admin-workstations
#   identity/key-id  : ctrl-control
#   validity         : +52w
# then copy the minted id_ctrl-cert.pub to ctrl:/home/almalinux/.ssh/
# (admin-workstations lets the host-CA autosign reach the admin-laptop/desktop/gpu-host boxes, which authorize
#  it -> root via the workstation role; see group_vars/all/principals.yml.)

Point ctrl's ssh at the key + cert (~almalinux/.ssh/config):

Host 10.20.* *.${domain}
    IdentityFile ~/.ssh/id_ctrl
    CertificateFile ~/.ssh/id_ctrl-cert.pub
    IdentitiesOnly yes

The principals are the routine zone + crown-jewel set (admin-mgmt,admin-servers, admin-dmz,admin-freeipa) plus admin-workstations — they reach every host (including freeipa) as almalinux without using the break-glass admin principal. admin-workstations extends that reach to the workstation boxes (admin-laptop/desktop/gpu-host) so the host-CA autosign (runbook 26) can deploy their host certs; they authorize it via the workstation role (workstation_ssh_principals). Rotate this cert on the host-cert cadence (§5).

b. The repo + KeePass DB:

git clone http://10.20.30.30:3000/<org>/myos.git /opt/myos     # needs read access (below)
sudo chown -R almalinux:almalinux /opt/myos                       # see §4a.1
# lab.kdbx is gitignored — copy it in out of band:
#   scp lab.kdbx ctrl:/opt/myos/ansible/group_vars/all/lab.kdbx

/opt/myos read credential (so ctrl can git pull to self-manage). Mint a read-only Forgejo token and store it for almalinux (runtime state, NOT in git; recreate after a rebuild). Mint on the forgejo host (no password needed): sudo podman exec -u git forgejo forgejo admin user generate-access-token --username operator --token-name ctrl-opt-myos-ro --scopes read:repository, then on ctrl as almalinux: umask 077; printf 'http://operator:<TOKEN>@10.20.30.30:3000\n' > ~/.git-credentials; chmod 600 ~/.git-credentials (origin is the internal forgejo over HTTP; credential.helper store). The API token-create endpoint needs basic auth (a token can't mint a token), which is why the CLI is used. Set up 2026-06-28.

c. Seal the vault passphrase to this host's TPM (no plaintext at rest). Get the passphrase from KeePass (secrets.sh av, or the Ansible Vault entry in lab.kdbx), then on ctrl:

umask 077
printf '%s' 'THE_VAULT_PASSPHRASE' \
  | systemd-creds encrypt --name=lab-vault --with-key=tpm2 - /etc/lab/vault.cred
# verify (prints nothing on success):
systemd-creds decrypt --name=lab-vault /etc/lab/vault.cred - >/dev/null && echo OK

vault-pass.sh reads $LAB_VAULT_CRED (set by the control role) and decrypts this at run time — bound to ctrl's TPM, useless if copied off the box.

d. (After runbook 24) Tang-gate the control key — network-bound fail-closed kill-switch. The control key reaches the whole fleet; binding it to tang1 means "no tang1 → no fleet reach" (an operational off-switch for automation god-mode, and the key is no longer plaintext at rest). The vmpool encryption already protects it at rest; this adds the runtime gate. Per the [[tang-clevis-backup-password-policy]], the passphrase is the KeePass break-glass.

# on ctrl (tang1 up). Generate a passphrase, store it in KeePass (lab.kdbx -> 'ctrl control
# SSH key'), and clevis-tang-encrypt it next to the key; then add it to id_ctrl:
PASS=$(tr -dc 'A-Za-z0-9' </dev/urandom | head -c 32)        # -> KeePass (break-glass)
printf '%s' "$PASS" | clevis encrypt tang '{"url":"http://10.20.10.11"}' -y \
  > ~/.ssh/id_ctrl.pass.jwe
ssh-keygen -p -f ~/.ssh/id_ctrl -P '' -N "$PASS"             # encrypt key at rest (pubkey/cert unchanged)
sudo install -m755 /opt/myos/scripts/load-control-key.sh /usr/local/bin/load-control-key

Then every ansible session loads the key via tang first:

eval "$(load-control-key)"     # tang1 up -> agent gets key+cert; tang1 down -> FAIL-CLOSED
ansible all_linux -m ping

Break-glass when tang1 is down: ssh-add ~/.ssh/id_ctrl and type the KeePass passphrase. (Tang gates the load; a key already in the agent stays until ssh-add -D/reboot.)

4. Cut over — run from `ctrl`¶

Open a fresh almalinux login shell on ctrl (so LAB_VAULT_CRED is exported), then:

cd /opt/myos/ansible
ansible all_linux -m ansible.builtin.ping      # the control cert reaches the whole fleet
ansible-playbook --check playbooks/site.yml    # vault resolves via TPM; no errors

4a. Bootstrap-completion notes (do these, or you'll rediscover them)¶

Own the repo as almalinux. A clone done as root leaves /opt/myos (incl. .git, group_vars/all/vault.yml, lab.kdbx) root:root, so almalinux can't git it (dubious ownership) or read the vault-encrypted vault.yml. Fix once: sudo chown -R almalinux:almalinux /opt/myos.
Run deploys as the almalinux login shell — NOT sudo. The vault resolves via $LAB_VAULT_CRED (TPM), exported by /etc/profile.d/lab-vault-cred.sh; sudo strips it, so vault-pass.sh fails. ctrl needs no root for Ansible — it connects out. (§6's timer sets Environment= for the same reason.)
The control cert must carry the zone + crown-jewel principals. admin-mgmt,admin-servers,admin-dmz,admin-freeipa is what reaches all hosts (incl. freeipa) as almalinux. Reserve the break-glass admin principal for emergencies — it is not the routine control credential.
Host-key trust ctrl→fleet. ctrl needs the @cert-authority line in /etc/ssh/ssh_known_hosts (from the common role) and the fleet needs signed host certs, or ctrl prompts (TOFU) on first connect to each host.

5. CA signing stays on the admin laptop (the split)¶

The YubiKey lives on the admin laptop, so anything that signs with the user CA (grants access) runs there; the control node does the rest. The seam is git.

Exception — the host CA is now online on ctrl (runbook 26). Routine host-cert signing is automated here: an online host-CA key, Clevis+Tang-sealed to tang1 (/etc/lab/host-ca.jwe), is decrypted per-signature by host-ca-autosign.timer, which renews short-lived host certs (gather → sign → --tags sshca deploy), ntfys every sign, and fails closed if tang1 is unreachable (so a stolen/off-LAN ctrl can't sign). Only the user CA stays YubiKey-gated on the admin laptop. One-time bootstrap (generate + seal + KeePass/offline escrow — the raw key is deliberately not in the auto-vault) is in runbook 26; the control role wires the timer + the JWE perms + the ntfy token.

yubikey-tool.sh (provision a token, sign user/host certs — including ctrl's own control cert) → the admin laptop (needs the YubiKey).
push-krl.sh (revoke + push the KRL) → ctrl. Revoking edits the KRL with the CA public key only — no YubiKey — so this runs entirely on the control node.

rotate-host-certs.sh is split into gather → sign → deploy so the YubiKey step is isolated from fleet/vault access (the gather phase writes a manifest that sign consumes, so the laptop needs neither inventory nor the vault):

ctrl>    scripts/rotate-host-certs.sh gather t1_canary        # refresh pubkeys + manifest
ctrl>    git add ... && git commit && git push
admin-laptop>  git pull && scripts/rotate-host-certs.sh sign t1_canary A   # PIN + touch per host
admin-laptop>  git add ansible/roles/common/files/host_certs/*-cert.pub && git commit && git push
ctrl>    git pull && scripts/rotate-host-certs.sh deploy t1_canary   # push + verify

Roll ctrl's own control cert in the same ceremony (re-sign id_ctrl.pub with the §3a principals) — it expires ~yearly, and the common role's cert-expiry metric warns before it lapses.

6. Scheduled drift detection (built 2026-07-16)¶

The always-on control node alerts on configuration drift. scripts/lab-drift-check.sh + the control-role lab-drift-check.timer (weekly, Sun 05:30) run a read-only ansible-playbook playbooks/site.yml --check --diff and push a transition-only ntfy summary of hosts that drifted from code (changed>0 under --check). It never applies anything.

Design (the noise-control that makes it usable): - Curated skips. The FreeIPA ipa_access play is command-driven register→gate chains that error under --check, so it's skipped via --skip-tags ipa_access (the play is tagged). Other check-mode-incompatible tasks (firewalld install-then-enable, register-gated command tasks) surface as failed/unreachable per host and are reported as caveats, never as drift. Scope is --limit all_linux (workstations + edges have their own playbooks). - Transition-only (like obs-deadman): re-alerts only when the drifted/unevaluated host set changes — persistent drift doesn't spam. A first clean run is silent; a "cleared" message only follows a run that had drift. - The run log is sensitive, and last-run-only. --diff renders template content — including secret-bearing templates (e.g. the authelia config) — so last-run.log is written 0600 in a 0700 state dir and truncated every run (matching its name). It never accumulates, and it is not readable by other local users. Treat it as secret material: don't copy it off ctrl, paste it, or widen its mode. (Hardened 2026-07-18 — it was previously 0644 and appended forever.) - The unit sets LAB_VAULT_CRED= (TPM vault; timers don't source /etc/profile.d) and runs eval "$(scripts/load-control-key.sh)" first for DMZ reach (fail-closed if tang1 is down → those hosts show unreachable, a caveat).

Ships DISABLED (drift_check_enabled: false). Enable after a manual trial run tunes the skip set:

# On ctrl, as almalinux (never sudo — it strips LAB_VAULT_CRED):
cd /opt/myos/ansible && eval "$(scripts/load-control-key.sh)"
ansible-playbook playbooks/site.yml --check --diff --skip-tags ipa_access -l all_linux | tail -40
# Review what reports `changed` — reconcile real drift (apply or commit), and if a role is
# inherently noisy in --check, add it to drift_check_skip_tags. Then set drift_check_enabled: true.

Validation¶

[ ] qm config 9128 | grep tpmstate shows a v2.0 vTPM.
[ ] ansible-playbook playbooks/site.yml -l ctrl is idempotent (2nd run = no changes).
[ ] On ctrl: ansible --version shows the pinned ansible-core; dnf versionlock list includes it; no community bundle installed.
[ ] On ctrl: systemd-creds decrypt --name=lab-vault /etc/lab/vault.cred - succeeds; a fresh almalinux shell has LAB_VAULT_CRED set; scripts/vault-pass.sh prints the passphrase.
[ ] From ctrl (as almalinux): ansible all_linux -m ansible.builtin.ping succeeds via the control cert, and a full ansible-playbook --check playbooks/site.yml runs clean.
[ ] ssh-keygen -L -f ~almalinux/.ssh/id_ctrl-cert.pub shows principals admin-mgmt,admin-servers,admin-dmz,admin-freeipa and a ~52w validity.
[ ] The sealed cred fails to decrypt if copied to another host (TPM-bound) — confirms no plaintext/portable secret.
[ ] CA signing (incl. re-signing ctrl's control cert) works from the admin laptop.