Skip to content

Runbook 16 — Ansible: config-as-code & maintenance (boring & pinned)

Goal: layer configuration + ongoing maintenance on the provisioned fleet (runbook 01) with Ansible — a common baseline, identity-DB backups, and coordinated updates. Built-in modules only, pinned ansible-core. This is the closing, ongoing-ops runbook.

When to run what: - Baseline (site.yml) — applied in runbook 09 (VMs networked, before any app install); re-run any time, it's idempotent. This runbook is its reference. - DB backups (db-backups.yml) — for the stateful DB hosts (FreeIPA, umami, seafile). Imported by site.yml since 2026-07-10, so the main converge covers every [db_servers] host automatically (a host once sat unbacked for weeks because this play was manual-only); the standalone playbook remains for targeted runs. - Updates (update.yml) — forever, on your cadence.

Scaffold lives at ansible/: 

ansible.cfg  inventory.ini  inventory.d/  group_vars/vps_edges.yml
playbooks/  site.yml  update.yml  db-backups.yml  edge.yml
roles/  common/  db_backups/  vps_edge/

Two inventory sources. ansible.cfg sets inventory = inventory.ini,inventory.d/. inventory.ini is the tracked, generic fleet; inventory.d/ carries the instance's own hosts (its interactive workstations) as a gitignored fragment deployed from the Tang-sealed instance-material bundle (runbook 33). Only the empty inventory.d/00-generic placeholder is tracked, so a clean clone still parses.

Two sharp edges, both of which fail silently: - Fragments must be EXTENSIONLESS. inventory_ignore_extensions skips *.ini inside an inventory directory; a 10-workstations.ini there yields an empty group with no error. - -i inventory.ini overrides ansible.cfg. Most commands in these runbooks pass it explicitly, so they see the generic inventory only. That's fine for the fleet, but a play that targets [workstations] must be run without -i (letting ansible.cfg supply both sources), or with -i inventory.ini -i inventory.d/. Likewise never let a fleet role resolve hostvars['<workstation>'] — pin such a backend to a literal IP:port in the overlay.

Origin vs. edge. site.yml / update.yml target the home fleet (inventory group [origin] + [identity], children of all_linux). The public edge VPSes ([vps_edges]) are remote/off-prem and managed separately by edge.yml (common baseline + the vps_edge role) — see runbook 17. They're deliberately not in all_linux so a home-fleet run never reaches across the tunnel.

Why this stack (stability rationale)

Install ansible-core only (not the big ansible bundle), so exotic collections can't creep in. On an EL control node, dnf install ansible-core gives you the distro-maintained LTS — pinned by the EL9 lifecycle, no version-chasing (the most boring option, and it matches the fleet). Playbook syntax + ansible.builtin.* are about as stable as infra tooling gets, and it's agentless (just SSH + Python — nothing running to rot).

Conventions: only ansible.builtin.* (prefer command/template/copy over adding a collection); secrets via Ansible Vault; per-host overrides in host_vars/<name>.yml (e.g. backend_app).

1. Control node + common baseline (applied in runbook 09)

The control node is a dedicated EL9 VM — see runbook 15 for standing it up (pinned ansible-core, the transport key + lab.kdbx, vault password TPM-sealed; CA signing split to the admin laptop). The common baseline (ansible-playbook playbooks/site.yml) is run during the build at runbook 09 — once the VMs are networked and before any app install, so each VM is hardened before it carries a service. The common role sets timezone, EPEL, qemu-guest-agent, fail2ban, SSH hardening (no root, key-only), dnf-automatic, and optional FreeIPA CA trust (lab_ipa_ca_file). serial: 1 so the origin services and IdP never drop together.

Re-run ansible-playbook playbooks/site.yml here whenever you change the role (idempotent) — e.g. after FreeIPA to pick up CA trust.

Two hosts are converged differently — exclude them from a fleet run, don't forget them

all_linux contains freeipa and ctrl, and neither is converged by an ordinary fleet run from the agent PAW. Both sit early in the alphabet, so before the fix in this playbook's Common baseline comment they aborted the whole serial: 1 run at host 3 of ~29 and everything after went silently unconverged (observed 2026-08-01).

From the agent PAW (claude-agent), run the fleet as:

ansible-playbook -i inventory.ini playbooks/site.yml --limit '!freeipa:!ctrl'

That is a routing decision, not a licence to skip them. Each has its own path, and a converge that never reaches them is drift:

Host Why a PAW fleet run can't do it How it IS converged
freeipa The agent cert deliberately carries no admin-freeipa principal — the identity/CA core is outside its reach (repo CLAUDE.md, runbook 28). SSH fails Permission denied (publickey). By the operator, whose credential holds that principal.
ctrl ansible_connection=local and claude-agent has no sudo on its own PAW, by charter — fact gathering fails sudo: a password is required. As almalinux, in a fresh login shell, from /opt/myos/ansible — the T1 automation path (control cert + TPM-sealed vault). See runbook 15 §4.

So the full picture is three invocations, not one: the --limit run above, the operator's run for freeipa, and runbook 15 §4 for ctrl. The playbook now reports and skips a host whose facts it cannot gather rather than aborting, so a straggler no longer blocks the rest — but a skipped host is still an unconverged host, and the skip message says so.

2. DB backups (identity services)

Image backups are crash/fs-consistent; add app-consistent logical dumps: 

ansible-playbook playbooks/db-backups.yml
The db_backups role installs a nightly systemd timer writing to the VM disk (so it rides the daily Proxmox/ZFS backup), with retention: - FreeIPA → ipa-backup --data --online (non-disruptive) - umami → pg_dump; seafile → mysqldump (via podman exec)

Per-host kind is set in the inventory (db_backup_kind=ipa|umami|seafile). Also take a periodic full/offline ipa-backup on a maintenance window for DR.

3. Backups posture (Proxmox/ZFS)

All VMs/CTs are on ZFS and Proxmox backs them up daily — the baseline; no restic/borg layer. The installed qemu-guest-agent lets Proxmox fsfreeze for fs-consistent images. Ensure backups land off the source pool/host (PBS / another pool / zfs send), and test a restore occasionally.

4. Keep it updated

  1. Per-host auto security updates — dnf-automatic (set by the common role).
  2. Snapshot before major upgrades — qm snapshot <vmid> pre-upgrade-<date>; instant rollback on ZFS.
  3. Coordinated full upgrades — ansible-playbook playbooks/update.yml (serial: 1, dnf upgrade, reboots only if required). Edges re-apply via ansible-playbook playbooks/edge.yml (also serial: 1, one edge at a time, so the public layer never blanks); their OS security patches still come from dnf-automatic set by the common baseline.
  4. Keep the template current — periodically boot it, dnf upgrade, re-seal, shut down, so fresh clones aren't months behind.
  5. App layer, deliberately (not auto):
  6. Authelia (podman quadlet): pin the image tag; read release notes before pulling the new image + restarting the quadlet.
  7. OPNsense: its own built-in update channel.

The full "rebuild the lab" flow

  1. Provision (runbook 01): ./lab-provision.sh rebuild all.
  2. Configure: ansible-playbook playbooks/site.yml, then the app runbooks (09/13) + db-backups.yml.
  3. Restore stateful data from the Proxmox/ZFS daily backups (FreeIPA, umami/seafile DBs, fileserver ZFS datasets); use the logical DB dumps for clean restores.
  4. OPNsense from its own config backup.

Provisioning + config are reproducible from this repo; only data comes from backups — the line you want between code and state.

Known FALSE failures in --check (read this BEFORE diagnosing a check-mode failure)

--check reports failures that do not occur in a real run. Check this list first — every entry below has been re-diagnosed from scratch more than once, which is pure waste.

Play/task Check-mode error Why it is false
alloy : Unpack alloy (every host, via common) Source '/tmp/alloy-<ver>.zip' does not exist The preceding Download alloy zip task is skipped in check mode, so the archive the unpack task wants was never fetched. Real runs download it and pass.

The rule, not just the list: before diagnosing any --check failure, re-run the same play unmodified (stash your change, or run it from a clean checkout of main) and compare recaps. If the failure count and task are identical, it is pre-existing/environmental — not your change, and usually not a real defect at all. Diagnosing before that comparison is how an hour disappears.

Corollary for isolated checkouts (git worktree, fresh clones): they carry no gitignored instance material, and that produces its own crop of false failures (undefined vault_* vars, missing CA pubkeys). Deploy it first: 

scripts/load-instance-material.sh <checkout>   # public material: CA pubkeys, ipa-ca.crt, zz-local
scripts/load-lab-secrets.sh      <checkout>    # the encrypted *.vault.yml + vault.yml
Note the secrets overlay is vault.yml AND *.vault.yml — a *.vault.yml glob silently misses the primary vault.yml, and the symptom is an undefined vault_* var several tasks later.

Never disable no_log to read a censored error. Those tasks render real secrets (Grafana admin password, OAuth client secret, Authelia config), so unmasking dumps crown jewels into the log. To get the error safely, render just that template ad-hoc with the role defaults in scope and grep only for the error class: 

ansible -i inventory.ini <host> -m ansible.builtin.template \
  -a "src=roles/<role>/templates/<f>.j2 dest=/tmp/probe" \
  -e @roles/<role>/defaults/main.yml --check 2>&1 | grep -oE "AnsibleUndefinedVariable[^\"]{0,120}"
(Ad-hoc module calls do not load role defaults — omit the -e @.../defaults/main.yml and you will chase a phantom "undefined" for a variable that is in fact defined.)

Validation

  • [ ] ansible all -m ansible.builtin.ping succeeds; site.yml is idempotent (second run = no changes).
  • [ ] SSH to a configured VM is key-only, no root; dnf-automatic active.
  • [ ] systemctl list-timers db-backup.timer scheduled on FreeIPA + umami + seafile; a manual run produces a dump.
  • [ ] update.yml upgrades hosts one-at-a-time, reboots only when required.
  • [ ] ansible --version shows the pinned ansible-core; no community bundle.