Runbook 16 — Ansible: config-as-code & maintenance (boring & pinned)¶

Goal: layer configuration + ongoing maintenance on the provisioned fleet (runbook 01) with Ansible — a common baseline, identity-DB backups, and coordinated updates. Built-in modules only, pinned ansible-core. This is the closing, ongoing-ops runbook.

When to run what: - Baseline (site.yml) — applied in runbook 09 (VMs networked, before any app install); re-run any time, it's idempotent. This runbook is its reference. - DB backups (db-backups.yml) — for the stateful DB hosts (FreeIPA, umami, invidious, someonetoldme). Imported by site.yml since 2026-07-10, so the main converge covers every [db_servers] host automatically (a host once sat unbacked for weeks because this play was manual-only); the standalone playbook remains for targeted runs. - Updates (update.yml) — forever, on your cadence.

Scaffold lives at ansible/:

ansible.cfg  inventory.ini  inventory.d/  group_vars/vps_edges.yml
playbooks/  site.yml  update.yml  db-backups.yml  edge.yml
roles/  common/  db_backups/  vps_edge/

Two inventory sources. ansible.cfg sets inventory = inventory.ini,inventory.d/. inventory.ini is the tracked, generic fleet; inventory.d/ carries the instance's own hosts (its interactive workstations) as a gitignored fragment deployed from the Tang-sealed instance-material bundle (runbook 33). Only the empty inventory.d/00-generic placeholder is tracked, so a clean clone still parses.

Two sharp edges, both of which fail silently: - Fragments must be EXTENSIONLESS. inventory_ignore_extensions skips *.ini inside an inventory directory; a 10-workstations.ini there yields an empty group with no error. - -i inventory.ini overrides ansible.cfg. Most commands in these runbooks pass it explicitly, so they see the generic inventory only. That's fine for the fleet, but a play that targets [workstations] must be run without -i (letting ansible.cfg supply both sources), or with -i inventory.ini -i inventory.d/. Likewise never let a fleet role resolve hostvars['<workstation>'] — pin such a backend to a literal IP:port in the overlay.

Origin vs. edge. site.yml / update.yml target the home fleet (inventory group [origin] + [identity], children of all_linux). The public edge VPSes ([vps_edges]) are remote/off-prem and managed separately by edge.yml (common baseline + the vps_edge role) — see runbook 17. They're deliberately not in all_linux so a home-fleet run never reaches across the tunnel.

Why this stack (stability rationale)¶

Install ansible-core only (not the big ansible bundle), so exotic collections can't creep in. On an EL control node, dnf install ansible-core gives you the distro-maintained LTS — pinned by the EL9 lifecycle, no version-chasing (the most boring option, and it matches the fleet). Playbook syntax + ansible.builtin.* are about as stable as infra tooling gets, and it's agentless (just SSH + Python — nothing running to rot).

Conventions: only ansible.builtin.* (prefer command/template/copy over adding a collection); secrets via Ansible Vault; per-host overrides in host_vars/<name>.yml (e.g. backend_app).

Invocation: run converges via scripts/converge.sh <playbook> [args] (it passes everything through to ansible-playbook, running from its own checkout's ansible/ dir, so it works from any cwd or worktree). The wrapper holds an exclusive per-user lock for the run's whole life, so two converges can never interleave — a stray parallel run from another session or a stale branch silently reverts deployed work (it happened for real 2026-08-01) — and on contention it fails fast naming the holder. See runbook 28 §Session topology.

1. Control node + common baseline (applied in runbook 09)¶

The control node is a dedicated EL9 VM — see runbook 15 for standing it up (pinned ansible-core, the transport key + lab.kdbx, vault password TPM-sealed; CA signing split to the admin laptop). The common baseline (ansible-playbook playbooks/site.yml) is run during the build at runbook 09 — once the VMs are networked and before any app install, so each VM is hardened before it carries a service. The common role sets timezone, EPEL, qemu-guest-agent, fail2ban, SSH hardening (no root, key-only), dnf-automatic, a journald volatile-journal cap (RuntimeMaxUse=64M drop-in — the default 10%-of-/run cap is computed from the host's tmpfs inside an LXC, so it can exceed the container's RAM; registry OOM'd on exactly this, 2026-08-14), and optional FreeIPA CA trust (lab_ipa_ca_file). serial: 1 so the origin services and IdP never drop together.

Re-run scripts/converge.sh playbooks/site.yml here whenever you change the role (idempotent) — e.g. after FreeIPA to pick up CA trust.

Two hosts are converged differently — exclude them from a fleet run, don't forget them¶

all_linux contains freeipa and ctrl, and neither is converged by an ordinary fleet run from the agent PAW. Both sit early in the alphabet, so before the fix in this playbook's Common baseline comment they aborted the whole serial: 1 run at host 3 of ~29 and everything after went silently unconverged (observed 2026-08-01).

From the agent PAW (claude-agent), run the fleet as:

scripts/converge.sh -i inventory.ini playbooks/site.yml --limit '!freeipa:!ctrl'

That is a routing decision, not a licence to skip them. Each has its own path, and a converge that never reaches them is drift:

Host	Why a PAW fleet run can't do it	How it IS converged
`freeipa`	The agent cert deliberately carries no `admin-freeipa` principal — the identity/CA core is outside its reach (repo `CLAUDE.md`, runbook 28). SSH fails `Permission denied (publickey)`.	By the operator, whose credential holds that principal.
`ctrl`	`ansible_connection=local` and `claude-agent` has no sudo on its own PAW, by charter — fact gathering fails `sudo: a password is required`.	As `almalinux`, in a fresh login shell, from `/opt/myos/ansible` — the T1 automation path (control cert + TPM-sealed vault). See runbook 15 §4.

So the full picture is three invocations, not one: the --limit run above, the operator's run for freeipa, and runbook 15 §4 for ctrl. The playbook now reports and skips a host whose facts it cannot gather rather than aborting, so a straggler no longer blocks the rest — but a skipped host is still an unconverged host, and the skip message says so.

The three pre_tasks that implement this are tags: always, and must stay that way. They are a precondition for the play, not a part of it. Tagged only common (as they were until 2026-08-19) they are filtered out of every tag-subset run — --tags sshca, --tags metrics, … — which silently removes both halves at once: no facts gathered, and no end_host guard. The play then dies on the first fact-dependent task and serial: 1 aborts the run: precisely the failure this mechanism exists to prevent, reintroduced through the tag door.

It stayed hidden because every tagged task in roles/common happened to reference no facts. The first one that did — the pending-reboot metric's ansible_os_family check — failed on host 1 of the run. Note --tags sshca is what the host-CA autosign signer runs (runbook 26): fine today only by that same accident, one fact reference away from the same abort, and running with no guard beneath it.

Verify after touching this play — --list-tasks shows it without contacting anything:
ansible-playbook playbooks/site.yml --tags sshca --list-tasks | grep -c 'Gather facts\|End this host'   # expect 2

2. DB backups (identity services)¶

Image backups are crash/fs-consistent; add app-consistent logical dumps:

scripts/converge.sh playbooks/db-backups.yml

The db_backups role installs a nightly systemd timer writing to the VM disk (so it rides the daily Proxmox/ZFS backup), with retention: - FreeIPA → ipa-backup --data --online (non-disruptive) - umami, invidious, someonetoldme → pg_dump (via podman exec) (seafile → mysqldump was removed 2026-08-31 with the Seafile retirement, rb41.)

Invidious joined the covered set on 2026-08-14 — its accounts/subscriptions had only the crash-consistent guest replication until then, and that state is not re-derivable.

Per-host kind is set in the inventory (db_backup_kind=ipa|umami|invidious|someonetoldme). Also take a periodic full/offline ipa-backup on a maintenance window for DR.

Postgres major upgrades (the #1259 datadir convention)¶

Crossing a postgres major on the quadlet postgres DBs (invidious-db, umami-db, and any other *-db quadlet) is a dump/restore into a fresh cluster, never pg_upgrade (the DBs are megabytes; restore also rebuilds indexes across any libc/collation difference and picks up 18's default data checksums). The 18+ images moved the mount convention to /var/lib/postgresql with the cluster in a <major>/docker subdir (docker-library #1259) — the entrypoint refuses to start if the old …/postgresql/data mount exists or a pre-18 cluster sits at the mount root. Procedure (details in the PR that migrated each host): stop the app → final pg_dump (plain + -Fc + globals, gzip -t + CREATE TABLE count before touching anything) → stop db → mv the old cluster aside (db-pgNN — this IS the rollback) → converge the new pin/Volume → the app auto-creates an empty schema; DROP DATABASE … WITH (FORCE) + CREATE + restore (ON_ERROR_STOP + --single-transaction) → ANALYZE → verify row-count baseline + app probes. Keep db-pgNN through 2 bizon replication cycles. Alpine images stay alpine (musl→glibc silently changes collation ordering).

3. Backups posture (Proxmox/ZFS)¶

All VMs/CTs are on ZFS and Proxmox backs them up daily — the baseline; no restic/borg layer. The installed qemu-guest-agent lets Proxmox fsfreeze for fs-consistent images. Ensure backups land off the source pool/host (PBS / another pool / zfs send), and test a restore occasionally.

4. Keep it updated¶

Per-host auto security updates — dnf-automatic (set by the common role).
Snapshot before major upgrades — qm snapshot <vmid> pre-upgrade-<date>; instant rollback on ZFS.
Coordinated full upgrades — scripts/converge.sh playbooks/update.yml (serial: 1, dnf upgrade, reboots only if required). Edges re-apply via scripts/converge.sh playbooks/edge.yml (also serial: 1, one edge at a time, so the public layer never blanks); their OS security patches still come from dnf-automatic set by the common baseline.
Keep the template current — periodically boot it, dnf upgrade, re-seal, shut down, so fresh clones aren't months behind.
App layer, deliberately (not auto):
Authelia (podman quadlet): pin the image tag; read release notes before pulling the new image + restarting the quadlet.
OPNsense: its own built-in update channel.

The full "rebuild the lab" flow¶

Provision (runbook 01): ./lab-provision.sh rebuild all.
Configure: scripts/converge.sh playbooks/site.yml, then the app runbooks (09/13) + db-backups.yml.
Restore stateful data from the Proxmox/ZFS daily backups (FreeIPA, umami/invidious DBs, fileserver ZFS datasets); use the logical DB dumps for clean restores.
OPNsense from its own config backup.

Provisioning + config are reproducible from this repo; only data comes from backups — the line you want between code and state.

Known FALSE failures in `--check` (read this BEFORE diagnosing a check-mode failure)¶

--check reports failures that do not occur in a real run. Check this list first — every entry below has been re-diagnosed from scratch more than once, which is pure waste.

Play/task	Check-mode error	Why it is false
(none currently)		`media_fetch : Unpack deno` (`Source '/tmp/deno-<ver>.zip' does not exist`) was fixed at the source 2026-09-13, same class and same one-line fix: `Current deno version` / `Current yt-dlp version` / `Query ffmpeg encoders` are pure reads that `--check` skipped, so the version gate was blind ("needs upgrade" on every host), the block ran, the `curl` was skipped too, and the `unarchive` false-failed on a zip nobody downloaded. It is what made the weekly drift check report `fileserver` as not fully evaluated. All four now carry `check_mode: false`. — The long-standing `alloy : Unpack alloy` entry (`Source '/tmp/alloy-<ver>.zip' does not exist`) was fixed at the source 2026-08-16: the `Current alloy version` gate is a pure-read command that `--check` used to skip, blinding the version gate so the install block always ran into the missing zip. It now carries `check_mode: false` (as do the forgejo gpg key reads, same class), so `--check` skips the block on up-to-date hosts. The residual case: a fresh, never-converged host still false-fails these blocks under `--check` — converge it for real first.

The rule, not just the list: before diagnosing any --check failure, re-run the same play unmodified (stash your change, or run it from a clean checkout of main) and compare recaps. If the failure count and task are identical, it is pre-existing/environmental — not your change, and usually not a real defect at all. Diagnosing before that comparison is how an hour disappears.

Corollary for isolated checkouts (git worktree, fresh clones): they carry no gitignored instance material, and that produces its own crop of false failures (undefined vault_* vars, missing CA pubkeys). Deploy it first:

scripts/load-instance-material.sh <checkout>   # public material: CA pubkeys, ipa-ca.crt, zz-local
scripts/load-lab-secrets.sh      <checkout>    # the encrypted *.vault.yml + vault.yml

Note the secrets overlay is vault.yml AND *.vault.yml — a *.vault.yml glob silently misses the primary vault.yml, and the symptom is an undefined vault_* var several tasks later.

Never disable no_log to read a censored error. Those tasks render real secrets (Grafana admin password, OAuth client secret, Authelia config), so unmasking dumps crown jewels into the log. To get the error safely, render just that template ad-hoc with the role defaults in scope and grep only for the error class:
ansible -i inventory.ini <host> -m ansible.builtin.template \
  -a "src=roles/<role>/templates/<f>.j2 dest=/tmp/probe" \
  -e @roles/<role>/defaults/main.yml --check 2>&1 | grep -oE "AnsibleUndefinedVariable[^\"]{0,120}"
(Ad-hoc module calls do not load role defaults — omit the -e @.../defaults/main.yml and you will chase a phantom "undefined" for a variable that is in fact defined.)

Validation¶

[ ] ansible all -m ansible.builtin.ping succeeds; site.yml is idempotent (second run = no changes).
[ ] SSH to a configured VM is key-only, no root; dnf-automatic active.
[ ] systemctl list-timers db-backup.timer scheduled on FreeIPA + umami + invidious; a manual run produces a dump.
[ ] update.yml upgrades hosts one-at-a-time, reboots only when required.
[ ] ansible --version shows the pinned ansible-core; no community bundle.

Trap (2026-09-13): {{ ansible_managed }} is only defined while a template renders. Inside copy: content: it is an undefined variable and fails the task — and a failed play discards every pending handler, so the restart you expected never happens. Use template for any managed-file header. Found on the fileserver's first deploy-shelf converge.