28 — AI agent operating model (PAW on `ctrl`, own YubiKey-signed cert)¶

How the IT agent (Claude) is run as the system administrator: from a Privileged Access Workstation (ctrl), holding its own fleet credential — a keypair on ctrl plus a short-lived user cert you sign from the YubiKey — never the automation control cert.

Why this shape. Best practice for a privileged automation/agent identity is the PAW + central-identity-gate pattern: the agent operates from one hardened, audited box; its credentials live on that box, not on a roaming client; access originates from the offline access-CA (your YubiKey 9c); sensitive actions stay human-gated; everything is audited; revocation is central. This reaches every host incl. DMZ (the user-CA is trusted fleet-wide) without enrolling DMZ in IPA and without any online access-CA — so it respects both the DMZ-isolation and the split-CA invariants (CLAUDE.md). See the "Credential custody model" section of CLAUDE.md for the four-tier picture.

A dedicated IPA identity for the agent (named user + HBAC + IPA sudo — the human-operator model) was declined 2026-07-07: an injectable agent must minimise standing privilege (elevation stays JIT), keep a uniform fail-closed cutoff (one cert kill-switch, not a split ipa-disable+KRL), and stay IPA-independent (so it can fix IPA when IPA is down). IPA-as-PAM is reserved for humans. Full rationale in the CLAUDE.md custody model.

Piece	What	Where
PAW	Claude runs as the `claude-agent` account, started manually with `agent` (run inside `tmux` for a persistent session)	`ctrl` (`roles/control`)
Agent key	own keypair; passphrase Clevis+Tang-sealed to tang1 (fail-closed off-LAN)	`claude-agent@ctrl:~/.ssh/id_agent` (+ `.pass.jwe`)
Agent cert (standing)	YubiKey 9c-signed; key-id `claude-agent`; principals `admin-mgmt,admin-servers,admin-dmz,admin-workstations` (NOT `admin-freeipa`/`admin-ctrl`/`admin-hv`); `+1w`	signed on demand by `scripts/yubikey-tool.sh sign-agent`
Task cert (JIT, ADDITIVE)	per-task elevation on a separate Tang-sealed key (`id_agent_task`), loaded ALONGSIDE the standing cert so it never disturbs baseline access; elevated principals (incl. crown jewels) for a short TTL (default `+4h`); key-id `claude-agent-task-<label>-<date>`	`scripts/yubikey-tool.sh sign-agent-task` — see "Task-scoped (JIT) signing"
Reach	standing: all zones except the identity/CA core and ctrl; lands as `almalinux` via the principal→account map, attributed by the cert key-id. A signed task cert reaches whatever its principals grant (e.g. `admin-ctrl` → ctrl-root via `almalinux`+sudo)	standing: fleet-wide (incl. DMZ), not ctrl; task: per-grant
Cutoff	EXPIRY (lapses in ~1w) · KRL (`scripts/push-krl.sh`, instant) · LIVE-KILL (`scripts/agent-killswitch.sh`)	operator/YubiKey-controlled

The agent runs ansible with its own cert (not the control cert) and uses a TPM-sealed vault credential of its own (/etc/lab/vault-agent.cred). Two distinct credentials are in play: you authenticate to the claude-agent account with your 9a login cert (ctrl's auth_principals/claude-agent); the agent reaches the fleet with its own id_agent cert.

Containment: minimal standing access, operator-signed elevation (JIT)¶

The model is not "the agent may never root its PAW" — it's least standing privilege + just-in-time, operator-signed, short-lived elevation. The agent's STANDING cert (+1w, admin-mgmt,servers,dmz) gets root on the fleet it services (become on the targets) but cannot root its own control node, enforced by THREE independent facts that hold for the standing cert: (1) the claude-agent account has no sudo on ctrl (the agent runs as claude-agent); (2) it can't read ctrl's privileged material (the almalinux control key, /etc/lab/host-ca.jwe, vault.cred — it only traverses to its own vault-agent.cred); (3) the standing fleet cert can't hop to almalinux@ctrl — ctrl is a crown-jewel host (admin-ctrl in principals.yml, like freeipa), so auth_principals/almalinux lists admin,admin-ctrl only; the agent's admin-mgmt is excluded.

Elevation is granted per task, not denied absolutely. When a task needs more (incl. ctrl-root or the identity core), the operator signs a short-TTL task cert with exactly those principals (sign-agent-task, below). The ctrl-root path: an admin-ctrl task cert → ssh almalinux@ctrl (the only SSH that lands there) → sudo -i (almalinux sudo is codified in the control role — NOPASSWD, gated upstream by who can reach almalinux@ctrl). The grant lapses (short TTL) and is KRL-revocable, so the kill-switch stays real. Root SSH stays off (PermitRootLogin no); root on ctrl is otherwise console-only (operator + KeePass). The standing cert still can't reach it — only a deliberately-signed task cert can. admin (break-glass) reaches almalinux@ctrl too. ctrl is ansible_connection=local + almalinux has sudo, so it self-manages in place.

Session topology — ONE hub, dispatched subagents (2026-08-02)¶

The operator talks to exactly one Claude session (the hub), which dispatches and coordinates subagents for parallel work. Running multiple peer sessions side by side is retired — it failed in practice and never bought what it appeared to:

No privilege separation. Every session runs as claude-agent and shares one ssh-agent (~/.ssh/agent.sock): a JIT task cert or an armed KeePass unlock is available to all windows, not just the one it was granted "for". Parallel sessions compartmentalize nothing.
Real collisions. The shared ~/src/myos checkout changed branches under a working agent; a converge from main silently reverted another session's deployed-but-unmerged work (fired for real 2026-08-01, caught by a different agent). The fleet is a single serialization domain — only one converge at a time — and peer sessions have no owner for that queue.

The four access levels are grant tiers of ONE account, not agent types — and they map to dispatch policy in the hub model:

Tier	Grant	Where it is used
1	KeePass armed (operator-armed, lapses)	Hub main loop only — never a background subagent
2	JIT task cert (`admin-ctrl`/`admin-hv`/…, short TTL)	Hub main loop only — operator just armed it, and the watched front loop is where crown-jewel actions belong
3	Standing fleet cert (myos sysadmin)	Hub, plus subagents for scoped prep work; the hub owns all merges + converges
4	App/instance work (same cert, narrower scope)	The natural subagent tier — per-app, worktree-isolated; hands converge needs back to the hub

Rules of the topology:

Front model: Fable 5 fronts the hub and dispatches Opus 4.8 subagents to execute. (Supersedes the 2026-07-25 two-mode "Fable-at-forks / Opus-on-straightaways" split: in hub mode the main loop is the fork point — triage, dispatch, premise-checks, merge decisions — and execution tokens move to the subagents.) Written artifacts (runbooks / decision records / memory) remain the only handoff medium between models and sessions.
Converges are serialized through scripts/converge.sh — a flock wrapper that fails fast, naming the holder, if another converge is running. Never invoke ansible-playbook directly for a converge (runbook 16). Merge before converging: deployed-but-unmerged work is reverted by the next converge from main.
Subagents work in their own git worktree (with the instance-material symlinks — group_vars/all overlays, host_vars, inventory.d — if they must run ansible), never in the shared checkout, and report per-task changed= results back to the hub.
The operator's authenticated tmux-window handoff is unchanged — for access the agent's certs don't cover, he still opens an authenticated window in the hub's tmux session and the hub drives it.

Operator touchpoints — what the operator is actually for¶

The hub is the interface: it fronts the myos repo, the lab infrastructure, and ntfy alarm triage, so the operator hears only what needs him. His own projects are his — the hub's involvement per project is a one-time stand-up of serving infrastructure + deploy automation, after which push-to-deploy works with nobody in the loop (requests arrive through the intake channel, runbook 51 §6). That leaves four touchpoints, every one of them either a master key or an approval click:

#	Touchpoint	Why it can't be delegated
1	Master keys — KeePass arm (rb47), JIT task-cert signature (above), the YubiKey user-CA	their entire value is that a human held them
2	Approve + merge T2 PRs in the Forgejo UI (runbook 51)	T2 = every path the host-CA signer executes; his approval is what the merge signature means (rb26)
3	Digest sampling — delegated merges + reviewer approve-rate appear in the 04:00 digest	reading it is the audit; acting on it is optional
4	Veto, not decision duty — shadow-mode exit and tier promotions announce themselves in the digest with evidence against pre-registered criteria	silence proceeds; one word stops it

Everything else — protection edits, account creation, ntfy users, negative tests, intake handling, project stand-ups — is the hub's work, not his.

One-time bootstrap (operator)¶

Prereqs: ctrl built (runbook 15), tang1 up (runbook 24), the control role applied (creates the claude-agent account, ~/.ssh, the SSH config, auth_principals/claude-agent, the ntfy token). Then, on ctrl as root/almalinux:

Generate the agent keypair + Tang-seal its passphrase (mirrors the control key, rb15/26):

sudo -u claude-agent bash -lc '
  PASS=$(openssl rand -base64 32)
  ssh-keygen -t ed25519 -C claude-agent@<instance> -N "$PASS" -f ~/.ssh/id_agent
  printf "%s" "$PASS" | clevis encrypt tang "{\"url\":\"http://10.20.10.11\"}" -y > ~/.ssh/id_agent.pass.jwe
  chmod 600 ~/.ssh/id_agent ~/.ssh/id_agent.pass.jwe
  echo "ALSO store this passphrase in KeePass lab.kdbx as \"claude-agent SSH key\" (break-glass):"
  printf "%s\n" "$PASS"
'

Record the passphrase in KeePass lab.kdbx (break-glass), then clear your scrollback.

Seal the agent's vault credential to ctrl's TPM (same vault password as almalinux's; the operator approved the agent+ansible sharing the vault):
```
umask 077; printf '%s' "$VAULT_PASSPHRASE" \
  | sudo systemd-creds encrypt --name=lab-vault --with-key=tpm2 - /etc/lab/vault-agent.cred
sudo chgrp claude-agent /etc/lab/vault-agent.cred && sudo chmod 640 /etc/lab/vault-agent.cred
```
Confirm claude-agent can decrypt it (TPM device access — add claude-agent to the tss group if systemd-creds decrypt is denied, as for almalinux in rb15). Re-run the control role so its perms/warn tasks settle.

Install Claude Code for claude-agent + log it in (interactive, once):

sudo -u claude-agent -i      # then install per the current Claude Code instructions, and `claude` to log in

Set the agent's API keys (T2) into the vault as needed (e.g. a Codeberg token), and the vault_agent_ntfy_token (write-only ntfy token) — re-run the control role to deploy the token.

Sign the first agent cert — on the atomic laptop, YubiKey inserted:

scripts/yubikey-tool.sh sign-agent      # PIN + tap; fetches the pubkey, signs +1w, deploys to ctrl

Per session (operator)¶

Insert the YubiKey, then:

scripts/yubikey-tool.sh sign-agent           # one PIN + tap (skip if the cert is still valid)
ssh claude-agent@ctrl.${domain}              # lands in a normal shell — no auto-start
agent                                         # start Claude when you're ready

Login lands in a plain shell (no auto-start). The agent command (~/.bashrc.d/agent.sh → scripts/agent-session.sh) does the bootstrap: Tang-load the key → ntfy session-start → cd the repo (so both CLAUDE.md load) → claude; the key is flushed on exit. Because agent runs as a child of your login shell, Ctrl-Z parks Claude and fg resumes it. For a persistent / handoff-capable session (survives a disconnect; lets the operator hand off an authenticated shell in a tmux window), run tmux new -A -s agent before agent. No expiry reminder — the agent only runs when you ask it to, so a lapsed cert is self-evident: you just sign-agent again.

Task-scoped (JIT) signing — grant exactly what a task needs (ADDITIVE)¶

The principle: sign the agent a cert for the access the task at hand needs, short-lived — and do it on a separate key so it stacks on top of the standing cert instead of replacing it. The standing +1w cert is the routine floor; sign-agent-task is how you ELEVATE (a crown jewel like admin-ctrl/admin-freeipa) for one job. It signs the task key (~/.ssh/id_agent_task, its own Tang-sealed keypair) and deploys id_agent_task-cert.pub. load-agent-key.sh loads it alongside the standing cert (both sit in the agent; sshd selects the one whose principals match per host), so: - the task cert does not disturb baseline fleet access — no restore step, and - you list only the elevated principals the task needs (e.g. just admin-ctrl) — the standing cert still covers admin-mgmt,admin-servers,admin-dmz,admin-workstations.

Scope change (2026-07-12, operator decision): admin-workstations moved from task-only into the STANDING default — the agent does general maintenance on the workstations (alloy, workstation role) like the rest of the fleet, and the extra reach is root on three interactive boxes, not a crown jewel. Crown jewels (admin-ctrl/admin-freeipa/admin-hv) stay explicit task certs.

# interactive (offers the principals catalog): pick principals, TTL (default +4h), a task label
scripts/yubikey-tool.sh sign-agent-task
# or scriptable:  sign-agent-task <principals> <ttl> <label>
scripts/yubikey-tool.sh sign-agent-task admin-ctrl +4h fix-ctrl     # ctrl-root task (-> ssh almalinux@ctrl -> sudo)
scripts/yubikey-tool.sh sign-agent-task admin-freeipa +2h ipa-fix   # identity-core task
# on ctrl the agent loads it additively (or it auto-loads on the next agent session):
eval "$(scripts/load-agent-key.sh)"                                 # standing + task cert both loaded

- Key-id is claude-agent-task-<label>-<date> (audit-distinct from the standing claude-agent); the tool prints the serial + the one-line KRL-revoke reminder. - Audit = that distinctive key-id + serial, logged by sshd on every host the cert touches (the tang/Loki chokepoint), plus the agent-session ntfy. (sign-agent-task also best-effort ntfys the grant if SIGN_NTFY_URL+SIGN_NTFY_TOKEN_FILE are set on the signing host.) - Cutoff is unchanged — a task cert lapses on its short TTL; ssh-add -D / agent-killswitch.sh evict both keys; the KRL revokes either serial. To end an elevation early without touching the standing cert, ssh-add -d ~/.ssh/id_agent_task or delete id_agent_task-cert.pub. - Recommended practice: minimal standing + JIT elevation. Sign a task cert only to reach a crown jewel; it simply lapses (no cleanup needed).

Task-key bootstrap (one-time). The task keypair is generated + Tang-sealed exactly like the standing key (its passphrase fail-closes off-LAN), and grants nothing until the operator signs it:

PASS="$(head -c 32 /dev/urandom | base64 -w0)"
ssh-keygen -t ed25519 -C claude-agent-task@<instance> -N "$PASS" -f ~/.ssh/id_agent_task
printf "%s" "$PASS" | clevis encrypt tang "{\"url\":\"http://10.20.10.11\"}" -y > ~/.ssh/id_agent_task.pass.jwe
chmod 600 ~/.ssh/id_agent_task ~/.ssh/id_agent_task.pass.jwe; unset PASS

Alert queue — the agent consumes `lab-alerts` (2026-07-13)¶

The agent doesn't receive ntfy pushes, so alerts used to reach only the operator's phone. Operator directive: all alerts land in a queue the agent periodically drains and triages. Implementation (deliberately minimal — ntfy's message cache IS the queue):

Credential: a dedicated claude-agent ntfy user, read-only on lab-alerts (the agent consumes alerts; write is withheld so a compromised agent can't spoof or flood the operator's alert channel). Token at ~claude-agent/.config/lab/ntfy-read.token (0600).
Consumer: scripts/agent-alert-queue.sh — polls GET /lab-alerts/json?poll=1&since=<cursor>, prints one line per new message, advances the cursor (~/.local/state/lab/alert-cursor, exactly-once; delete to replay). Empty output = queue clean.
Cadence (rev. 2026-08-23 — polled doorbell; the streaming Monitor retired): the channel is agent-alert-scan.timer on ctrl (control role; every agent_alert_scan_interval_sec, default 120 s): each tick, /usr/local/bin/agent-alert-scan does a read-only peek of lab-alerts since the cursor (cursor untouched) and — only when a triage-tagged message is pending (keyword convention below) — spawns one headless triage agent (claude -p "/triage-alerts"), serialized by a flock. A quiet lab costs zero model invocations and no idle session; a real alarm costs ≤ one tick of latency. The spawned agent's cursor drain (agent-alert-queue.sh) stays the exactly-once durable record (and surfaces untagged messages). Failure honesty: a peek failure logs (ntfy down also trips the deadman); a crashed spawn is pushed to lab-alerts untagged (visible, never self-retriggering); a crash before the drain leaves the cursor behind, so the next tick retries.
Spawn-failure paging is CLASSIFIED and TRANSITION-ONLY (rev. 2026-08-31). The tick is 2 minutes, so any failure the agent cannot fix by itself used to page every tick: a 2h45m egress outage on 2026-08-31 produced 53 identical "Triage agent FAILED to run" pages, none of which stated the one fact that explained the whole night's alarm storm — the lab had no internet. Two fixes, both borrowed from checkers that already got this right: (1) classify before paging — the script probes SCAN_API_PROBE_URL and distinguishes "the API is unreachable from ctrl, so this is the lab's egress path, not the agent" (title "Triage agent deferred — lab EGRESS is down", pointing at runbook 05) from "the API was reachable and the spawn still died, so this is the agent"; (2) page on the transition, not the tick — the failure signature is persisted (~/.local/state/lab/triage-agent-failing, SCAN_FAIL_STATE) and a page is emitted only when it CHANGES, with one "Triage agent recovered" notice on the next success. This is fw-conformance-run.sh's failing-set pattern exactly: a persistent failure pages once, a NEW failure always pages, and an alarm that fires by construction — the kind the operator learns to dismiss — cannot form. Both notices stay untagged, so a failing spawn can never re-trigger its own doorbell. Fixed in the same change: rc was read inside if ! cmd; then, where $? is the negated status, so every historical failure page reported the nonsensical "exited rc=0"; it is now captured from a bare invocation. The script is installed root-owned (the agent account cannot rewrite what its own timer executes) and the spawn passes no permission flag — the unattended permission posture is an operator-owned setting in ~claude-agent/.claude (the same lever as the hub's auto mode), so the timer can never grant the agent more than the operator configured. This replaced the 2026-08-04 model (agent-alert-watch.sh streaming under a session-bound harness Monitor): the stream wedged in practice, needed a wedge-checker watching the watcher, and — decisively — required a hub session to sit open waiting (operator directive 2026-08-23: scan first, start an agent only if needed). The watch script remains in scripts/ as a manual fallback; arming it at session start is no longer a duty. The operator's phone remains the independent human channel; the digest + deadman planes are unchanged.
The spawn is MODEL-PINNED, and "no model quota" is its own failure class (rev. 2026-09-08). The spawn used to be a bare claude -p "/triage-alerts", inheriting the account's default model — which is the operator's own interactive hub model. So the doorbell was silently spending from, and gated by, the same per-model quota the operator spends from: when that limit was reached on 2026-09-07 15:23, every subsequent 2-minute tick spawned an agent that printed "You've reached your … limit" and exited rc=1. 803 consecutive dead spawns, ~27 hours with no triage at all, while the fleet itself was healthy. Three things were wrong and all three are fixed:
The spawn now pins its model — --model "$TRIAGE_MODEL" (default opus) plus --fallback-model "$TRIAGE_FALLBACK_MODEL" (default sonnet). A dispatched execution agent is Opus per the session topology above anyway, so this is the charter shape rather than a new rule; the point is that the doorbell no longer shares a quota pool with the hub, and no single model's limit can take triage down.
A quota failure names itself. The old classifier knew only egress-down vs the agent crashed, so a quota block — an operator-owned condition, fixable by nobody else — paged as a crash. The run log carries a definitive signature, so it is now matched FIRST (grep, no probe needed) and pages as "Triage agent BLOCKED — model usage limit", sig=model-limit, still transition-only.
Log retention runs on both paths. The tail -n +31 prune sat after the rc check, so the one case that writes a log every 2 minutes was the one case that never pruned — the outage left 804 run logs behind. It now runs immediately after rc is captured, before the classifier reads the log (the current log is newest, so it always survives). The detection that caught it was fleet-unit-failed, not the doorbell's own paging: the transition-only page fired once, correctly, and then — also correctly — stayed quiet, while the systemd-unit-failed rule flapped every ~20 minutes for a day. That is the intended division of labour, but it is worth stating: the doorbell cannot page about its own death loudly, so the unit-failed rule is what makes it observable.
The keyword convention (operator directive 2026-08-04): ntfy tag triage = agent-actionable. The wake decision is an ALLOWLIST evaluated entirely in shell — no model run is spent deciding whether a model run is needed. Publishers: Grafana adds the tag in its ntfy.payload template (roles/obs; resolve notifications share that template, so the watch additionally excludes the Grafana resolved title); fleet scripts opt in by adding triage to the Tags header of failure messages only (routine/positive notices — session start, delegated merge, host-CA signed, digest all-clear — stay untagged). Tagged as of 2026-08-08: obs-deadman.sh (monitoring DOWN), host-ca-autosign.sh (FAIL-CLOSED / refused-unsigned / sign-failed / deploy-errors / HOST KEY CHANGED), lab-drift-check.sh (drift detected / failed-to-run), proton-endpoint-monitor.sh (tunnel DOWN), and the RPM/DEB mirror sync scripts (FAILED / had-errors). lab-daily-digest.sh was tagged at prio≥4 until 2026-09-06 and is now deliberately UNTAGGED: with the one-page-then-digest policy (runbook 27) every item in the digest already woke the agent when it began, so a tagged digest was a guaranteed daily wake to re-state known conditions. Mechanism: the shared ntfy() helper took an optional 4th tags arg (local tags=(); [ -n "${4:-}" ] && tags=(-H "Tags: $4") — word-split-safe); update-availability nags (image/appliance-update-check) stay untagged deliberately — actionable but not urgent, they reach the agent via the drain. The agent's own Agent triage: resolutions never carry the tag. Tradeoff, accepted: an untagged alarm gets no real-time wake — it still reaches the phone and the next cursor drain; when adding a NEW failure publisher, adding the tag is part of the change (negative-test both directions: tagged wakes, untagged doesn't).
The operator can wake the agent on demand (2026-08-15): forward an alert to the triage@ XMPP contact (runbook 53 §The ntfy bridge). The bot republishes the forwarded text to lab-alerts tagged triage (title Operator triage request, write-only token) — the same allowlist wake as any tagged alarm, no new mechanism — and alarms themselves reach the phone via the alerts@ relay, so the whole loop (alarm → forward → agent → Agent triage: resolution) lives in the chat app.
Progress talks back (2026-08-23): every triage run narrates itself into the triage@ conversation. The working agent posts one-line checkpoints (picked up / root cause / fix verified / outcome) via scripts/agent-triage-status.sh — a write-only grant on the dedicated triage-status topic, which the triage@ bot relays into the chat (rb53 §bridge status leg). So the thread reads request → progress → outcome, with the formal Agent triage: resolution still arriving via alerts@ exactly as before; progress never touches lab-alerts (the alarm bus stays clean, and the checkpoint contract lives in the triage-alerts skill). Scope, per operator decision 2026-08-23: all triage runs narrate — Grafana-fired alarms the operator never forwarded included — so the triage@ chat is the live triage log. Status posts are best-effort (|| true): a dead chat plane never blocks a fix.

Alert triage → resolution (2026-08-04)¶

Operator directive: draining is not enough — every alarm gets troubleshot to root cause, and the outcome is published where the operator reads it. Until 2026-09-06 that was lab-alerts itself (the resolution under the alarm on the phone); since then it is the chat — operator directive: "report feedback here in the future instead of alerts" — so the resolution is the closing reply of the triage@ thread the alarm opened, and lab-alerts carries only alarms, the digest and system notices. The /triage-alerts command (.claude/commands/triage-alerts.md) carries the full protocol; the publisher is scripts/agent-alert-resolve.sh.

Publisher: agent-alert-resolve.sh <resolved|cleared|action-needed> "<issue>" "<body>" → the triage-status topic (relayed into the chat by the triage@ bridge, runbook 53; AGENT_NTFY_URL overrides), title prefix Agent triage: (the bridge relays title + body; the drain loop still skips that prefix should one ever land on lab-alerts — negative-test this filter when changing either side). Mapping: resolved = fixed & verified (prio 3, robot,white_check_mark); cleared = alarm factually true but no defect, state reset on the host (prio 3, robot,broom); action-needed = needs the operator — keys, T2 merge, hardware (prio 4, robot,warning). Non-zero exit if the publish fails (a silently-dropped notification is a control that turned itself off).
Two tokens, deliberately split: the drain uses the READ-ONLY claude-agent token (~/.config/lab/ntfy-read.token); the publisher uses the WRITE-ONLY agent-notify token (/etc/lab/agent-ntfy.token, 0640 group claude-agent). The split is an anti-spoofing control — never merge them into one credential.
Autonomy line (operator decision 2026-08-04): in-reach reversible host fixes (restart / reset-failed / disable a broken unit, re-converge from main) are applied immediately and reported after; anything needing the operator's keys, a T2 merge, or hardware becomes an action-needed alert instead. Alarms are never quieted at the Grafana/collector layer as a shortcut — those are T2 design changes, only for a genuinely wrong alarm class.
One message per distinct root cause, not per firing — and no message at all for a repeat of an already-triaged, unchanged condition (2026-09-06). Grafana now pages each alarm ONCE (repeat_interval 120h, re-paging at most every ~5 days — runbook 27 §One page per alarm) and the 04:00 digest carries it until it clears, so a standing alarm the agent has already root-caused gets a one-line chat note at most, never a fresh triage or a daily action-needed.

Cutoff drills (prove all three)¶

Expiry (passive): the cert is +1w; once it lapses the agent can't auth anywhere until you re-sign. (ssh-keygen -L -f id_agent-cert.pub shows the validity window.)
KRL (durable, instant): read the serial (ssh-keygen -L -f ~claude-agent/.ssh/id_agent-cert.pub | grep Serial), add it to the KRL, then scripts/push-krl.sh → the agent is refused fleet-wide, even mid-validity. Re-sign to restore.
Live-kill (stop now): sudo scripts/agent-killswitch.sh on ctrl → terminates the claude-agent user (the running claude, its ssh-agent — evicting the in-RAM key — and any tmux) + ntfys. Use for a runaway/compromised session; follow with the KRL drill if it's a real compromise.

Troubleshooting — diagnose before you assume a cutoff¶

Fleet-wide Permission denied (publickey) in a tool shell is almost never a real cutoff — it's a missing SSH_AUTH_SOCK. The agent's key lives in an ssh-agent, and a freshly spawned tool shell doesn't inherit the environment that points ssh at it, so every host refuses the (now keyless) connection at once. The tell: ssh -v shows Server accepts key: … id_agent and then a denial at the signing step — the host knows the cert, the client just never presented a signature. Fix by exporting the socket before sshing: export SSH_AUTH_SOCK=$HOME/.ssh/agent.sock (then ssh-add -l should list id_agent). Only after that check comes back clean should you suspect the durable cutoffs below (KRL / expiry / a failed Tang unseal) — a genuine cutoff denies before "Server accepts key", or the key simply isn't loaded at all.

DR & residual risk¶

Lose tang1: the agent key won't load (fail-closed) → the agent can't start; you still reach everything via your own 9a login cert / almalinux. Rebuild tang1 (rb24); re-seal id_agent.pass.jwe from the KeePass passphrase.
Lose ctrl: rebuild (rb15) + re-run the control role; redo this bootstrap (the agent key/cert are runtime, not escrowed beyond the KeePass passphrase). Automation + your own access are unaffected.
Operator never locked out (the asymmetry): revoking/expiring the agent cert never touches your YubiKey access or the almalinux break-glass — the agent is killable; you are not.
Accepted residual: a live agent session has broad fleet admin (minus the identity core). It's bounded by short validity + KRL + the live-kill + full audit, and irreversible/outward/destructive actions still require explicit operator approval (the lab's standing rule).

Verification¶

From the agent tmux: ansible obs -m ping and ansible caddy -m ping succeed as the agent cert (confirm key-id claude-agent in the target's journalctl -u sshd); ansible freeipa -m ping fails (identity core correctly out of scope).
Audit: the agent's logins (key-id claude-agent) + sudo show in Loki; the session-start ntfy fired.
All three cutoff drills above behave as described.

Repo integrity — signed commits¶

ctrl (and your laptop) run tooling straight out of this repo — yubikey-tool.sh drives the offline CA, load-agent-key.sh handles the fleet key, the timers converge the fleet. So before you git pull and run any of it (especially with the YubiKey inserted), you want assurance the code is authentic. We do that with SSH-signed commits, reusing the keys we already have — no new key type.

Trust file (in-repo): scripts/allowed_signers lists the SSH pubkeys allowed to sign commits. The tracked copy ships as placeholders; the real rows are instance material (overlay), so a clone of this repo trusts nobody until an operator populates it.

The agent's key is deliberately NOT an allowed signer (decided 2026-07-26). The agent signs its commits with id_agent — so git log --show-signature attributes them — but that signature is attribution, not authorisation: it stays U (signed, untrusted). Trusting it would let an injected agent produce a tip the host-CA autosigner accepts and advances /opt/myos to, which is precisely the boundary runbook 26's guard exists to hold. Don't add id_agent here to make a red check turn green. What legitimises an agent-authored change is the operator-approved merge, signed by Forgejo — see the two-signer table in runbook 26.

Configure a clone to sign + verify (once):

git config gpg.format ssh
git config user.signingkey ~/.ssh/<your-signing-key>.pub    # laptop: your YubiKey-backed key; ctrl agent: ~/.ssh/id_agent.pub
git config commit.gpgsign true                              # sign every commit
git config gpg.ssh.allowedSignersFile "$(git rev-parse --show-toplevel)/scripts/allowed_signers"

- Operator: nothing to do — you do not have to sign commits (decided 2026-07-27). Branch protection means every tip of main is a Forgejo-created merge commit, signed by Forgejo, and that is what the host-CA guard verifies (runbook 26). Your control point is the approval click. scripts/allowed_signers stays placeholders; the config block above applies only if you want your own hand-made commits to verify — then add your pubkey line (<email> <keytype> AAAA… label) to the overlay copy and re-seal the bundle (runbook 33). A YubiKey PIV key works, but note this lab has already hit PKCS11-into-ssh-agent friction (ykcs11 hides retired slots; opensc refused to load) — a FIDO2 ed25519-sk key is the smoother path. The agent signs with id_agent via ssh-agent, but is not in that file (box above).

Verify before you trust/run — the whole point:

git pull --ff-only
git log --show-signature -1        # or: git verify-commit HEAD   -> "Good git signature for <signer>"
git config merge.verifySignatures true   # optional: refuse to merge unsigned commits locally

A G in git log --format='%G?' = good signature from an allowed signer; N/E = unsigned/bad — don't run the tool from an unverified checkout.

Enforce on main (Forgejo): main is branch-protected — direct pushes blocked and 1 approval required, so nothing lands unreviewed. Forgejo (not the merger) then signs the merge commit, which is what makes the resulting tip verifiable to the host-CA guard. The agent cannot self-approve (Forgejo blocks that by identity, unconditionally) and cannot merge T2 at all (protected_file_patterns + the approvals whitelist, runbook 51 §3.3): it opens a PR and waits. Optionally also register each human signer's SSH key on their Forgejo account as a signing key so the web UI shows a "Verified" badge.

The approval requirement and the trust placed in Forgejo's merge signature are one mechanism, not two. Dropping the approval turns "an approved PR was merged" into "somebody merged" — and the agent can merge. Remove one, remove the other.

Tiered approval does not weaken this, on one condition: every path the host-CA signer executes stays T2, so those merges still mechanically require the operator's own approval. The delegated tiers (docs/**, README.md, TODO.md) contain nothing the signer runs. Promote anything under ansible/** and Forgejo must leave the signer's trust store in the same change — runbooks 51 §3.4 and 26.

Transport note: the internal git remote is http://10.20.30.30:3000 — plaintext, but it rides the WireGuard tunnel off-LAN and the trusted LAN on-prem, and signed commits give the integrity that matters here (forgejo SSH :2222 is firewalled from the workstation/roaming zones by design; see the DMZ isolation in CLAUDE.md).

Landing infra PRs¶

The agent works on a branch and opens a PR against main. What happens next depends on what the PR touches — approval is routed by path, per the approval tiers in runbook 51 (the governing document; this is the operating summary):

Tier	Paths	Approver	Merged by
T0	`docs/**`, `README.md`, `TODO.md`	`casey` or `agent-reviewer`	the agent, `scripts/forgejo-merge-pr.sh`
T1	(empty for now)	`casey` or `agent-reviewer`	the agent
T2 (default — anything unlisted)	`scripts/`, `.forgejo/`, `ansible/`, `provision/`, `tests/`, `CLAUDE.md`, `.claude/`, `.gitignore`	`casey`	`casey`, in the Forgejo UI

scripts/forgejo-merge-pr.sh <PR#>      # merge-commit only — no method argument

- Merge-only. The squash|rebase argument is gone (2026-08-02): the repo rejects both styles anyway (a non-Forgejo-created tip silently freezes the host-CA signer, rb26), and accepting an argument the server refuses reads as a script bug. An extra argument is a usage error, exit 2. - The script routes, it does not decide. Before merging it classifies the PR's changed paths, resolves the required approver, and requires an APPROVED review on the exact head commit that is neither dismissed nor stale. T2 it refuses outright — and branch protection (protected_file_patterns) blocks the agent's merge server-side regardless, which is where the T2 guarantee actually lives. Delegated merges push an ntfy to the operator and show up in the 04:00 digest. - Why the helper and nothing else. A raw git push origin main and a raw call to the Forgejo merge API are both deliberately blocked — pushing/publishing to main is gated. forgejo-merge-pr.sh is a narrow, allowlisted escape hatch (it's the only command permitted for this in .claude/settings.json): it hits just the control-plane myos repo's /merge endpoint for one numeric PR — it can't push arbitrary refs, choose a merge style, target another repo, or advance main any other way. - Token custody. The merge token is read from ~/.config/forgejo-agent.token — the AGENT's own ctrl-agent account, 0600, outside the repo tree and never on the command line; the script fails closed if it's missing or unreadable. The delegated-review identity is a separate account with its own token (~/.config/forgejo-reviewer.token, used only by scripts/forgejo-submit-review.sh) — see runbook 51 §4 for what that separation does and does not buy. - After it lands. The PR's signed commits (see above) mean main advances only with a known signature, and the docs CD picks up the change on the path filter; confirm the publish if the PR touched docs/**, runbooks/**, or CLAUDE.md.

28 — AI agent operating model (PAW on ctrl, own YubiKey-signed cert)¶