28 — AI agent operating model (PAW on `ctrl`, own YubiKey-signed cert)¶

How the IT agent (Claude) is run as the system administrator: from a Privileged Access Workstation (ctrl), holding its own fleet credential — a keypair on ctrl plus a short-lived user cert you sign from the YubiKey — never the automation control cert.

Why this shape. Best practice for a privileged automation/agent identity is the PAW + central-identity-gate pattern: the agent operates from one hardened, audited box; its credentials live on that box, not on a roaming client; access originates from the offline access-CA (your YubiKey 9c); sensitive actions stay human-gated; everything is audited; revocation is central. This reaches every host incl. DMZ (the user-CA is trusted fleet-wide) without enrolling DMZ in IPA and without any online access-CA — so it respects both the DMZ-isolation and the split-CA invariants (CLAUDE.md). See the "Credential custody model" section of CLAUDE.md for the four-tier picture.

A dedicated IPA identity for the agent (named user + HBAC + IPA sudo — the human-operator model) was declined 2026-07-07: an injectable agent must minimise standing privilege (elevation stays JIT), keep a uniform fail-closed cutoff (one cert kill-switch, not a split ipa-disable+KRL), and stay IPA-independent (so it can fix IPA when IPA is down). IPA-as-PAM is reserved for humans. Full rationale in the CLAUDE.md custody model.

Piece	What	Where
PAW	Claude runs as the `claude-agent` account, started manually with `agent` (run inside `tmux` for a persistent session)	`ctrl` (`roles/control`)
Agent key	own keypair; passphrase Clevis+Tang-sealed to tang1 (fail-closed off-LAN)	`claude-agent@ctrl:~/.ssh/id_agent` (+ `.pass.jwe`)
Agent cert (standing)	YubiKey 9c-signed; key-id `claude-agent`; principals `admin-mgmt,admin-servers,admin-dmz,admin-workstations` (NOT `admin-freeipa`/`admin-ctrl`/`admin-hv`); `+1w`	signed on demand by `scripts/yubikey-tool.sh sign-agent`
Task cert (JIT, ADDITIVE)	per-task elevation on a separate Tang-sealed key (`id_agent_task`), loaded ALONGSIDE the standing cert so it never disturbs baseline access; elevated principals (incl. crown jewels) for a short TTL (default `+4h`); key-id `claude-agent-task-<label>-<date>`	`scripts/yubikey-tool.sh sign-agent-task` — see "Task-scoped (JIT) signing"
Reach	standing: all zones except the identity/CA core and ctrl; lands as `almalinux` via the principal→account map, attributed by the cert key-id. A signed task cert reaches whatever its principals grant (e.g. `admin-ctrl` → ctrl-root via `almalinux`+sudo)	standing: fleet-wide (incl. DMZ), not ctrl; task: per-grant
Cutoff	EXPIRY (lapses in ~1w) · KRL (`scripts/push-krl.sh`, instant) · LIVE-KILL (`scripts/agent-killswitch.sh`)	operator/YubiKey-controlled

The agent runs ansible with its own cert (not the control cert) and uses a TPM-sealed vault credential of its own (/etc/lab/vault-agent.cred). Two distinct credentials are in play: you authenticate to the claude-agent account with your 9a login cert (ctrl's auth_principals/claude-agent); the agent reaches the fleet with its own id_agent cert.

Containment: minimal standing access, operator-signed elevation (JIT)¶

The model is not "the agent may never root its PAW" — it's least standing privilege + just-in-time, operator-signed, short-lived elevation. The agent's STANDING cert (+1w, admin-mgmt,servers,dmz) gets root on the fleet it services (become on the targets) but cannot root its own control node, enforced by THREE independent facts that hold for the standing cert: (1) the claude-agent account has no sudo on ctrl (the agent runs as claude-agent); (2) it can't read ctrl's privileged material (the almalinux control key, /etc/lab/host-ca.jwe, vault.cred — it only traverses to its own vault-agent.cred); (3) the standing fleet cert can't hop to almalinux@ctrl — ctrl is a crown-jewel host (admin-ctrl in principals.yml, like freeipa), so auth_principals/almalinux lists admin,admin-ctrl only; the agent's admin-mgmt is excluded.

Elevation is granted per task, not denied absolutely. When a task needs more (incl. ctrl-root or the identity core), the operator signs a short-TTL task cert with exactly those principals (sign-agent-task, below). The ctrl-root path: an admin-ctrl task cert → ssh almalinux@ctrl (the only SSH that lands there) → sudo -i (almalinux sudo is codified in the control role — NOPASSWD, gated upstream by who can reach almalinux@ctrl). The grant lapses (short TTL) and is KRL-revocable, so the kill-switch stays real. Root SSH stays off (PermitRootLogin no); root on ctrl is otherwise console-only (operator + KeePass). The standing cert still can't reach it — only a deliberately-signed task cert can. admin (break-glass) reaches almalinux@ctrl too. ctrl is ansible_connection=local + almalinux has sudo, so it self-manages in place.

One-time bootstrap (operator)¶

Prereqs: ctrl built (runbook 15), tang1 up (runbook 24), the control role applied (creates the claude-agent account, ~/.ssh, the SSH config, auth_principals/claude-agent, the ntfy token). Then, on ctrl as root/almalinux:

Generate the agent keypair + Tang-seal its passphrase (mirrors the control key, rb15/26):

sudo -u claude-agent bash -lc '
  PASS=$(openssl rand -base64 32)
  ssh-keygen -t ed25519 -C claude-agent@<instance> -N "$PASS" -f ~/.ssh/id_agent
  printf "%s" "$PASS" | clevis encrypt tang "{\"url\":\"http://10.20.10.11\"}" -y > ~/.ssh/id_agent.pass.jwe
  chmod 600 ~/.ssh/id_agent ~/.ssh/id_agent.pass.jwe
  echo "ALSO store this passphrase in KeePass lab.kdbx as \"claude-agent SSH key\" (break-glass):"
  printf "%s\n" "$PASS"
'

Record the passphrase in KeePass lab.kdbx (break-glass), then clear your scrollback.

Seal the agent's vault credential to ctrl's TPM (same vault password as almalinux's; the operator approved the agent+ansible sharing the vault):
```
umask 077; printf '%s' "$VAULT_PASSPHRASE" \
  | sudo systemd-creds encrypt --name=lab-vault --with-key=tpm2 - /etc/lab/vault-agent.cred
sudo chgrp claude-agent /etc/lab/vault-agent.cred && sudo chmod 640 /etc/lab/vault-agent.cred
```
Confirm claude-agent can decrypt it (TPM device access — add claude-agent to the tss group if systemd-creds decrypt is denied, as for almalinux in rb15). Re-run the control role so its perms/warn tasks settle.

Install Claude Code for claude-agent + log it in (interactive, once):

sudo -u claude-agent -i      # then install per the current Claude Code instructions, and `claude` to log in

Set the agent's API keys (T2) into the vault as needed (e.g. a Codeberg token), and the vault_agent_ntfy_token (write-only ntfy token) — re-run the control role to deploy the token.

Sign the first agent cert — on the atomic laptop, YubiKey inserted:

scripts/yubikey-tool.sh sign-agent      # PIN + tap; fetches the pubkey, signs +1w, deploys to ctrl

Per session (operator)¶

Insert the YubiKey, then:

scripts/yubikey-tool.sh sign-agent           # one PIN + tap (skip if the cert is still valid)
ssh claude-agent@ctrl.${domain}              # lands in a normal shell — no auto-start
agent                                         # start Claude when you're ready

Login lands in a plain shell (no auto-start). The agent command (~/.bashrc.d/agent.sh → scripts/agent-session.sh) does the bootstrap: Tang-load the key → ntfy session-start → cd the repo (so both CLAUDE.md load) → claude; the key is flushed on exit. Because agent runs as a child of your login shell, Ctrl-Z parks Claude and fg resumes it. For a persistent / handoff-capable session (survives a disconnect; lets the operator hand off an authenticated shell in a tmux window), run tmux new -A -s agent before agent. No expiry reminder — the agent only runs when you ask it to, so a lapsed cert is self-evident: you just sign-agent again.

Task-scoped (JIT) signing — grant exactly what a task needs (ADDITIVE)¶

The principle: sign the agent a cert for the access the task at hand needs, short-lived — and do it on a separate key so it stacks on top of the standing cert instead of replacing it. The standing +1w cert is the routine floor; sign-agent-task is how you ELEVATE (a crown jewel like admin-ctrl/admin-freeipa) for one job. It signs the task key (~/.ssh/id_agent_task, its own Tang-sealed keypair) and deploys id_agent_task-cert.pub. load-agent-key.sh loads it alongside the standing cert (both sit in the agent; sshd selects the one whose principals match per host), so: - the task cert does not disturb baseline fleet access — no restore step, and - you list only the elevated principals the task needs (e.g. just admin-ctrl) — the standing cert still covers admin-mgmt,admin-servers,admin-dmz,admin-workstations.

Scope change (2026-07-12, operator decision): admin-workstations moved from task-only into the STANDING default — the agent does general maintenance on the workstations (alloy, workstation role) like the rest of the fleet, and the extra reach is root on three interactive boxes, not a crown jewel. Crown jewels (admin-ctrl/admin-freeipa/admin-hv) stay explicit task certs.

# interactive (offers the principals catalog): pick principals, TTL (default +4h), a task label
scripts/yubikey-tool.sh sign-agent-task
# or scriptable:  sign-agent-task <principals> <ttl> <label>
scripts/yubikey-tool.sh sign-agent-task admin-ctrl +4h fix-ctrl     # ctrl-root task (-> ssh almalinux@ctrl -> sudo)
scripts/yubikey-tool.sh sign-agent-task admin-freeipa +2h ipa-fix   # identity-core task
# on ctrl the agent loads it additively (or it auto-loads on the next agent session):
eval "$(scripts/load-agent-key.sh)"                                 # standing + task cert both loaded

- Key-id is claude-agent-task-<label>-<date> (audit-distinct from the standing claude-agent); the tool prints the serial + the one-line KRL-revoke reminder. - Audit = that distinctive key-id + serial, logged by sshd on every host the cert touches (the tang/Loki chokepoint), plus the agent-session ntfy. (sign-agent-task also best-effort ntfys the grant if SIGN_NTFY_URL+SIGN_NTFY_TOKEN_FILE are set on the signing host.) - Cutoff is unchanged — a task cert lapses on its short TTL; ssh-add -D / agent-killswitch.sh evict both keys; the KRL revokes either serial. To end an elevation early without touching the standing cert, ssh-add -d ~/.ssh/id_agent_task or delete id_agent_task-cert.pub. - Recommended practice: minimal standing + JIT elevation. Sign a task cert only to reach a crown jewel; it simply lapses (no cleanup needed).

Task-key bootstrap (one-time). The task keypair is generated + Tang-sealed exactly like the standing key (its passphrase fail-closes off-LAN), and grants nothing until the operator signs it:

PASS="$(head -c 32 /dev/urandom | base64 -w0)"
ssh-keygen -t ed25519 -C claude-agent-task@<instance> -N "$PASS" -f ~/.ssh/id_agent_task
printf "%s" "$PASS" | clevis encrypt tang "{\"url\":\"http://10.20.10.11\"}" -y > ~/.ssh/id_agent_task.pass.jwe
chmod 600 ~/.ssh/id_agent_task ~/.ssh/id_agent_task.pass.jwe; unset PASS

Alert queue — the agent consumes `lab-alerts` (2026-07-13)¶

The agent doesn't receive ntfy pushes, so alerts used to reach only the operator's phone. Operator directive: all alerts land in a queue the agent periodically drains and triages. Implementation (deliberately minimal — ntfy's message cache IS the queue):

Credential: a dedicated claude-agent ntfy user, read-only on lab-alerts (the agent consumes alerts; write is withheld so a compromised agent can't spoof or flood the operator's alert channel). Token at ~claude-agent/.config/lab/ntfy-read.token (0600).
Consumer: scripts/agent-alert-queue.sh — polls GET /lab-alerts/json?poll=1&since=<cursor>, prints one line per new message, advances the cursor (~/.local/state/lab/alert-cursor, exactly-once; delete to replay). Empty output = queue clean.
Cadence: the agent arms an in-session recurring check (~30 min) that runs the script and troubleshoots anything new. The schedule is session-bound — re-arming it is part of the agent's session start (see memory). The operator's phone remains the independent human channel; the digest + deadman planes are unchanged.

Cutoff drills (prove all three)¶

Expiry (passive): the cert is +1w; once it lapses the agent can't auth anywhere until you re-sign. (ssh-keygen -L -f id_agent-cert.pub shows the validity window.)
KRL (durable, instant): read the serial (ssh-keygen -L -f ~claude-agent/.ssh/id_agent-cert.pub | grep Serial), add it to the KRL, then scripts/push-krl.sh → the agent is refused fleet-wide, even mid-validity. Re-sign to restore.
Live-kill (stop now): sudo scripts/agent-killswitch.sh on ctrl → terminates the claude-agent user (the running claude, its ssh-agent — evicting the in-RAM key — and any tmux) + ntfys. Use for a runaway/compromised session; follow with the KRL drill if it's a real compromise.

Troubleshooting — diagnose before you assume a cutoff¶

Fleet-wide Permission denied (publickey) in a tool shell is almost never a real cutoff — it's a missing SSH_AUTH_SOCK. The agent's key lives in an ssh-agent, and a freshly spawned tool shell doesn't inherit the environment that points ssh at it, so every host refuses the (now keyless) connection at once. The tell: ssh -v shows Server accepts key: … id_agent and then a denial at the signing step — the host knows the cert, the client just never presented a signature. Fix by exporting the socket before sshing: export SSH_AUTH_SOCK=$HOME/.ssh/agent.sock (then ssh-add -l should list id_agent). Only after that check comes back clean should you suspect the durable cutoffs below (KRL / expiry / a failed Tang unseal) — a genuine cutoff denies before "Server accepts key", or the key simply isn't loaded at all.

DR & residual risk¶

Lose tang1: the agent key won't load (fail-closed) → the agent can't start; you still reach everything via your own 9a login cert / almalinux. Rebuild tang1 (rb24); re-seal id_agent.pass.jwe from the KeePass passphrase.
Lose ctrl: rebuild (rb15) + re-run the control role; redo this bootstrap (the agent key/cert are runtime, not escrowed beyond the KeePass passphrase). Automation + your own access are unaffected.
Operator never locked out (the asymmetry): revoking/expiring the agent cert never touches your YubiKey access or the almalinux break-glass — the agent is killable; you are not.
Accepted residual: a live agent session has broad fleet admin (minus the identity core). It's bounded by short validity + KRL + the live-kill + full audit, and irreversible/outward/destructive actions still require explicit operator approval (the lab's standing rule).

Verification¶

From the agent tmux: ansible obs -m ping and ansible caddy -m ping succeed as the agent cert (confirm key-id claude-agent in the target's journalctl -u sshd); ansible freeipa -m ping fails (identity core correctly out of scope).
Audit: the agent's logins (key-id claude-agent) + sudo show in Loki; the session-start ntfy fired.
All three cutoff drills above behave as described.

Repo integrity — signed commits¶

ctrl (and your laptop) run tooling straight out of this repo — yubikey-tool.sh drives the offline CA, load-agent-key.sh handles the fleet key, the timers converge the fleet. So before you git pull and run any of it (especially with the YubiKey inserted), you want assurance the code is authentic. We do that with SSH-signed commits, reusing the keys we already have — no new key type.

Trust file (in-repo): scripts/allowed_signers lists the SSH pubkeys allowed to sign commits. The tracked copy ships as placeholders; the real rows are instance material (overlay), so a clone of this repo trusts nobody until an operator populates it.

The agent's key is deliberately NOT an allowed signer (decided 2026-07-26). The agent signs its commits with id_agent — so git log --show-signature attributes them — but that signature is attribution, not authorisation: it stays U (signed, untrusted). Trusting it would let an injected agent produce a tip the host-CA autosigner accepts and advances /opt/myos to, which is precisely the boundary runbook 26's guard exists to hold. Don't add id_agent here to make a red check turn green. What legitimises an agent-authored change is the operator-approved merge, signed by Forgejo — see the two-signer table in runbook 26.

Configure a clone to sign + verify (once):

git config gpg.format ssh
git config user.signingkey ~/.ssh/<your-signing-key>.pub    # laptop: your YubiKey-backed key; ctrl agent: ~/.ssh/id_agent.pub
git config commit.gpgsign true                              # sign every commit
git config gpg.ssh.allowedSignersFile "$(git rev-parse --show-toplevel)/scripts/allowed_signers"

- Operator: nothing to do — you do not have to sign commits (decided 2026-07-27). Branch protection means every tip of main is a Forgejo-created merge commit, signed by Forgejo, and that is what the host-CA guard verifies (runbook 26). Your control point is the approval click. scripts/allowed_signers stays placeholders; the config block above applies only if you want your own hand-made commits to verify — then add your pubkey line (<email> <keytype> AAAA… label) to the overlay copy and re-seal the bundle (runbook 33). A YubiKey PIV key works, but note this lab has already hit PKCS11-into-ssh-agent friction (ykcs11 hides retired slots; opensc refused to load) — a FIDO2 ed25519-sk key is the smoother path. The agent signs with id_agent via ssh-agent, but is not in that file (box above).

Verify before you trust/run — the whole point:

git pull --ff-only
git log --show-signature -1        # or: git verify-commit HEAD   -> "Good git signature for <signer>"
git config merge.verifySignatures true   # optional: refuse to merge unsigned commits locally

A G in git log --format='%G?' = good signature from an allowed signer; N/E = unsigned/bad — don't run the tool from an unverified checkout.

Enforce on main (Forgejo): main is branch-protected — direct pushes blocked and 1 approval required, so nothing lands without the operator reviewing it. Forgejo (not the merger) then signs the merge commit, which is what makes the resulting tip verifiable to the host-CA guard. The agent therefore cannot self-merge to main: it opens a PR and waits. Optionally also register each human signer's SSH key on their Forgejo account as a signing key so the web UI shows a "Verified" badge.

The approval requirement and the trust placed in Forgejo's merge signature are one mechanism, not two. Dropping the approval turns "an approved PR was merged" into "somebody merged" — and the agent can merge. Remove one, remove the other.

Transport note: the internal git remote is http://10.20.30.30:3000 — plaintext, but it rides the WireGuard tunnel off-LAN and the trusted LAN on-prem, and signed commits give the integrity that matters here (forgejo SSH :2222 is firewalled from the workstation/roaming zones by design; see the DMZ isolation in CLAUDE.md).

Landing infra PRs¶

The agent works on a branch, opens a PR against main, and merges it only once the operator has approved it — a merge to main is an outward-facing act (it fires the myos.io docs CD, so what lands is what publishes), which the charter reserves for operator sign-off. When cleared, the agent lands the PR with exactly one command:

scripts/forgejo-merge-pr.sh <PR#> [merge|squash|rebase]   # default: merge

- Why the helper and nothing else. A raw git push origin main and a raw call to the Forgejo merge API are both deliberately blocked — pushing/publishing to main is gated. forgejo-merge-pr.sh is a narrow, allowlisted escape hatch (it's the only command permitted for this in .claude/settings.json): it hits just the control-plane myos repo's /merge endpoint for one numeric PR — it can't push arbitrary refs, target another repo, or advance main any other way. So the operator-gate holds while the agent can still land an approved change unaided. - Token custody. The merge token is read from ~/.config/forgejo-agent.token (the AGENT's own account; 0600, gitignored, never in the repo or on the command line) — the script fails closed if it's missing or unreadable. - After it lands. The PR's signed commits (see above) mean main advances only with a known signature, and the docs CD picks up the change on the path filter; confirm the publish if the PR touched docs/**, runbooks/**, or CLAUDE.md.

28 — AI agent operating model (PAW on ctrl, own YubiKey-signed cert)¶