43 — Lab secrets store (`lab-secrets` repo → vault overlay)¶

Status: built 2026-07-18. The instance's Ansible secrets live in a dedicated lab-secrets Forgejo repo (control-plane git.${domain}, lab_org/lab-secrets), separate from the myos IaC repo. A control-role oneshot pulls it and overlays the encrypted files into the checkouts' ansible/group_vars/all/ before any converge.

An instance file can be covered by NEITHER store (2026-08-02)¶

There are two instance stores, and a file can fall between them:

store	mechanism	covers
Tang-sealed bundle	`/etc/lab/instance-material.tar.gz.jwe` → `load-instance-material.sh`	`zz-local`, most `host_vars`, `inventory.d`, role files
lab-secrets repo	`load-lab-secrets.sh`	`vault.yml` + `*.vault.yml` + `lab.kdbx`

ansible/host_vars/mirror.yml was in neither — gitignored (so not in git), absent from the bundle, and not matching *.vault.yml so the secrets sync skipped it. It held an ansible-vault !vault block, mirror_ntfy_token.

This is the second instance of the same bug. load-lab-secrets.sh lines 79-80 record the first: vault.yml does not match *.vault.yml, was never synced, and drifted per checkout — which caused the 2026-07-18 TSIG desync.

Why it was nasty: it failed green. mirror_ntfy_token defaulted to "" and the task writing the token file was gated on length > 0. Losing the file (ctrl rebuild, fresh clone, a worktree) produced a successful converge whose only casualty was the mirror silently ceasing to authenticate to ntfy, which runs auth-default-access: deny-all — so supply-chain mirror sync failures stopped being reported, with nothing anywhere saying so.

Fixed by moving the token to group_vars/all/mirror-ntfy.vault.yml — the mechanism that already exists for secrets — rather than widening the Tang bundle, which would create a second divergent store for the same class of data. rpm_mirror now asserts instead of skipping: empty token + configured mirror_ntfy_url is a hard error, and clearing mirror_ntfy_url is the supported way to run without notifications.

Audit: which instance files is no mechanism managing?¶

tar restores whole-second mtimes; locally-written files keep nanoseconds. So bundle-sourced files show .000000000 and unmanaged ones do not:

cd ~/src/myos && git ls-files --others --ignored --exclude-standard ansible/ \
  | xargs stat -c '%y %n' | grep -v '\.000000000'

Expect exactly the lab-secrets files. Anything else is orphaned.

Why a separate repo¶

The myos repo is the IaC + the source of the public docs (myos.io), so secrets must never live in it — *.vault.yml and lab.kdbx are gitignored there. But gitignored files aren't reproducible: a fresh checkout has none, so a converge runs with undefined vault vars (this is exactly how the Seafile OIDC secret went missing and broke the sso converge). lab-secrets is the reproducible, version-controlled home for those files.

What it holds¶

The ansible-vault-encrypted *.vault.yml files (the automation secrets: authelia, oidc client secrets, origin WireGuard key, DNSSEC escrow, etc.).
The legacy monolithic vault.yml (DNS/TSIG among others). It is $ANSIBLE_VAULT-encrypted like the rest, but note it does not match a *.vault.yml glob — see the warning below.
lab.kdbx — the KeePass break-glass DB (human tier).
agent-tokens.vault.yml (added 2026-08-22) — the AI agent's three Forgejo API tokens: vault_agent_forgejo_token (ctrl-agent @ the control-plane instance), vault_agent_forgejo_reviewer_token (the delegated T0/T1 reviewer) and vault_agent_forgejo_pub_token (ctrl-agent @ forgejo-pub, scope write:repository, collaborator on one repo). Escrowed because the custody model says the agent's API keys live in ctrl's vault and they did not: all three were hand-placed 0600 files under ~claude-agent/.config covered by neither store — the exact shape §"An instance file can be covered by NEITHER store" warns about. A ctrl rebuild lost them silently, and the loss surfaced only later as an API call returning 404/401. The control role now writes them back from these variables (see below). Note these are API tokens only — the agent's git push rides its SSH cert, so none of them grants repository write over ssh.

⚠️ vault.yml is not *.vault.yml. Until 2026-07-19 the overlay globbed only *.vault.yml + lab.kdbx, so the monolithic vault.yml was never synced — it was purely per-checkout local state, and /opt/myos and the agent clone drifted independently. Whichever checkout you happened to converge from silently won. That is the root cause of the 2026-07-18 DNS/TSIG desync (a converge from one checkout pushed a stale TSIG key fleet-wide). Both the script and this list now name vault.yml explicitly. If you add another file whose name does not end in .vault.yml, add it explicitly too.

Security model (defense-in-depth)¶

Everything in the repo is encrypted at rest: *.vault.yml by the Ansible vault password (TPM-sealed on ctrl, NOT in the repo), lab.kdbx by its master password (also not in the repo). Read access to lab-secrets yields only ciphertext.
The repo is Tier-2 private (WG/control-plane only, like the control-plane myos).
ctrl pulls it with a read-only deploy key (~${admin}/.ssh/id_lab_secrets) — so the automation can refresh secrets without holding a human credential, and cannot write even if ctrl is compromised. Publishing uses a separate write key that is passphrase-protected, the passphrase escrowed in lab.kdbx (entry lab-secrets push key passphrase (id_lab_secrets_push), added 2026-08-01 — until then this sentence described an entry that did not exist, and the passphrase was a single point of loss). Write stays operator-present, because lab.kdbx is YubiKey-gated (runbook 47): presence moved from typing the passphrase to arming the DB. Note the corollary — an armed lab.kdbx therefore grants write to this store, not merely read. See "Deploy keys — two, split by capability". The sync key is bootstrap material: it is NOT in any repo; restore it from escrow (KeePass/offline) on a ctrl rebuild before the first converge. It is Tang-sealed (like the control key), so a stolen/off-LAN ctrl fails closed.

The mechanism (Model A — overlay-as-files)¶

Same shape as the instance-material overlay (runbook 33), a different source:

scripts/load-lab-secrets.sh — git pulls ${lab_secrets_dir} (default /opt/lab-secrets) with the deploy key and install -m600s every *.vault.yml + vault.yml + lab.kdbx into each checkout's ansible/group_vars/all/. A pull failure with a cached clone present is non-fatal (forgejo down → use the cache, so a converge still runs); only a clone with no cache at all is fatal.
lab-secrets-sync.service (control role) — a root oneshot, Before=host-ca-autosign and run on boot + on every control converge, so the secrets are in the checkouts before anything reads the vault. control_repo_dir (/opt/myos) + agent_repo_dest are both synced.
Vars (control defaults): lab_secrets_enabled, lab_secrets_url, lab_secrets_dir, lab_secrets_deploy_key.

Adding or updating a secret¶

# On ctrl, in the lab-secrets clone (deploy key has read; a WRITE path needs write access):
cd /opt/lab-secrets
ansible-vault edit <name>.vault.yml         # uses the TPM vault pass via scripts/vault-pass.sh
git commit -am "update <name>" && git push  # requires write access to lab-secrets
# For a NEW vault file (as opposed to editing one), the flow is: build the plaintext in a
# 0600 temp file -> `ansible-vault encrypt --output <name>.vault.yml` -> verify it round-trips
# (`ansible-vault view` and check the VARIABLE NAMES, never the values) -> copy the ENCRYPTED
# file into /opt/lab-secrets -> commit -> push with the WRITE key. The push passphrase lives in
# lab.kdbx, so it can be driven end-to-end from an armed DB without a human typing it:
#     kdbx-get | ssh almalinux@ctrl 'bash -s' <<'EOF'   # NOTE: see the stdin trap below
# ⚠ A heredoc IS stdin. `ssh 'bash -s' <<EOF` with a `read` inside makes that read consume the
#   SCRIPT's next line, not the piped password — it fails as an unbound variable several lines
#   later. Send the script as an argument (base64) and keep stdin for the password.
#
# For the RECURRING push — re-escrowing the instance bundle + the kdbx after a re-seal or
# entry edit — use the worked script instead of reconstructing the flow:
#     kdbx-get | ssh almalinux@ctrl 'ESCROW_NOTE="why" bash /opt/myos/scripts/escrow-push.sh'
# (armed DB required; it refreshes the tar attachment + manifest AND pushes in one motion)
# then re-sync + converge the consumer:
sudo systemctl start lab-secrets-sync.service
cd /opt/myos/ansible && ansible-playbook playbooks/site.yml -l <host>

To rotate an OIDC client secret, change it in oidc-client-secrets.vault.yml and converge BOTH sides together (the provider sso/Authelia and the client, e.g. files/Seafile or the git server) — otherwise the provider and client mismatch and that login breaks.

Restoring the agent's API tokens on a rebuilt ctrl¶

The control role writes ~claude-agent/.config/{forgejo-agent,forgejo-reviewer,forgejo-pub}.token (0600, owned by the agent) from the three variables above. Two properties worth knowing:

It takes two converges on a fresh ctrl, by construction. The same role installs lab-secrets-sync, and Ansible reads vault variables once at play start — so on a box that has never synced, the overlay cannot exist yet during the first run. The tasks therefore skip rather than fail, or the bootstrap would deadlock on itself. Run systemctl start lab-secrets-sync and converge control again.
A skip is reported per file, naming the file, the consequence ("the agent cannot authenticate to that Forgejo") and the fix. That is deliberate: a silent skip on a missing vault variable is what kept the dead-man's-switch ntfy token missing until the watchdog was needed (runbook 27).

Deploy keys — two, split by capability¶

The repo has two deploy keys, and the split is the point: the credential that is loaded automatically on every boot and converge must not be able to write.

key	file on ctrl	Forgejo access	unlocked by	used by
sync	`~${admin}/.ssh/id_lab_secrets`	read-only	Tang (`tang1`), automatic	`lab-secrets-sync.service` / `load-lab-secrets.sh`
push	`~${admin}/.ssh/id_lab_secrets_push`	write	passphrase in `lab.kdbx`, operator-present	humans/agent, by hand, when publishing secrets

Why not one key with write. The sync key is Tang-sealed, so it unseals by itself on-LAN and is effectively always available to anything running on ctrl. Attaching write to it means a ctrl compromise can rewrite the secrets history, not merely read ciphertext it cannot decrypt. Since the standing job is only ever git pull, write buys nothing there.

Why the push key is deliberately NOT Tang-sealed. Its passphrase lives in lab.kdbx, so using it requires an armed DB — i.e. the operator physically present with a YubiKey. That converts write from a standing capability into a presence-gated one. load-lab-secrets.sh keys its Tang path off the existence of <key>.pass.jwe; the push key has no .jwe, so the loader can never pick it up.

Residual risk, stated plainly: while a lab.kdbx arm is live, anything running as the admin user on ctrl could use the push key. Presence-gating shrinks the window from always to while you are at the keyboard; it does not eliminate it. Eliminating it entirely means holding the write key off ctrl altogether and pushing from the workstation — a real option, at the cost of taking ctrl (and the agent) out of the publish path.

# SYNC key (read-only) — generate on ctrl as the admin account:
ssh-keygen -t ed25519 -f ~/.ssh/id_lab_secrets -N "" -C "ctrl lab-secrets read-only deploy key"
cat ~/.ssh/id_lab_secrets.pub    # Forgejo -> lab-secrets -> Settings -> Deploy Keys, write UNCHECKED

# PUSH key (write) — passphrase generated, escrowed to lab.kdbx, never echoed:
PASS=$(openssl rand -base64 32 | tr -d '\n')
ssh-keygen -q -t ed25519 -f ~/.ssh/id_lab_secrets_push -N "$PASS" \
  -C "ctrl lab-secrets PUSH key (write; passphrase in lab.kdbx)"
# store $PASS in lab.kdbx as "lab-secrets push deploy key (write)", then unset it
cat ~/.ssh/id_lab_secrets_push.pub   # Forgejo -> Deploy Keys, write CHECKED

Escrow the sync private key (KeePass/offline) — it is required to bootstrap a rebuilt ctrl. The push key is regenerable and need not be escrowed: lose it and you mint a new one and re-register it.

Forgejo cannot toggle write on an existing deploy key — changing it means delete + re-add.

Verifying the split (do this, don't assume)¶

Five checks. The UI checkbox is not proof; the server's own refusal is.

# 1+2 sync key: fetch must SUCCEED, push must be REFUSED
GIT_SSH_COMMAND="ssh -i ~/.ssh/id_lab_secrets -o IdentitiesOnly=yes" git fetch origin
GIT_SSH_COMMAND="ssh -i ~/.ssh/id_lab_secrets -o IdentitiesOnly=yes" git push --dry-run origin HEAD:main
#   expect: Forgejo: Deploy Key: <n>:<name> is not authorized to write to <org>/lab-secrets.
# 3+4 push key (needs an armed lab.kdbx for its passphrase): both must SUCCEED
# 5 the one that must not break — the service still pulls on the read-only key:
sudo systemctl start lab-secrets-sync.service   # exit 0 AND no "pull failed ... using cached copy"

Check 5 matters most: a broken pull is non-fatal by design (falls back to the cached clone), so the unit still reports success. Confirm the absence of the cache-fallback warning, not just exit 0.

Match on the server's message, not on words like "denied" — Forgejo's refusal reads ... is not authorized to write to ..., and a grep for "denied/refused/permission" misses it and reports a false failure.

Optional: Tang-seal the deploy key (fail-closed off-LAN)¶

By default the key is a plaintext 0600 file (low-risk — it only reaches ciphertext). To make it fail-closed like the control key (unusable off-LAN / on a stolen ctrl), passphrase-protect it and Tang-seal the passphrase — load-lab-secrets.sh auto-detects the <key>.pass.jwe and loads the key via a tang1-gated agent (else it falls back to the direct key):

# One-time, on ctrl as the admin account, AFTER the Tang-aware loader is deployed:
ssh-keygen -p -f ~/.ssh/id_lab_secrets            # set a strong passphrase (record it in KeePass)
printf '%s' '<that-passphrase>' \
  | clevis encrypt tang '{"url":"http://<tang1>"}' > ~/.ssh/id_lab_secrets.pass.jwe
chmod 600 ~/.ssh/id_lab_secrets.pass.jwe
# verify: `scripts/load-lab-secrets.sh /opt/myos` still syncs (loads the key via tang1).

Escrow the passphrase in KeePass (break-glass: ssh-add ~/.ssh/id_lab_secrets with it). Seal only after the Tang-aware loader is in place, or the sync (direct -i) breaks on the now passphrase-protected key.

Troubleshooting: `Permission denied (publickey)` on a by-hand run¶

A manual load-lab-secrets.sh / git -C /opt/lab-secrets fetch as the admin account can fail Permission denied (publickey) while the systemd oneshot keeps working. That is almost never a revoked deploy key — it is client-side identity pinning:

~${admin}/.ssh/config starts with Host 10.20.* *.${domain} + IdentityFile ~/.ssh/id_ctrl + IdentitiesOnly yes. That glob matches forgejo (10.20.30.30), so ssh offers only id_ctrl and the deploy key is never offered at all. lab-secrets-sync.service is User=root and root has no ~/.ssh/config, so the service is immune — which is exactly why the symptom looks selective.

Diagnose before touching Forgejo:

GIT_SSH_COMMAND="ssh -v" git -C /opt/lab-secrets fetch 2>&1 | grep Offering
# offers id_ctrl  -> client-side pinning (this section)
# offers id_lab_secrets and still denied -> genuine server-side authz

Fixed in the loader as of 2026-07-19 (both branches now pass -i "$KEY" -o IdentitiesOnly=yes). For an ad-hoc git command, pass the same flags yourself.

Two traps worth knowing:

ssh -T -p 2222 git@<forgejo> is not a valid deploy-key test. Deploy keys are repo-scoped and never get the interactive greeting, so they are refused there even when perfectly valid. Test with a real git fetch.
lab-secrets synced -> … is printed on the cache path too. A pull failure with a cached clone is non-fatal by design, so the unit still exits 0/SUCCESS. It does print warning: lab-secrets pull failed … using cached copy to stderr — check the journal for that warning, and when it matters verify by sha256sum on the destination file rather than trusting the success line.

DR / rebuild¶

A rebuilt ctrl needs, in order: the TPM-sealed vault cred (rb15), then the lab-secrets deploy key from escrow, then lab-secrets-sync populates group_vars/all/, then converges run. Without the deploy key the checkouts have no secrets and converges fail on undefined vault vars.

43 — Lab secrets store (lab-secrets repo → vault overlay)¶