43 — Lab secrets store (`lab-secrets` repo → vault overlay)¶

Status: built 2026-07-18. The instance's Ansible secrets live in a dedicated lab-secrets Forgejo repo (control-plane git.${domain}, lab_org/lab-secrets), separate from the myos IaC repo. A control-role oneshot pulls it and overlays the encrypted files into the checkouts' ansible/group_vars/all/ before any converge.

Why a separate repo¶

The myos repo is the IaC + the source of the public docs (myos.io), so secrets must never live in it — *.vault.yml and lab.kdbx are gitignored there. But gitignored files aren't reproducible: a fresh checkout has none, so a converge runs with undefined vault vars (this is exactly how the Seafile OIDC secret went missing and broke the sso converge). lab-secrets is the reproducible, version-controlled home for those files.

What it holds¶

The ansible-vault-encrypted *.vault.yml files (the automation secrets: authelia, oidc client secrets, origin WireGuard key, DNSSEC escrow, etc.).
The legacy monolithic vault.yml (DNS/TSIG among others). It is $ANSIBLE_VAULT-encrypted like the rest, but note it does not match a *.vault.yml glob — see the warning below.
lab.kdbx — the KeePass break-glass DB (human tier).

⚠️ vault.yml is not *.vault.yml. Until 2026-07-19 the overlay globbed only *.vault.yml + lab.kdbx, so the monolithic vault.yml was never synced — it was purely per-checkout local state, and /opt/myos and the agent clone drifted independently. Whichever checkout you happened to converge from silently won. That is the root cause of the 2026-07-18 DNS/TSIG desync (a converge from one checkout pushed a stale TSIG key fleet-wide). Both the script and this list now name vault.yml explicitly. If you add another file whose name does not end in .vault.yml, add it explicitly too.

Security model (defense-in-depth)¶

Everything in the repo is encrypted at rest: *.vault.yml by the Ansible vault password (TPM-sealed on ctrl, NOT in the repo), lab.kdbx by its master password (also not in the repo). Read access to lab-secrets yields only ciphertext.
The repo is Tier-2 private (WG/control-plane only, like the control-plane myos).
ctrl pulls it with a read-only deploy key (~${admin}/.ssh/id_lab_secrets) — so the automation can refresh secrets without holding a human credential, and cannot write even if ctrl is compromised. Publishing uses a separate write key that is passphrase-protected, the passphrase escrowed in lab.kdbx (entry lab-secrets push key passphrase (id_lab_secrets_push), added 2026-08-01 — until then this sentence described an entry that did not exist, and the passphrase was a single point of loss). Write stays operator-present, because lab.kdbx is YubiKey-gated (runbook 47): presence moved from typing the passphrase to arming the DB. Note the corollary — an armed lab.kdbx therefore grants write to this store, not merely read. See "Deploy keys — two, split by capability". The sync key is bootstrap material: it is NOT in any repo; restore it from escrow (KeePass/offline) on a ctrl rebuild before the first converge. It is Tang-sealed (like the control key), so a stolen/off-LAN ctrl fails closed.

The mechanism (Model A — overlay-as-files)¶

Same shape as the instance-material overlay (runbook 33), a different source:

scripts/load-lab-secrets.sh — git pulls ${lab_secrets_dir} (default /opt/lab-secrets) with the deploy key and install -m600s every *.vault.yml + vault.yml + lab.kdbx into each checkout's ansible/group_vars/all/. A pull failure with a cached clone present is non-fatal (forgejo down → use the cache, so a converge still runs); only a clone with no cache at all is fatal.
lab-secrets-sync.service (control role) — a root oneshot, Before=host-ca-autosign and run on boot + on every control converge, so the secrets are in the checkouts before anything reads the vault. control_repo_dir (/opt/myos) + agent_repo_dest are both synced.
Vars (control defaults): lab_secrets_enabled, lab_secrets_url, lab_secrets_dir, lab_secrets_deploy_key.

Adding or updating a secret¶

# On ctrl, in the lab-secrets clone (deploy key has read; a WRITE path needs write access):
cd /opt/lab-secrets
ansible-vault edit <name>.vault.yml         # uses the TPM vault pass via scripts/vault-pass.sh
git commit -am "update <name>" && git push  # requires write access to lab-secrets
# then re-sync + converge the consumer:
sudo systemctl start lab-secrets-sync.service
cd /opt/myos/ansible && ansible-playbook playbooks/site.yml -l <host>

To rotate an OIDC client secret, change it in oidc-client-secrets.vault.yml and converge BOTH sides together (the provider sso/Authelia and the client, e.g. files/Seafile or the git server) — otherwise the provider and client mismatch and that login breaks.

Deploy keys — two, split by capability¶

The repo has two deploy keys, and the split is the point: the credential that is loaded automatically on every boot and converge must not be able to write.

key	file on ctrl	Forgejo access	unlocked by	used by
sync	`~${admin}/.ssh/id_lab_secrets`	read-only	Tang (`tang1`), automatic	`lab-secrets-sync.service` / `load-lab-secrets.sh`
push	`~${admin}/.ssh/id_lab_secrets_push`	write	passphrase in `lab.kdbx`, operator-present	humans/agent, by hand, when publishing secrets

Why not one key with write. The sync key is Tang-sealed, so it unseals by itself on-LAN and is effectively always available to anything running on ctrl. Attaching write to it means a ctrl compromise can rewrite the secrets history, not merely read ciphertext it cannot decrypt. Since the standing job is only ever git pull, write buys nothing there.

Why the push key is deliberately NOT Tang-sealed. Its passphrase lives in lab.kdbx, so using it requires an armed DB — i.e. the operator physically present with a YubiKey. That converts write from a standing capability into a presence-gated one. load-lab-secrets.sh keys its Tang path off the existence of <key>.pass.jwe; the push key has no .jwe, so the loader can never pick it up.

Residual risk, stated plainly: while a lab.kdbx arm is live, anything running as the admin user on ctrl could use the push key. Presence-gating shrinks the window from always to while you are at the keyboard; it does not eliminate it. Eliminating it entirely means holding the write key off ctrl altogether and pushing from the workstation — a real option, at the cost of taking ctrl (and the agent) out of the publish path.

# SYNC key (read-only) — generate on ctrl as the admin account:
ssh-keygen -t ed25519 -f ~/.ssh/id_lab_secrets -N "" -C "ctrl lab-secrets read-only deploy key"
cat ~/.ssh/id_lab_secrets.pub    # Forgejo -> lab-secrets -> Settings -> Deploy Keys, write UNCHECKED

# PUSH key (write) — passphrase generated, escrowed to lab.kdbx, never echoed:
PASS=$(openssl rand -base64 32 | tr -d '\n')
ssh-keygen -q -t ed25519 -f ~/.ssh/id_lab_secrets_push -N "$PASS" \
  -C "ctrl lab-secrets PUSH key (write; passphrase in lab.kdbx)"
# store $PASS in lab.kdbx as "lab-secrets push deploy key (write)", then unset it
cat ~/.ssh/id_lab_secrets_push.pub   # Forgejo -> Deploy Keys, write CHECKED

Escrow the sync private key (KeePass/offline) — it is required to bootstrap a rebuilt ctrl. The push key is regenerable and need not be escrowed: lose it and you mint a new one and re-register it.

Forgejo cannot toggle write on an existing deploy key — changing it means delete + re-add.

Verifying the split (do this, don't assume)¶

Five checks. The UI checkbox is not proof; the server's own refusal is.

# 1+2 sync key: fetch must SUCCEED, push must be REFUSED
GIT_SSH_COMMAND="ssh -i ~/.ssh/id_lab_secrets -o IdentitiesOnly=yes" git fetch origin
GIT_SSH_COMMAND="ssh -i ~/.ssh/id_lab_secrets -o IdentitiesOnly=yes" git push --dry-run origin HEAD:main
#   expect: Forgejo: Deploy Key: <n>:<name> is not authorized to write to <org>/lab-secrets.
# 3+4 push key (needs an armed lab.kdbx for its passphrase): both must SUCCEED
# 5 the one that must not break — the service still pulls on the read-only key:
sudo systemctl start lab-secrets-sync.service   # exit 0 AND no "pull failed ... using cached copy"

Check 5 matters most: a broken pull is non-fatal by design (falls back to the cached clone), so the unit still reports success. Confirm the absence of the cache-fallback warning, not just exit 0.

Match on the server's message, not on words like "denied" — Forgejo's refusal reads ... is not authorized to write to ..., and a grep for "denied/refused/permission" misses it and reports a false failure.

Optional: Tang-seal the deploy key (fail-closed off-LAN)¶

By default the key is a plaintext 0600 file (low-risk — it only reaches ciphertext). To make it fail-closed like the control key (unusable off-LAN / on a stolen ctrl), passphrase-protect it and Tang-seal the passphrase — load-lab-secrets.sh auto-detects the <key>.pass.jwe and loads the key via a tang1-gated agent (else it falls back to the direct key):

# One-time, on ctrl as the admin account, AFTER the Tang-aware loader is deployed:
ssh-keygen -p -f ~/.ssh/id_lab_secrets            # set a strong passphrase (record it in KeePass)
printf '%s' '<that-passphrase>' \
  | clevis encrypt tang '{"url":"http://<tang1>"}' > ~/.ssh/id_lab_secrets.pass.jwe
chmod 600 ~/.ssh/id_lab_secrets.pass.jwe
# verify: `scripts/load-lab-secrets.sh /opt/myos` still syncs (loads the key via tang1).

Escrow the passphrase in KeePass (break-glass: ssh-add ~/.ssh/id_lab_secrets with it). Seal only after the Tang-aware loader is in place, or the sync (direct -i) breaks on the now passphrase-protected key.

Troubleshooting: `Permission denied (publickey)` on a by-hand run¶

A manual load-lab-secrets.sh / git -C /opt/lab-secrets fetch as the admin account can fail Permission denied (publickey) while the systemd oneshot keeps working. That is almost never a revoked deploy key — it is client-side identity pinning:

~${admin}/.ssh/config starts with Host 10.20.* *.${domain} + IdentityFile ~/.ssh/id_ctrl + IdentitiesOnly yes. That glob matches forgejo (10.20.30.30), so ssh offers only id_ctrl and the deploy key is never offered at all. lab-secrets-sync.service is User=root and root has no ~/.ssh/config, so the service is immune — which is exactly why the symptom looks selective.

Diagnose before touching Forgejo:

GIT_SSH_COMMAND="ssh -v" git -C /opt/lab-secrets fetch 2>&1 | grep Offering
# offers id_ctrl  -> client-side pinning (this section)
# offers id_lab_secrets and still denied -> genuine server-side authz

Fixed in the loader as of 2026-07-19 (both branches now pass -i "$KEY" -o IdentitiesOnly=yes). For an ad-hoc git command, pass the same flags yourself.

Two traps worth knowing:

ssh -T -p 2222 git@<forgejo> is not a valid deploy-key test. Deploy keys are repo-scoped and never get the interactive greeting, so they are refused there even when perfectly valid. Test with a real git fetch.
lab-secrets synced -> … is printed on the cache path too. A pull failure with a cached clone is non-fatal by design, so the unit still exits 0/SUCCESS. It does print warning: lab-secrets pull failed … using cached copy to stderr — check the journal for that warning, and when it matters verify by sha256sum on the destination file rather than trusting the success line.

DR / rebuild¶

A rebuilt ctrl needs, in order: the TPM-sealed vault cred (rb15), then the lab-secrets deploy key from escrow, then lab-secrets-sync populates group_vars/all/, then converges run. Without the deploy key the checkouts have no secrets and converges fail on undefined vault vars.

43 — Lab secrets store (lab-secrets repo → vault overlay)¶