45 — YubiKey operations (the device operating manual)¶

Status: written 2026-07-18. The token itself — its applets, its secrets, and every lab operation performed with it.

This runbook is about the device. Runbook 20 is about the certificate authority that happens to live on it. The dividing test:

If the fact changes when you swap or replace a token, it lives here. If it changes when you alter what the fleet trusts or what a cert may do, it lives in rb20.

So: PINs, applets, slots, enrolling a new key, retiring a lost one, and the drills that prove any of it works are here. The trust files, principals, cert validity, and the KRL policy are rb20's.

Related: 20 SSH CA · 26 the online host CA · 28 agent certs · 36 the KeePass vault · 47 the agent's presence-unlock for lab.kdbx · 43 the secrets store.

1 — The device model: one stick, several independent applications¶

A YubiKey is not one thing. It is several separate applications sharing a piece of plastic — each with its own storage, its own PINs, its own retry counters, and its own tool. They share nothing. Resetting one does not touch another. This is the single most useful fact about the device, and the source of most confusion.

ykman info lists which applications are enabled.

Applet	Holds	Tool	Our use
PIV (smartcard)	keys in numbered hex slots	`ykman piv`	the SSH CAs — see §1.2
FIDO2 / WebAuthn	resident credentials + SSH `sk-` keys	`ykman fido`	browser logins (§5), optional `ed25519-sk`
OTP	2 programmable slots	`ykman otp`	slot 2 = a retired `lab.kdbx` CR factor (§3.5); `lab.kdbx` is now password-only (rb47)
OATH	TOTP/HOTP seeds	`ykman oath`	see §5
OpenPGP	3 GPG keys	`ykman openpgp` / `gpg`	unused — §10

1.1 — The mental model in one paragraph¶

The YubiKey is the certificate authority. Three keys live in its PIV slots, none of them exportable, each requiring a physical touch to use. The fleet is told to trust the public halves, so any certificate this token signs is accepted everywhere — nothing is registered per-user or per-key on the servers. That is the whole trick: adding a token means pasting two public-key lines into the repo; it does not mean touching 30 hosts.

1.2 — PIV slots¶

PIV defines a fixed set of hex-named slots, each with a standard purpose. We use three:

Slot	Standard purpose	Our use	Touch policy
`9a`	PIV Authentication (login)	admin SSH login key (its cert signed by this token's own 9c)	`cached`
`9c`	Digital Signature	this token's user CA — signs login certs	`always`
`9d`	Key Management	this token's host CA — signs host certs	`always`
`9e`	Card Authentication (PIN-less)	unused — §10
`82`–`95`	Retired Key Management	all free — slot `82` holds a dead ED25519 `kdbx-unlock` key from an abandoned PKCS#11 approach (see §10)
`f9`	Attestation	unused — §10

9c is the signature slot, which is why the signing CA lives there; 9d is simply the next main slot. All three are generated on-device as ECCP384 with --pin-policy once.

Why ECCP384 and not Ed25519 — the constraint is the toolchain, not the crypto. Firmware 5.7 does support Ed25519 in PIV slots, so the hardware is not the limit. But the CA is used through ssh-keygen -D libykcs11.so, and OpenSSH's PKCS#11 layer cannot use an Ed25519 PIV key. Verified 2026-07-19 on both tokens (OpenSSH 10.0p2, libykcs11 5.9.0, firmware 5.7.1): every ECCP384 slot enumerates fine, while the ED25519 key in slot 82 fails with skipping unsupported key type and unknown certificate key type.

So do not convert 9a to Ed25519 — it is the SSH login key and would become unusable through the very path that uses it. If Ed25519 for login is the goal, the native route is a FIDO2 ed25519-sk key, which bypasses PKCS#11 entirely (§5). And 9c/9d should stay ECCP384 regardless: changing a CA's algorithm is not a re-key but a new CA, requiring new trust lines and a fleet-wide re-trust (§7.7) for no security gain — P-384 on a hardware RNG is not a weakness.

The confusing part: you never name a slot when signing. ssh-keygen selects the key by matching the public key you hand it, not by slot number. -s user_ca.pub reaches slot 9c because that file is 9c's public half; -s host_ca.pub reaches 9d. The .pub argument is the selector. This is why a wrong-CA mistake produces a confusing error rather than an obvious one.

1.3 — PIN and touch policies¶

Two independent policies are fixed at key generation and cannot be changed afterwards without regenerating the key (which is what yubikey-tool.sh option R is for).

Policy	Values	Ours	Feels like
`--pin-policy`	`never` · `once` · `always`	`once`	type the PIV PIN once per session, then sign freely
`--touch-policy`	`never` · `always` · `cached`	`always` (CAs), `cached` (9a)	CAs: tap per signature. 9a: one tap + ~15 s grace, so you can hop hosts

The CAs are deliberately touch=always: a signature is an act of authority and should require a human present. The login key is cached because tapping once per SSH hop makes multi-hop jump paths miserable.

1.4 — Generated on-device vs imported¶

Two ways to get a key onto the device, with opposite backup stories:

Generated on-device — ykman piv keys generate. The private half never exists anywhere else and cannot be extracted. No backup is possible. This is how every one of our CAs is made.
Imported — ykman piv keys import. Made elsewhere, loaded in; a copy may exist off the token. Not used for our CAs. (FIDO2 keys cannot be imported at all.)

The consequence is the whole Tier-1 custody model: there is no .bak, no escrow, and nothing to steal from a backup — because no copy was ever created. Redundancy comes from a second token with its own keys, not from a copy of this one. That is §2.

2 — The two-key redundancy model¶

The common misconception, stated plainly so it can be corrected: the two YubiKeys are not a key and its spare. Nothing is cloned between them. They are two independent certificate authorities that the fleet has been told to trust equally.

The analogy that works: not "two copies of the house key" but "two locksmiths, each with their own machine, and every lock in the building accepts a key cut by either one."

Losing a locksmith does not lock you out. It means you have one locksmith.

2.1 — What is independent (almost everything)¶

Every one of these exists separately on each token, generated on that token, never copied:

Independent per token	Notes
9c user-CA key + its self-signed X.509	different key, different fingerprint
9d host-CA key + its cert	"
9a login key + its cert	signed by that same token's 9c — not the other's
FIDO2 credentials	see §5 — this matters for web logins
PIV PIN, PUK, management key	separate values are fine; they gate different chips
Every retry counter	blocking one token's PIN does nothing to the other

Because none of it is exportable, there is nothing to back up, by design.

2.2 — What is shared (nothing today — since 2026-07-18)¶

Nothing is copied between the tokens. Every applet is independent (§2.1), and the redundancy for the two things that need either-key access is achieved additively, not by a shared secret:

SSH signing — both tokens' CA lines are trusted fleet-wide (§2.3).
lab.kdbx presence-unlock — the per-session master password (master.age, rb47) is age-encrypted to both tokens' FIDO2 recipients, so either key decrypts it. That is make-before-break additive trust, exactly like the CA lines — no symmetric secret on the tokens.

History — the one thing that used to be shared (retired 2026-07-18). lab.kdbx formerly AND-combined its master password with an OTP slot-2 HMAC-SHA1 challenge-response secret programmed identically on both tokens, because a KeePass composite key cannot express "key A or key B" — the only way either token could answer was to copy the secret. That factor was retired: the DB is now password-only and the either-key redundancy moved into rb47's multi-recipient master.age. The slot-2 secret, if still programmed on a token, is inert (§3.5). Do not re-add it — rekeying the DB back to CR breaks the agent's arm-kdbx unlock.

2.3 — Where the redundancy actually lives¶

Not in the tokens. In the trust files:

File	Contents
`ssh_user_ca.pub`	both tokens' 9c lines → `TrustedUserCAKeys` fleet-wide
`ssh_host_ca.pub`	both tokens' 9d lines + a third line — the online host CA on `ctrl` (rb26)

Every host accepts a cert from any listed CA. That is the redundancy. It is why adding a third token costs one commit and no ceremony, and why retiring one is a deletion rather than a revocation (§7).

2.4 — Loss matrix¶

Event	Broken immediately	Still works	Action
Lose the carry token	nothing structurally — you are down to one signer	the safe token signs everything: user, host, agent, task certs; and (rb47) still arms `lab.kdbx`, since `master.age` is encrypted to both tokens	§7 Case B in full, including re-cutting `master.age` to the surviving token and rotating the `lab.kdbx` master password (§7.6)
Lose the safe token	nothing today	everything	same checklist; provision a replacement promptly — you are now single-fault
Lose both	all YubiKey signing	fleet SSH still works: Ansible's transport is the static `almalinux` key, not a cert, so a CA loss can never lock you out. Host certs keep verifying while their CA lines remain trusted	provision a new token → publish its CA lines → `--tags sshca`; delete the dead lines last
PIV PIN blocked (3 wrong)	that token, until unblocked	the other token	`ykman piv access unblock-pin` (§3, drill D10)
PIV PUK exhausted	that token's CAs are gone	the other token	`ykman piv reset` → it is now a new device: enroll per §6, distrust the old lines per §7
One token factory reset	its CA lines are now orphaned trust	the other token	treat as a new device — the old lines must be deleted, or you are trusting a CA nobody controls

2.5 — The durable rule¶

Redundancy lives in the trust store, not in duplicated key material.

Everything else follows from it: why there is no backup, why the tokens can have different PINs, why adding one is cheap, and why the thing you must actually protect is the list of trusted CAs — not any individual token.

Prove it rather than believing it: drill D4 (§8) unplugs every other token and runs a full working day's operations on one.

3 — Every secret the key holds¶

Five secrets, on three independent applets. Each has its own retry counter, and exhausting one has a different consequence from exhausting another. This table is the part worth internalising:

Secret	Applet	Gates	Retries	At zero
PIV PIN	PIV	every signature (once per session)	3	PIN blocked → unblock with the PUK
PIV PUK	PIV	unblocking a blocked PIN	3	terminal — only `piv reset`, which destroys the CAs
PIV management key	PIV	generating/importing keys into slots	n/a	n/a — see below
FIDO2 PIN	FIDO2	WebAuthn credentials (§5)	8	`fido reset` — all credentials lost
OTP slot-2 CR secret	OTP	retired — was the `lab.kdbx` CR factor; now inert (§3.5, rb47)	n/a	reprogram or clear the slot

3.1 — PIV PIN¶

Your day-to-day secret. With --pin-policy once you type it once per session and then sign freely until the token is removed or the session ends. Factory default is 123456; ours is changed at provisioning time (option P, step 1).

Three consecutive wrong entries block it. The token still holds the keys — it simply will not use them until unblocked with the PUK. This is recoverable and drilled in D10.

3.2 — PIV PUK¶

Exists for exactly one purpose: unblocking a blocked PIN.

ykman piv access unblock-pin      # PUK + a new PIN

Check this first, on every token: ykman piv info prints WARNING: Using default PUK! if it was never changed. A default PUK defeats the PIN entirely — an attacker with physical possession deliberately blocks the PIN with three wrong entries, then uses the published default PUK to unblock-pin and set a PIN of their own. They now hold a working user CA. Touch is still required, but a thief has the token in their hand. Rotate it with ykman piv access change-puk and escrow the new value in lab.kdbx.

Factory default 12345678. Three wrong PUKs is terminal: the only remaining action is ykman piv reset, which erases the PIV applet — meaning this token's three CA keys are gone forever, because they were generated on-device and never backed up. The token then becomes a new device: enroll per §6, distrust its old lines per §7.

This is the single most likely way to destroy a token, and it is why a second enrolled token is not optional. Burning the counters on your only key is unrecoverable.

3.3 — PIV management key¶

Required to generate or import keys into slots — not to use them. You touch it at provisioning and essentially never again.

We set it with:

ykman piv access change-management-key --generate --protect

--generate makes it random; --protect stores it on the token behind the PIN. The consequence is that there is no management key to escrow or remember — knowing the PIN is sufficient. This is why it is absent from lab.kdbx, which otherwise looks like an omission.

Check for WARNING: Using default Management key! too. ykman piv info warns separately for the PIN, the PUK and the management key. A default management key lets anyone holding the token write to its slots — overwrite the CA keys, import their own, or plant a credential. It does not expose the existing private keys (those are still non-exportable), so the blast radius is tampering and denial rather than forgery — but on a token whose whole job is to be a trustworthy CA, "someone else could have rewritten this slot" is a bad property to have.

3.4 — FIDO2 PIN¶

Independent of everything PIV. Gates WebAuthn credentials — the browser logins in §5.

The retry behaviour is more subtle than PIV's: 8 consecutive failures locks the applet, but there is also a 3-per-power-cycle sub-limit — after three wrong attempts the token refuses further tries until it is physically unplugged and reinserted. That reinsert requirement is a feature (it defeats automated guessing), not a fault.

At zero: ykman fido reset wipes the applet and every credential on it, which means re-registering this key with every website (§5.1). Nothing in PIV is affected.

3.5 — OTP slot-2 challenge-response secret (RETIRED 2026-07-18)¶

No longer an unlock factor. A 20-byte HMAC-SHA1 secret in OTP slot 2, once programmed identically on every token as a factor of the lab.kdbx composite key. When the presence-unlock landed (rb47), lab.kdbx was rekeyed to password-only and the either-key redundancy moved into rb47's multi-recipient master.age. The slot-2 secret is therefore inert today — harmless if still programmed, but nothing reads it.

ykman otp chalresp --touch 2 <20-byte-hex>     # HISTORY — how it was set; do NOT re-add

Do not re-program this as a lab.kdbx factor. Rekeying the DB back to challenge-response would break the agent's arm-kdbx presence-unlock (rb47). If you want the slot genuinely empty, clear it (ykman otp delete 2); leaving it set is only a cosmetic legacy. The OTP applet remains available for other uses (§10).

3.6 — Where these live, and the chicken-and-egg¶

The PINs are recorded in lab.kdbx (rb36, rb43) — but lab.kdbx is itself opened with the master password (typed directly, or presented by the rb47 presence-unlock). That circularity is real and must be resolved deliberately:

Escrow rule: the lab.kdbx master password must exist outside the database — in the safe, on paper — and rb47's master.age + identity-{A,B}.txt must be backed up off atomic. Everything else (PINs, PUKs) may live inside the DB. Never let the only copy of a factor be behind the factor. (The old OTP CR secret was in this rule too; it is retired — §3.5.)

3.7 — What we deliberately do not set¶

OATH password — see §5; not currently needed.
OpenPGP PINs — the applet is unused (§10).
PIN-less operation (slot 9e) — deliberately unused; it contradicts the custody model.

3.8 — Rotation¶

Secret	Rotate in place?
PIV PIN	yes — `ykman piv access change-pin`, no re-enrollment
PIV PUK	yes — `ykman piv access change-puk`
Management key	yes — re-run `change-management-key --generate --protect`
FIDO2 PIN	yes — `ykman fido access change-pin`; credentials survive
OTP CR secret	n/a — retired (§3.5); `lab.kdbx` is password-only. Rotate the master password + re-cut `master.age` instead (§7.6, rb47)

The CA keys themselves cannot be rotated in place — a new key is a new CA, which means publishing new trust lines (§6) and distrusting the old (§7).

4 — Command reference, by lab operation¶

4.1 — Where you run this¶

Signing happens where the token is plugged in — the atomic admin laptop, inside a toolbox (it shares the host's USB, $HOME, network and D-Bus, so the key, ~/.ssh and KeePassXC all work). Setup, and the ssh-agent -P gotcha that Ansible runs depend on, are in rb20.

scripts/yubikey-tool.sh with no arguments opens the intent-first "what do you need?" menu — the two routine actions (renew your login cert; sign the agent a JIT task cert) are a single keypress and show a preview → confirm/edit step before anything is minted. It also shows live token + cert-expiry status. The classic letter menu (P/L/R/U/H/A/T/ i/b/e/c, referenced throughout this runbook) is still there — every letter works straight from the intent screen, and --menu (or 6) More…) opens the full classic layout. Press ? in either for the built-in explainer. (Note: from the intent screen, L and T route to the new preview-driven renew / task-cert flows rather than the classic step-by-step prompts; open --menu if you want the exact classic prompt sequence.)

4.2 — Inventory and health (read-only, no PIN)¶

ykman info                 # which applets are enabled; firmware; serial
ykman piv info             # slots populated, retries remaining, policies
ykman fido info            # FIDO2 PIN set? retries?
ykman otp info             # which OTP slots are programmed
ykman list                 # every connected token

Reading a slot's public half needs no PIN — the token hands it out freely:

ykman piv keys export 9c - | ssh-keygen -i -m PKCS8    # this token's user-CA pubkey

4.3 — Signing operations¶

Every row needs PIN + touch. The tool wraps each; the raw form is the escape hatch.

Operation	Tool	Raw equivalent
Renew my login cert (routine, ~weekly)	menu `L`	`ssh-keygen -s <user_ca.pub> -D $PKCS11 -I <id> -n admin -V +1w -z $(date +%s) ~/.ssh/id_piv9a_<L>.pub`
Sign a user cert for a person	menu `U`	as above, against their `.pub`
Sign a host cert	menu `H`	add `-h`, `-n <names,ip>`, `-V +52w`
Sign the standing agent cert	menu `A` or `yubikey-tool.sh sign-agent`	signs `id_agent`, deploys to `ctrl` (rb28)
Sign a JIT task cert for the agent	menu `T` or `yubikey-tool.sh sign-agent-task [PRINCIPALS] [TTL] [LABEL]`	signs the separate `id_agent_task` key — additive, lapses on its TTL
Bulk host-cert rotation	`scripts/rotate-host-certs.sh <tier> <token-label>`	PIN + touch per host; see rb20
Revoke a cert	`scripts/push-krl.sh` after editing the KRL	rb20
~~Program the KeePassXC factor~~ (retired — §3.5)	menu `c` (legacy)	`ykman otp chalresp --touch 2 <secret>` — do not re-add; `lab.kdbx` is password-only (rb47)
Re-key 9a / change touch policy	menu `R`	regenerates the key — the old one is discarded
Provision a fresh token	menu `P`	§6

Only two operations are scriptable non-interactively: sign-agent and sign-agent-task. Everything else is menu-only by design — they are rare, consequential, and benefit from the confirmation prompts.

4.4 — PIN / PUK / management-key commands¶

ykman piv access change-pin                              # rotate the PIN
ykman piv access change-puk                              # rotate the PUK
ykman piv access unblock-pin                             # PUK -> new PIN (after 3 bad PINs)
ykman piv access change-management-key --generate --protect
ykman fido access change-pin                             # FIDO2 PIN

4.5 — Destructive commands (know these before you need them)¶

ykman piv keys delete 9c        # one slot
ykman piv reset                 # WIPES PIV: all three CAs gone, PIN/PUK/mgmt reset
ykman fido reset                # WIPES FIDO2: every WebAuthn credential gone
ykman oath reset
ykman openpgp reset

There is no single "wipe the whole key" — you reset each applet separately. ykman piv reset on an enrolled token means re-provisioning (§6) and swapping its trust lines (§7).

4.6 — Troubleshooting¶

Symptom	Cause	Fix
`ykman` can't see the key	pcscd not running (common in a fresh toolbox)	`sudo dnf install -y pcsc-lite && sudo systemctl enable --now pcscd`
`agent refused operation` while signing	agent lacks the key, or the touch timed out	re-run `ssh-add -s`, and tap the moment it blinks
`No PKCS#11 provider found`	`libykcs11` missing	install `yubico-piv-tool`, or set `YK_PKCS11=/usr/lib64/libykcs11.so.2`
Signing produces an empty cert on a fresh token	pcscd not settled	the tool retries 3× automatically; otherwise wait and retry
Ansible fails instantly with permission denied	no tty, so it cannot prompt for the PIN	load the key into a real OpenSSH agent first — rb20

Current state, stated plainly: no web UI in this lab uses the YubiKey today. The FIDO2 applet is provisioned on both tokens and wired to SSH (ed25519-sk) and the lab.kdbx presence-unlock (rb47's age-plugin-fido2-hmac) — but not to any browser login. Every browser login in the lab — including Proxmox and OPNsense root — is a single password out of KeePass. (The OTP applet's old lab.kdbx role is retired — §3.5.)

This section documents that reality, the one rule that makes web use safe on a two-key setup, and the procedure to change it.

5.1 — The rule that matters most¶

WebAuthn credentials are per-token and non-exportable. Unlike a symmetric secret (such as the now-retired OTP challenge-response factor, §2.2/§3.5), a passkey cannot be copied to your second key. There is no equivalent of "program the same value on both" — the same constraint rb47 works around by encrypting master.age to both tokens' recipients rather than sharing one secret.

Therefore: register BOTH tokens at EVERY site, at enrolment time. Turning on a second factor with one token registered is a self-inflicted lockout — lose or reset that token and the account is gone, exactly as if you had one key and no spare. This is the single most common way people destroy their own access with hardware keys.

Corollaries:

A site that allows only one security key is a site you must not gate with a YubiKey alone — use TOTP with a saved seed, or keep recovery codes.
ykman fido reset erases every credential on that token (§3.4). After a reset you must re-register that key at every site it was enrolled with. Keeping the inventory in §5.2 current is what makes that recoverable rather than an archaeology exercise.
Always capture the site's recovery codes at enrolment and store them in lab.kdbx. They are the third factor that saves you when both tokens are unavailable.

5.2 — External accounts (operator-maintained)¶

Passkeys on external accounts are used, but the lab repo cannot see them — they live with the provider. Maintain the inventory here, because after a token loss or a FIDO2 reset this list is the recovery checklist.

Site	Key A registered	Key B registered	Recovery codes in `lab.kdbx`	Notes
(fill in)

Candidates worth auditing if they are not already listed: the domain registrar (owns DNS delegation — arguably the highest-value account outside the lab), the VPS vendors (own the edges), email, code-hosting, and the password manager itself.

5.3 — Why no lab site uses it today¶

Authelia is the one place a second factor would take effect across many services at once, and it is switched off:

Fact	Where
No `webauthn:` block at all — no pinned `rp.id`, no `user_verification`	`roles/authelia/templates/configuration.yml.j2`
`authelia_access_policy: "one_factor"` — the global default	`roles/authelia/defaults/main.yml:29`
Grafana, Forgejo and Seafile OIDC clients each hardcode `authorization_policy: 'one_factor'`	`configuration.yml.j2`
TOTP is present but issuer-only, and no policy ever demands it	`configuration.yml.j2:10-11`

So flipping the global knob is not sufficient — the three OIDC clients carry their own policy and need separate edits. Assuming one switch covers everything is the trap here.

Services outside Authelia (Proxmox, OPNsense, FreeIPA, UniFi, Home Assistant, Nextcloud, ntfy, tvheadend, umami, Radicale) each own their own login and would need enabling individually. Several support native 2FA and have it unconfigured.

The asymmetry worth internalising: SSH to root@pve requires an operator-signed, +4h, YubiKey-gated admin-hv task cert (rb20, rb28). The browser path to the same full-fleet privilege is one reusable password. The custody of the two paths to identical authority differs by an order of magnitude.

5.4 — Enabling it for the lab (procedure)¶

Not yet done. When you choose to:

Pin the WebAuthn relying party. Add a webauthn: block with an explicit rp.id and display_name. This matters more here than in a single-domain deployment: credentials are rp-id-scoped, and this lab serves several apexes with their own cookie domains (authelia_cookie_domains_extra, rb13). A credential enrolled at auth.${domain} will not work at another apex's portal — each needs its own enrolment.
Enrol before enforcing. Register both tokens while the policy is still one_factor. Enforcing first can lock you out of the portal you need in order to enrol.
Mind the notifier. Authelia's notifier is filesystem — the enrolment link is written to a file inside the container, not emailed. Retrieve it on the sso host (podman exec/read notification.txt) to complete registration. Following rb13's MFA stub literally will stall here.
Then raise the policies — authelia_access_policy to two_factor, plus the three OIDC clients individually.
Back up the credential store. Registered credentials live in Authelia's db.sqlite3, which rb13 currently describes as having nothing to back up. Once devices are enrolled that is no longer true — losing the sso guest would lock you out of every two_factor domain. Add it to db_backups in the same change.
Keep the break-glass paths. Grafana's local admin form and Forgejo's local password login are deliberately retained (rb13, rb36); do not remove them while adding a factor.

5.5 — Known-stale claims to correct alongside¶

Two published statements describe a YubiKey web login that does not exist:

runbooks/41-seafile-file-sync.md:78 — describes the phone SSO flow as "Authelia (IPA + YubiKey)". Seafile's OIDC client is one_factor; no YubiKey is involved.
docs/use/index.md:55-56 — advises users to enable "a security key or app code" if SSO offers one. It does not.

5.6 — The OATH applet¶

OATH holds TOTP/HOTP seeds and is managed with the Yubico Authenticator GUI. It is the right tool for sites that don't support WebAuthn.

Its backup story is the inverse of FIDO2's, and this is the useful bit: TOTP seeds can be loaded onto both tokens, because a seed is a symmetric secret — but only if you save the seed (or its QR code) at enrolment. It cannot be read back off a token afterwards. A seed captured once and stored in lab.kdbx gives you the same either-key redundancy the KDBX factor has; a seed enrolled directly by scanning and never saved gives you a single-token dependency you cannot repair.

6 — Add a new key (enrolment)¶

Adding a token is cheap by design: generate its own keys on-device, publish two public lines, and the whole fleet accepts it. There is no migration, no import, and nothing to copy from an existing token.

You will need: the new token, the laptop with ykman + libykcs11, the repo, the lab.kdbx master password (to re-cut master.age for the new key — §6.3, rb47), and the ability to commit + run Ansible.

6.1 — Provision on-device¶

scripts/yubikey-tool.sh          # option P

Pick a label (A, B, C…) — it flows into every filename and certificate CN, so keep it short and distinct. The tool then:

Sets all four token secrets — PIV PIN, PIV PUK, protected management key, and FIDO2 PIN (§3). Each defaults to set (press Enter); you can skip one you've already set on this token. The PUK is now included — it used to be omitted (§6.1.1).
Generates 9c, 9d, 9a on-device (ECCP384, pin-policy once; CAs touch=always, 9a touch=cached) each with a self-signed X.509 so PKCS#11 exposes it.
Signs the 9a login cert with this token's own 9c.
Optionally creates a FIDO2 resident SSH key.
Verifies no factory defaults remain (ykman piv info) and refuses to call the token done while any survive — then prints the two public-key lines and their fingerprints.

You will touch the key repeatedly — once per slot's certificate, once for the login cert. Tap it whenever it blinks. Escrow the PIN + PUK per token in lab.kdbx afterward.

6.1.1 — Confirm no factory defaults remain (the step that used to get missed)¶

This gap is now closed in the tool — but verify anyway, because the failure is silent and terminal. Historically, yubikey-tool.sh option P never offered a PUK change at all and its other prompts defaulted to N and were independently skippable, so a token could come out of provisioning looking fully set up while silently retaining one or more factory defaults — and the two tokens could end up hardened differently.

This was not hypothetical. A verified check on 2026-07-19 found exactly that split:

	PIN	PUK	Management key
Token A	set	factory default	protected
Token B	set	set	factory default

Each token had a different hole, and both had passed provisioning. The tool now (a) prompts for all four secrets including the PUK, defaulting to set, and (b) runs a Step 5 verification (ykman piv info) that flags any surviving factory default instead of quietly finishing. Still confirm it yourself — run this on every token and check the result rather than assuming:

ykman piv access change-pin
ykman piv access change-puk                                 # option P never asks — do it here
ykman piv access change-management-key --generate --protect
ykman piv info                                              # expect NO "WARNING: Using default …"

Escrow the PIN and PUK in lab.kdbx per token (the management key needs no escrow — §3.3). ykman piv info warns separately for each of the three; all three warnings must be absent before the token is considered enrolled.

The PUK has 3 tries and exhausting it is terminal (§3.2) — it forces ykman piv reset and destroys that token's CA keys. Verify with ykman piv info, never by test-failing.

Principal check. The 9a login cert must be signed with a principal that actually grants admin (admin, or the appropriate scoped principal). A cert signed with a principal no host maps to will authenticate and then land nowhere useful. rb20 records a real instance of this: one token's 9a cert was signed -n user, leaving it unable to drive rotate-host-certs.sh. Drill D9 (§8) exists to catch it.

6.2 — Publish the trust¶

Paste the two printed lines into the trust files — one line per token — then commit and deploy. Trust-file mechanics, the appliance updates (Proxmox + OPNsense are not Ansible-managed), and the fingerprint table are rb20's; see it for the exact steps.

Until this is done the new token signs certs that nothing accepts.

6.3 — Enrol the token for the `lab.kdbx` presence-unlock (rb47)¶

lab.kdbx is password-only, so there is no OTP secret to copy anymore. To let the new token arm the DB, generate its FIDO2 identity and re-cut master.age to include it (full procedure in rb47 §Build):

# on atomic, with the new token plugged in
age-plugin-fido2-hmac -g > ~/.config/kdbx/identity-<label>.txt   # answer yes to require-PIN + separate-identity
scripts/kdbx/make-master.sh                                       # re-encrypt master.age to ALL tokens' recipients

make-master.sh reads the recipient (# public key:) from every identity-*.txt, so after this either the existing tokens or the new one can arm lab.kdbx. Back up the new identity file off atomic (it is required to decrypt — rb47). Skipping this leaves the new token able to sign SSH certs but unable to arm the vault.

The legacy path — scripts/yubikey-tool.sh option c, programming a shared OTP CR secret — is retired (§3.5). Do not run it; it would not help and re-adding a CR factor to the DB breaks the agent unlock.

6.4 — Register it everywhere a key is used¶

This is the step people skip, and it is what turns a "spare" into an actual spare:

Every external site with a passkey (§5.2) — WebAuthn credentials cannot be copied, so the new token must be registered individually at each one. Update the §5.2 inventory.
lab.kdbx — add or update the token's entry (label, serial, PINs, date enrolled).
rb20's enrolled-token table — add its fingerprints.

6.5 — Acceptance gate¶

Do not rely on the new token until it has passed §8. At minimum run D2 (every slot signs) and D4 (it works with every other token unplugged). A token that has never signed anything is not a spare; it is an assumption.

7 — Cancel, revoke, retire, rotate a key¶

7.0 — The concept correction, first¶

These are three different operations and conflating them causes mistakes:

You want to…	Mechanism	Effect
Kill a certificate	add its serial to the KRL, `scripts/push-krl.sh`	immediate, fleet-wide, read at auth time — no restart
Stop trusting a CA	delete its line from the trust files, redeploy	that token's future and existing certs stop being accepted
Neutralise a token	wipe the applets (`piv reset`, `fido reset`)	only possible if you physically hold it

There is no such thing as "revoking" a 9c key. A CA is not revoked; it is distrusted, by removing it from the list every host reads. The KRL revokes certs, by serial. Reaching for the KRL when you meant to distrust a CA — or vice versa — leaves the hole open.

7.1 — Case A: a cert is compromised or mis-issued¶

The token is fine; one certificate it signed is not. KRL the serial and push. See rb20 for the exact commands and the KRL policy. Short cert validity keeps the exposure window small even between pushes.

7.2 — Case B: a token is lost or stolen¶

Ordered, because the order matters:

Confirm the other token works — before changing anything, verify you can still sign. (This is D4. If it fails, stop and fix that first.)
Delete the lost token's two CA lines from the trust files; commit; deploy fleet-wide and update the appliances by hand (rb20). Until this lands, a thief with the PIN can mint certs the fleet accepts.
Re-sign any host certs the lost token signed — they chain to a CA you just stopped trusting. rb26's online host CA covers routine host certs; anything signed by the dead 9d needs reissuing.
KRL any outstanding certs it signed that have meaningful validity left, rather than waiting for them to lapse.
Re-cut master.age without the lost token, and rotate the lab.kdbx master password — §7.6, rb47. lab.kdbx is password-only, so a stolen token alone holds no DB factor (arming also needs the identity file + FIDO2 PIN, both kept on atomic); this drops the lost key's arm path and closes the residual exposure.
De-register it at every external site (§5.2) and remove its passkey. A registered credential on a stolen token is a live second factor in someone else's pocket.
Update the records — rb20's fingerprint table, the lab.kdbx entry, §5.2.
Provision a replacement (§6). Until you do, you are single-fault.

7.3 — Case C: planned retirement of a token you still hold¶

Everything in Case B, plus — after the trust lines are removed and verified — wipe it:

ykman piv reset      # destroys its three CAs
ykman fido reset     # destroys its WebAuthn credentials (incl. its rb47 kdbx arm credential)
ykman otp delete 2   # clears the retired OTP slot (legacy KDBX factor — §3.5)

Wipe last. A token that has been reset cannot sign the certs you might still need during the transition.

7.4 — Case D: all tokens lost¶

Not a lockout. Ansible's transport is the static almalinux key, not a certificate, so the fleet remains reachable and manageable. Per-host break-glass (local accounts, the Proxmox console, OPNsense's console) is documented in rb20.

Recovery: provision a new token (§6), publish its CA lines, deploy, then delete the dead lines. New-trust-before-old-removal, so you are never trusting nothing.

7.5 — Time-to-safe¶

Action	Closes	Bounded by
KRL push	immediately — read at auth time	reaching each host
Deleting a CA trust line	on deploy	`--tags sshca` + the manual appliance push
Existing certs from that CA	on deploy (the CA is gone)	—
A stolen token's `lab.kdbx` arm path	only after re-cutting `master.age` + rotating the master password	§7.6, rb47
An unrotated external passkey	never, until de-registered	you, per site

7.6 — The step people miss¶

lab.kdbx is password-only (rb47), so on token loss there is no shared OTP secret to rotate. Two things to do instead, and neither is the old CR rekey:

Re-cut master.age without the lost token (rb47): re-run scripts/kdbx/make-master.sh after removing the lost key's identity-*.txt, so its recipient is dropped and only the surviving tokens can arm the DB.
Rotate the lab.kdbx master password (change it in KeePassXC, re-cut master.age against the new value, push the updated lab.kdbx to lab-secrets). This is the only real DB factor now — do it because the lost token, plus its identity file if it ever leaked from atomic, is the sole theoretical arm path.
Verify the surviving token can arm-kdbx and open the DB before you put it away.

History — the retired CR rekey. This used to be a single atomic session reprogramming a shared OTP CR secret on every token and rekeying the KDBX to match; partial completion locked you out. That factor is retired (§3.5) — do not reprogram OTP slot 2 as a DB factor.

7.7 — Rotating a CA (as opposed to retiring a token)¶

A fourth operation, distinct from the three in §7.0. Retiring removes a token you no longer have or want; rotating replaces a CA key with a fresh one while keeping the same number of tokens in service.

The important constraint: a CA key cannot be rotated in place (§3.8). It is generated on-device and non-exportable, so "rotate the user CA" really means provision a new CA, publish it alongside the old, migrate, then distrust the old — §6 followed by §7, with a period where the fleet trusts both.

That overlap is the point, and it is why this is safe:

Add the new CA's line to the trust file — trust is additive, so both work at once.
Deploy fleet-wide and push to the appliances by hand.
Re-sign the certs that chained to the old CA.
Only then remove the old line.

Never remove the old CA before every host has converged with the new one. The reassuring invariant throughout: Ansible's transport is the static almalinux key, not a certificate, so no CA change can lock you out of the fleet you need in order to fix it.

Rotating the YubiKey CAs is one phase of a much larger picture. The lab holds five root secrets and a dependency graph in which several credentials are required in order to rotate others — get the order wrong and you destroy the thing you need next. The full dependency-ordered program, the circular dependencies, and the catalogue of lockout orderings are runbook 46.

8 — Stress test: non-destructive drill suite¶

Everything here is safe to run on an enrolled, in-service token. Budget ~30 minutes for the full pass. Run it after enrolling a token (§6.5), annually on each token, and before relying on a token you have not used in a while.

A drill suite that has never been run is fiction. The two that matter most are D4 and D5 — they prove the claims most likely to have rotted silently.

D1 — Applet census. ykman info · piv info · fido info · otp info. Expect: PIV/FIDO2 enabled; slots 9a/9c/9d populated; PIN and PUK retries at full; no WARNING: Using default … for the PIN, PUK or management key. (OTP slot 2 may be programmed or empty — it is a retired factor now, §3.5, so its state is not a pass/fail criterion.) Record the serial (ykman info) alongside every result — with two tokens in play it is easy to attribute a finding to the wrong device. If it fails: an empty slot means this token is not what you think it is — treat as un-enrolled and go to §6. A default PUK or management key is a live finding, not a nit (§3.2, §3.3, §6.1.1).

Run ykman fido info over a direct USB session, not through a container. FIDO2 uses hidraw, not pcscd, so a toolbox/podman exec invocation reports Unsupported connection type — which is a plumbing artefact, not evidence that the applet is empty. Do not record a FIDO2 result gathered that way.

D2 — Every slot signs. Sign a throwaway user cert (9c), a throwaway host cert (9d), and renew the 9a login cert (menu U, H, L). Expect: three valid certs; ssh-keygen -L parses each. If it fails: one dead CA — the other token still works; re-provision this one.

D3 — Policies are enforced, not just requested. Start a 9c signature and do not touch the key; then perform two signatures after a single PIN entry. Expect: the untouched signature never completes (touch=always is real); the second signature needs no second PIN (pin-policy once is real). If it fails: the key was generated with different policies than documented — re-key it (menu R).

D4 — Standalone key test. (highest value) Unplug every other token. With only this one: renew the login cert, sign a canary host cert, sign an agent task cert, and arm lab.kdbx (rb47: arm-kdbx <this-key> on atomic, then kdbx-cli db-info from ctrl). Expect: all four succeed. If it fails: the redundancy claim is false and you are single-fault right now — fix before anything else. A failed arm means this token's FIDO2 recipient was never added to master.age (§6.3, rb47).

D5 — The KRL actually refuses. (highest value) Generate a scratch keypair, sign it a short cert, log in to a t1_canary host, then KRL its serial, scripts/push-krl.sh, and retry. Expect: login succeeds before, is refused after, with no sshd restart. If it fails: revocation does not work — every "we can revoke it" assumption in this lab is void until fixed. Never run this with your 9a cert or against a tier-0 host.

D6 — KDBX arm (presence-unlock). Arm lab.kdbx with this token alone (rb47: arm-kdbx <this-key> on atomic → kdbx-cli db-info from ctrl). Expect: success. If it fails: this token's FIDO2 recipient is missing from master.age, or its arm credential/PIN is wrong (§6.3, rb47) — not an OTP-slot problem (that factor is retired, §3.5).

D7 — FIDO2 is alive. ykman fido credentials list; complete one real WebAuthn login at a site where this token is registered (§5.2). Expect: the credential list is non-empty and matches your inventory. If it fails: the applet was reset at some point — re-register per §5.1, and treat the §5.2 inventory as the recovery list.

D8 — Fleet acceptance. With a cert from this token, SSH to one host per zone and an appliance (Proxmox/OPNsense). Confirm the key-id in the target's sshd log. Expect: all succeed. If it fails on appliances only: the manual appliance trust push was missed — a classic gap, since Ansible does not cover them.

D9 — Cert hygiene. ssh-keygen -L -f every live cert this token signed. Expect: correct principals, sane validity, distinct serials. If it fails: an over-broad or under-scoped principal — this is the exact check that catches the -n user mis-signing recorded in rb20.

9 — Destructive drills (recovery rehearsal)¶

These burn retry counters and erase applets. They rehearse the recovery paths that are otherwise untested until a real emergency — which is the worst time to discover a gap.

Get a third token first. D10–D12 require a sacrificial key ("key S") — never enrolled in the trust files, never added to master.age, never registered at any site. A token is cheap; running these on a live key means a forced re-enrolment cycle at best and a self-inflicted outage at worst. Without key S, read these and do not run them.

Safety rules: never drill with only one working token present · verify break-glass (almalinux key + console) before starting · never drill on a token holding an unexpired agent task cert · log every drill (date, token, result) in lab.kdbx.

D10 — PIN lockout → PUK recovery. (key S) Enter three wrong PINs; confirm signing is blocked; ykman piv access unblock-pin; confirm signing resumes. Proves: a blocked PIN is an inconvenience, not a loss — and that you know the PUK works before you need it.

D11 — PUK exhaustion → reset → re-provision. (key S) Exhaust the PUK; confirm the only remaining action is ykman piv reset; then run a full §6 provisioning against a scratch label. Proves: the terminal path really is terminal, and rehearses §6 and §7 in one pass. The highest-value drill in this section.

D12 — FIDO2 lockout → reset. (key S) Exhaust the FIDO2 PIN — note the 3-per-power-cycle sub-limit forces reinserts (§3.4) — then ykman fido reset. Proves: the FIDO2 applet is independent; PIV survives untouched. Also demonstrates why §5.2's inventory exists.

D13 — Lost-token fire drill. (the safe token, annually — no key S needed) Walk §7 Case B for real: pretend the safe token is gone, remove its trust lines, deploy, verify the fleet still works on the carry token, then re-enrol it per §6.

Defensible precisely because the safe token's contents are regenerable by design — and it exercises the parts of retirement that actually break: the trust-file edit, the Ansible deploy, the manual appliance updates, and host-cert re-signing.

10 — Unused capacity (the extension surface)¶

Everything below is present on the hardware and unused here. This is the section a downstream clone edits first.

Applet / slot	What it is	Why unused here	Adopt if…
OATH	TOTP/HOTP seeds	our external 2FA is WebAuthn where available	you depend on services that only offer TOTP — see §5.6 for the seed-backup rule
OpenPGP	3 GPG keys	we chose KeePassXC/KDBX over `pass`; commits are signed with SSH keys, not GPG	you move to `pass`, or need GPG-signed commits/email
PIV `9e`	Card Authentication, PIN-less	PIN-less contradicts the custody model	you need unattended badge-style auth — accepting that it weakens the model
PIV `82`–`95`	Retired Key Management	`82` holds a dead ED25519 `kdbx-unlock-A` key on token A — an artifact of an abandoned ssh-agent/PKCS#11 approach to the `lab.kdbx` presence-unlock; the shipped design (rb47) uses `age-plugin-fido2-hmac` (FIDO2, no PIV slot), so this key is inert and safe to leave or clear. `83`–`95` are free	you need to decrypt material sealed to a superseded key
Attestation `f9`	Yubico factory key that proves a slot's key was generated on-device	not yet adopted — see below	recommended

10.1 — Attestation is worth adopting¶

rb20's central Tier-1 claim — keys are "generated on-device, non-exportable" — is currently asserted. ykman piv keys attest 9c emits a certificate chaining to Yubico's root that makes it verifiable: proof that the key in that slot was generated on that device and never imported.

It is cheap, purely additive, breaks nothing, and turns a trust assumption into evidence. Proposed, not implemented — raise it with the operator before adding it to provisioning.

11 — Customizing this for your own lab¶

This runbook is written to be forked. What is instance-specific versus universal:

Universal (keep)	Instance-specific (change)
§1 device model, slots, policies	which slots you use, and for what
§2 the redundancy model and loss matrix	number of tokens; labels
§3 the five secrets and their retry counters	where you escrow them
§5.1 the both-keys-or-lockout rule	§5.2's site inventory — entirely yours
§7.0 revoke-vs-distrust distinction	your trust-file layout
§8/§9 the drill suite	which hosts are canaries

Two tokens is the floor, not the goal. One token is a single point of failure with no backup by construction. A third (key S, §9) converts the recovery drills from terminal to routine, which is the only way they get run at all.

Extending the model: adding a use — a new applet, a new site, a new slot — should answer the same three questions this runbook answers for every existing use: what secret gates it, what happens when that secret is exhausted, and how does the second token cover it? If the third question has no answer, you have added a single point of failure.

Validation¶

[ ] ykman info on each token shows the expected applets (D1).
[ ] Each token independently signs a user cert, a host cert, and its own login cert (D2).
[ ] Each token alone — every other unplugged — performs a full working day's operations, including arming lab.kdbx via the rb47 presence-unlock (D4).
[ ] A revoked cert is refused fleet-wide after scripts/push-krl.sh, with no restart (D5).
[ ] Cert principals and validity are as intended on every live cert (D9).
[ ] Both tokens are registered at every site in §5.2, and recovery codes are in lab.kdbx.
[ ] The lab.kdbx master password exists outside the database (safe/paper), and rb47's master.age + identity-{A,B}.txt are backed up off atomic (§3.6).
[ ] rb20's enrolled-token fingerprint table matches the tokens you actually hold.