Runbook 13 — Authelia SSO/MFA + Caddy forward-auth + OIDC¶

Goal: an identity-aware SSO layer in front of the lab's apps. Authelia (sso, 10.20.20.41) authenticates users — SSO + MFA — against FreeIPA (LDAP), and either (a) gates a site via Caddy forward_auth before traffic reaches it, or (b) is the OIDC provider for apps that speak OIDC natively (Grafana, the control-plane Forgejo). It replaced Authentik on 2026-07-06 — same FreeIPA-backed model, far less machinery.

Prereqs: runbook 10 (FreeIPA) + runbook 09 (origin Caddy). This is the "zero-trust / BeyondCorp" pattern: authenticate before the app, using one identity store.

Why Authelia over Authentik. Authentik was a Docker-host VM running server + worker + PostgreSQL + Redis. Authelia is a single Go binary in a podman-quadlet LXC, config- as-code (one configuration.yml, secrets inline from vault like searxng's settings), SQLite + filesystem notifier — no DB/cache/worker to run or back up. Lower churn, smaller blast radius, the same LDAP→FreeIPA identity and Caddy integration. The authelia role owns it end to end.

Placement¶

sso — an unprivileged LXC in SERVERS (10.20.20.41), authelia role, one podman quadlet (docker.io/authelia/authelia:4.38.x, digest-verify at deploy — the config schema is version-sensitive). Runs as a deterministic host uid:gid; /var/lib/authelia bind-mounts /config (config + SQLite db + notifications).
Its portal (auth.${domain}) is published through the origin Caddy like any site; the app traffic itself is gated by forward_auth or brokered by OIDC.
LDAP source → FreeIPA (ldaps:// to ${ipa_server}, the shared svc-ldap read bind, same DNs/filters the fleet uses; fleet-admins is the admin group). tls_skip_verify: true — encrypted to a trusted internal SERVERS box (harden later by mounting the IPA CA).

0. One-time: the `svc-ldap` read bind (FreeIPA sysaccount)¶

Authelia (and Jellyfin, runbook 21) bind to FreeIPA as the shared svc-ldap read-only system account — uid=svc-ldap,cn=sysaccounts,cn=etc,${basedn}. It's a directory sysaccount (not an IPA user, so it's HBAC-immune and never a login identity), created once on the FreeIPA server. Its password is vault_ldap_bind_password in the vault (the same value every LDAP consumer uses):

# on the FreeIPA server, with the vault_ldap_bind_password value at hand:
ldapadd -x -D "cn=Directory Manager" -W <<EOF
dn: uid=svc-ldap,cn=sysaccounts,cn=etc,${basedn}
objectClass: account
objectClass: simplesecurityobject
uid: svc-ldap
userPassword: <vault_ldap_bind_password>
passwordExpirationTime: 20380119031407Z
nsIdleTimeout: 0
EOF

Verify the bind works: ldapsearch -x -D uid=svc-ldap,cn=sysaccounts,cn=etc,${basedn} -W -b cn=users,cn=accounts,${basedn} uid=operator returns the user.

1. Deploy (the `authelia` role)¶

Config-as-code — nothing manual. Secrets are injected per-play from the vault (see the sso play in playbooks/site.yml): authelia_session_secret, authelia_storage_encryption_key, authelia_reset_jwt_secret, authelia_oidc_hmac_secret, authelia_oidc_issuer_key (RSA PEM), the LDAP bind password, and each OIDC client secret.

cd ansible && ansible-playbook playbooks/site.yml --limit sso

The template (roles/authelia/templates/configuration.yml.j2) renders configuration.yml (0640); the quadlet restarts on change. Verify: podman logs authelia clean, https://auth.${domain} serves the portal, an IPA user can log in.

2. Forward-auth (gate a site in Caddy)¶

For sites without native OIDC, Caddy asks Authelia to authorize each request. Protected domains are authelia_forward_auth_domains (role default: search, fmd) + instance overlays (authelia_access_rules_extra — e.g. a casey-only app, or a gated sub-path). In the origin Caddy:

auth.${domain} {
    reverse_proxy 10.20.20.41:9091          # the Authelia portal
}

search.${domain} {
    forward_auth 10.20.20.41:9091 {
        uri /api/authz/forward-auth
        copy_headers Remote-User Remote-Groups Remote-Name Remote-Email
    }
    reverse_proxy <backend>
}

Per-domain access rules live in access_control (rendered from authelia_forward_auth_domains + authelia_access_rules_extra): a bare {domain} gates exactly the forwarded requests to any authenticated user; add subject: ['user:casey'] to restrict, or resources: for path regexes (path scoping is usually done in Caddy — only the gated paths are forward_auth'd). Default policy is one_factor (LDAP password); set two_factor to force TOTP/WebAuthn.

Cross-apex gating. Authelia scopes a session to its cookie domain, so a gated site on a different apex than ${domain} (a subdomain of one apex can't share another apex's cookie) needs its own cookie domain and portal. Four pieces: (1) authelia_cookie_domains_extra — {domain: <apex>, authelia_url: https://auth.<apex>}; (2) an auth.<apex> portal vhost (origin_vhosts_extra → Authelia :9091); (3) a public DNS record auth.<apex> → edge; (4) auth.<apex> in the edge SNI passthrough (edge_sni_names_extra). Miss any and Authelia 400s (it can't manage a session for an unconfigured apex). Example: a gated sub-path on a second apex — example.org/media-trust/* — gates via auth.example.org.

Long URLs and the header budget (431 Request Header Fields Too Large). Caddy's forward_auth replays the entire original browser header set to Authelia and adds X-Forwarded-Uri carrying the full path+query — so a long query string is charged to the budget twice (once as the URI, again in the browser's Referer on in-app navigation). Authelia is fasthttp-based and its server.buffers.read defaults to 4096 B; past that it returns 431, which Caddy relays verbatim (Via: 1.1 Caddy on the response — the error is Authelia's, not Caddy's, which happily accepts 6 KB+ on a non-gated vhost). Hit on 2026-07-26 by research.${domain} (rb44), whose UI puts whole prompts in the query string: a 1.7 KB query = a 1742 B X-Forwarded-Uri, 42% of the old budget before one browser header. The write side matters too — Authelia's 302 to the portal embeds the original URL doubly-encoded in ?rd=, so its Location header alone was ~2.4 KB (58% of the old 4096); left too small this fails as a broken redirect, which is harder to read than a clean 431. Both are now raised to 16384 (authelia_buffer_read / authelia_buffer_write). This is plane-wide, not a research quirk: it applies to every Tier-1 forward-auth'd vhost. To diagnose, bisect the hop — backend direct, then Authelia direct, then the public URL:
curl -sS -o /dev/null -w '%{http_code}\n' "http://<backend>/<long-url>"        # app itself
curl -sS -o /dev/null -w '%{http_code}\n' -H "X-Forwarded-Uri: <long-uri>" \
     -H 'X-Forwarded-Method: GET' -H 'X-Forwarded-Proto: https' \
     -H 'X-Forwarded-Host: <vhost>' http://10.20.20.41:9091/api/authz/forward-auth
curl -sS -o /dev/null -w '%{http_code}\n' -H 'X-Junk: <6KB>' https://<gated-vhost>/   # vs a
curl -sS -o /dev/null -w '%{http_code}\n' -H 'X-Junk: <6KB>' https://myos.io/         # non-gated
Config-key changes are version-sensitive (server.buffers.* replaced 4.37's server.read_buffer_size) — validate against the pinned image before converging: docker run --rm -v $PWD/cfg.yml:/cfg.yml:ro authelia/authelia:<pinned-tag> \ authelia validate-config --config /cfg.yml (it reports configuration key not expected).

Apps that do OIDC skip forward-auth and log in against Authelia. Clients live under identity_providers.oidc.clients in the config; the issuer is https://auth.${domain}. Current clients:

Grafana — redirect_uris: [https://mon.${domain}/login/generic_oauth], fleet-admins→Admin.
Forgejo (control-plane, git.${domain}) — IPA users log into the private git via Authelia. Forgejo is a DMZ box that can't reach the identity core, so its OIDC token-exchange hairpins through the origin Caddy (AddHost=auth.${domain}:<origin> in the quadlet; TLS still verified — the origin holds a valid cert). Auto-registration is bounded (Authelia only authenticates IPA users — no public signup); new users land as plain users; fleet-admins→site admin via the groups claim; the local admin stays break-glass. The public forgejo-pub is deliberately NOT a client — keeping the internet-facing instance off the identity core (runbook 35 exposure model). Wired by the backends role (forgejo_oidc_enabled: true).
Seafile (files.${domain}) — IPA users log into Seafile (web + sync-client browser login) via Authelia. Redirect authelia_oidc_seafile_redirect (https://files.${domain}/oauth/callback/); enabled by the backends role (seafile_oidc_enabled: true, set in host_vars/files.yml). Also a DMZ box, so like Forgejo its token-exchange hairpins through the origin Caddy (AddHost in the seafile quadlet). Seafile/oauthlib sends the secret via HTTP Basic, so the client is token_endpoint_auth_method: client_secret_basic, require_pkce: false; the local seafile_admin stays break-glass (runbook 41).

Merging a pre-SSO local account with its SSO identity. ACCOUNT_LINKING = auto only links when the auto-registered username/email would collide with an existing account — otherwise the first OIDC login mints a fresh, empty user (this is how the operator ended up with casey beside the manual account). Forgejo resolves an OIDC login by (1) user.login_name+login_source+login_type (how auto-registered users are stored), then (2) an external_login_user row (what the UI "link account" flow writes). To merge (done 2026-07-10 for the operator account): back up gitea.db, forgejo admin user delete --id <auto-created> (verify it owns nothing first), rename the old account to the SSO username via POST /api/v1/admin/users/{old}/rename ({"new_username": …}; a throwaway write:admin token, revoked after), then INSERT INTO external_login_user (external_id, user_id, login_source_id, provider) VALUES ('<authelia subject UUID>', <uid>, <source id>, 'authelia'). The account keeps login_type=0, so the local password (break-glass) keeps working alongside SSO — never convert it to login_type=6, which would kill password login.

Add a client = a new entry in the template (mirror Grafana) + its client_secret in the vault + wired in the sso play. require_pkce: true where the client supports it (forgejo:9's client does not send PKCE, so it stays false).

4. MFA¶

Add a TOTP/WebAuthn second factor by setting a domain's policy to two_factor (or globally via authelia_access_policy). WebAuthn/passkeys are the forward-looking choice. Enrolment happens at the auth.${domain} portal.

Validation¶

[ ] https://auth.${domain} serves the Authelia portal; an IPA user logs in (LDAP).
[ ] A forward_auth'd site (search.${domain}) redirects to the portal, requires login, then lands on the app; a user not in the allowed group/subject is denied.
[ ] SSO: after one protected app, a second doesn't re-prompt (shared session cookie on ${domain}).
[ ] Unauthenticated direct hits to a backend are blocked at Caddy (no bypass).
[ ] OIDC: Grafana + the control-plane Forgejo log in via Authelia; fleet-admins get admin.
[ ] OIDC: Seafile (files.${domain}) logs in via Authelia (web + sync-client browser flow); the local seafile_admin still works as break-glass.
[ ] Authelia is not internet-reachable except its portal via Caddy; it sits in SERVERS.

Then → runbook 14 (end-to-end validation).