Runbook 09 — Origin: internal reverse-proxy (Caddy) + hidden Bind primary

Goal: stand up the origin half of the edge/origin design. The reverse proxy is Caddy on its own small VM in DMZ — it terminates TLS, obtains/renews Let's Encrypt certs automatically via DNS-01/RFC2136 (this is your ACME), serves HTTP/3, and reverse-proxies to backends. Bind is the hidden authoritative primary for your public zones (internal, not delegated).

Both are internal-only — there are no WAN port-forwards. Public traffic reaches the origin Caddy, and zone transfers reach the secondaries, over the edge WireGuard tunnels (runbook 17). SSO/MFA in front of apps is added in runbook 13 (Authelia) via Caddy's forward_auth.

Prereqs: runbooks 06–08 validated. (Edge VPSes are runbook 17 / Phase 2 — this runbook builds the origin side, which works internally on its own first.)

Edge/origin recap (CLAUDE.md): the ISP and the VPS vendors are adversaries, so home serves nothing publicly and no vendor sees plaintext. The edges do TLS-SNI passthrough (caddy-l4) to this origin Caddy, which terminates TLS. The hidden Bind primary feeds public Bind secondaries on the edges via AXFR; the registrar delegates to the edges, never to home.

---

0. Baseline the provisioned VMs first (Ansible — fleet-wide, once)

This is the right point for the Ansible common baseline: the VMs are provisioned (runbook 01), the network + firewall are up (03–06) so they have DNS + egress + are reachable, and no app software is installed yet — so every VM is hardened/configured before it carries a service. Run it once for the whole fleet now (it covers Caddy, Bind, FreeIPA, Authelia), not per-app.

From an AlmaLinux admin host on MGMT (the firewall lets MGMT reach all zones):

sudo dnf -y install ansible-core         # distro-maintained LTS — pinned by EL9
cd ansible
ansible all -m ansible.builtin.ping      # SSH + Python reachable?
ansible-playbook playbooks/site.yml      # common baseline → every VM, serial:1

(Control node not EL? pipx install 'ansible-core==2.16.*' and pin that line.) The common role applies: timezone, EPEL, qemu-guest-agent, fail2ban, SSH hardening (no root, key-only), and dnf-automatic. Full details + conventions are in runbook 16.

Re-run site.yml after runbook 10 (FreeIPA) so the common role can add FreeIPA CA trust (lab_ipa_ca_file) to every host — it's idempotent.

---

Host allocation (static; record as host-overrides)

Host	Zone	IP	Purpose
caddy (origin)	DMZ	10.20.30.10	internal reverse proxy / TLS terminator
Nextcloud	DMZ	10.20.30.20	Nextcloud (web + Talk) backend
bind (hidden primary)	DMZ	10.20.30.40	internal authoritative primary; AXFR → edge secondaries
(other sites)	DMZ	10.20.30.x	web backends
sso (Authelia)	SERVERS	10.20.20.41	SSO / forward-auth (runbook 13)

No public SSH bastion. Admin access is the road-warrior WireGuard (.69:51820, runbook 07) — the single inbound exception. There are no WAN service forwards (no port-22, no 80/443/53).

---

1. No WAN port-forwards (this is the point)

Under edge/origin there are no Destination NAT rules for Caddy or Bind. Do not forward 80/443/53/22 on WAN. The only WAN inbound is the WG admin endpoint (.69:51820, runbook 06/07).

How the public actually reaches these internal hosts: - HTTPS: client → edge:443 → edge Caddy-L4 SNI-passthrough → WG edge tunnel → origin Caddy 10.20.30.10:443 (terminates TLS) → backend. - Public DNS: the world queries the edge Bind secondaries; those pull the zone from 10.20.30.40 (hidden primary) via AXFR/IXFR over the WG edge tunnel (TSIG). The hidden primary is never queried by the public.

The pass rules permitting edge tunnel → 10.20.30.10:443 and edge tunnel → 10.20.30.40:53 live on the WG-edge interface, source-restricted to the per-VPS tunnel /32s (runbook 17). Until the first edge exists (Phase 2), both hosts are reachable/​testable internally (from MGMT / via Unbound host-overrides).

---

2. Build the origin Caddy VM (from your AlmaLinux template)

Clone your AlmaLinux 9 template into a small VM in DMZ (10.20.30.10, gw .1, DNS 10.20.20.1; 1–2 vCPU, 1–2 GB RAM) — follow runbook 01 for the clone + NIC-tag (tag=30) + identity-hygiene steps.

Caddy now ships as a container, not a native build. As of 2026-06-28 the origin Caddy runs the shared B9 image (one digest-pinned image with the rfc2136 DNS plugin and caddy-l4 baked in, built in CI to the internal zot registry) as a podman quadlet via the caddy_container role — this replaces the per-host dnf copr install + xcaddy build shown below. Follow runbook 29 for the container model and the staged plane conversion; the native-install steps here are kept as first-principles background (and still describe the Caddyfile the container mounts).

Native build (retired — background only):

dnf -y install dnf-plugins-core
dnf -y copr enable @caddy/caddy
dnf -y install caddy
caddy version

DNS-01 is the chosen challenge (wildcards, no reliance on inbound :80 — home accepts no public inbound). Because you self-host authoritative DNS (Bind), the natural method is RFC2136 — Caddy makes TSIG-authenticated nsupdate calls to the local hidden primary to write the _acme-challenge TXT records. The shared B9 image already bundles the rfc2136 plugin (runbook 29); the native xcaddy build it replaced (the packaged Caddy has no DNS plugins) was:

dnf -y install golang
go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest
~/go/bin/xcaddy build --with github.com/caddy-dns/rfc2136
install -m0755 ./caddy /usr/bin/caddy && systemctl restart caddy

Prereq on Bind: define a TSIG key and allow-update for the challenge records (or a dedicated _acme-challenge zone). Give Caddy the key (next step). The challenge TXT is written on the primary and rides AXFR to the secondaries, so Let's Encrypt (querying the public edges) sees it. (Alternative: a cloud-provider plugin + API token if you'd rather not update your own zone.)

---

3. Caddyfile — automatic HTTPS + routing

/etc/caddy/Caddyfile:

{
    email admin@example.com        # ACME account (Let's Encrypt)
    acme_dns rfc2136 {               # DNS-01 against your own hidden primary
        server  "10.20.30.40:53"
        key_name "acme."
        key_alg "hmac-sha256"
        key     {env.RFC2136_TSIG_KEY}
    }
}

cloud.${domain} {
    reverse_proxy 10.20.30.20:11000    # Nextcloud AIO apache
}

# add one block per site …
# site2.${domain} { reverse_proxy 10.20.30.21:8080 }

Supply the TSIG key to Caddy's service (not in the Caddyfile): create /etc/caddy/caddy.env with RFC2136_TSIG_KEY=... (mode 0600), and add EnvironmentFile=/etc/caddy/caddy.env via systemctl edit caddy.

That's it — Caddy obtains and auto-renews the certs (DNS-01), redirects HTTP→HTTPS, and serves HTTP/3. Reload after edits:

caddy validate --config /etc/caddy/Caddyfile
systemctl reload caddy

Why TLS terminates here, not at the edge: the edge does dumb SNI passthrough (forward_tcp of the still-encrypted stream), so the cert + private key live only on this origin box. No VPS vendor can read the plaintext or impersonate the site. The public hostname (cloud.${domain}) resolves to the edge IP; the edge forwards the bytes here; this Caddy presents the cert.

The forward_auth block that gates sites behind Authelia SSO/MFA is added in runbook 13 — for now these sites are open (stand up only non-sensitive ones until SSO is in place).

---

4. Bind — hidden authoritative primary (DMZ, internal)

Bind is the hidden primary: authoritative source of truth for your public zone(s), but internal-only — not in the registrar's NS records and not publicly reachable. It serves AXFR/IXFR to the edge secondaries (which are delegated) over the WG edge tunnels, and answers the origin Caddy's RFC2136 ACME updates. It is not an internal resolver (that's Unbound, runbook 04).

• Host: clone the AlmaLinux template into DMZ (10.20.30.40, runbook 01); dnf -y install bind bind-utils (service named).
• Zones: author your zone file(s) here; set this server as primary (type primary). The public NS/A records inside the zone point at the edge IPs (clients hit the edges), even though this box is where you edit.
• Listen / access (internal only): listen-on { 127.0.0.1; 10.20.30.40; }; No WAN exposure. Bind to the DMZ + the WG-edge tunnel reachability only.
• TSIG + transfer to secondaries: define an AXFR TSIG key and allow-transfer { key edge-xfr; }; (per-zone), plus also-notify the edge secondaries' tunnel IPs so changes propagate immediately. The secondaries' config (runbook 17) sets primaries { 10.20.200.1 key edge-xfr; } (the home tunnel IP). Keep a second TSIG key (acme.) for the RFC2136 challenge updates, scoped tightly (ideally only _acme-challenge records).
• Hardening: authoritative-only — recursion off (recursion no;). Even though it's internal, keep it from ever acting as a resolver.

Delegation lives at the registrar → edges, not here. You do not open WAN:53. The registrar's NS records point at the edge secondaries (runbook 17 / Phase 3 sets up NS diversity across vendors). This box is the quiet origin behind them.

Escrow — bind's state is NOT rebuildable (do this before trusting any rebuild)

bind is one of the two DMZ hosts holding state peers depend on (the CLAUDE.md statefulness caveat): the per-zone DNSSEC CSKs in /var/named/dynamic/ (plus key-manager .state files) and the zone files themselves (this box is the source of truth). Losing them ≠ a rebuild inconvenience: after DS records are published at the registrars, losing the CSKs takes the zones dark for validating resolvers until a registrar-side DS change propagates.

Escrow (first taken 2026-07-13; re-take after any key rollover or zone add):

# on bind:   flush journals, tar the whole dynamic dir
rndc sync -clean
tar czf /root/bind-dnssec-escrow.tgz -C /var/named dynamic
# on ctrl:   fetch, then base64 into the gitignored vault file
#             ansible/group_vars/all/bind-dnssec-escrow.vault.yml
#             (vault_bind_dnssec_escrow_tgz_b64 + date + sha256; ansible-vault encrypt)
# verify:    ansible-vault view | b64 -d | sha256sum  ==  source hash
# cleanup:   shred -u both plaintext tarballs; operator: attach a copy to lab.kdbx

Restore onto a fresh bind VM: run the bind_primary role (config layer), then ansible-vault view ... | grep _b64 | cut -d' ' -f2 | base64 -d | tar xzf - -C /var/named, chown -R named:named /var/named/dynamic, rndc reload. Confirm serials + RRSIGs (dig @localhost <zone> SOA +dnssec) and that the edges re-sync (dns-watch stays green). The TSIG secrets (edge-xfr, acme) are already vaulted separately.

5. DMZ egress / isolation reminder

Runbook 06: DMZ→RFC1918 blocked except the specific backend/Authelia path; DMZ outbound rides Proton; inbound only via the edge tunnels (WG-edge interface rules, runbook 17). Confirm a compromised origin box can't reach MGMT/clients and only reaches the exact SERVERS ports it needs.

---

9. Server-side visit counting (opt-in per vhost)

The problem. Some public sites deliberately ship no analytics JavaScript — their whole claim is that the page never phones home, and that a saved copy of the .html behaves exactly like the hosted one. The lab's umami plane (stats.${domain}) can't serve them: it needs a beacon in the page. But "how many people visited?" is still a fair question.

The answer. Count the request the browser already made. Caddy writes one access-log line per request; the page stays silent. Nothing new is installed — no GoAccess, no log file, no logrotate, no extra listener — because the transport already exists:

caddy (stdout, JSON) → podman journald → Alloy → Loki → Grafana

Turn it on by adding one key to that vhost in origin_vhosts_extra (zz-local.yml):

  - host: metronome.${domain_alt}
    root: /var/www/metronome
    encode: true
    access_log: true          # <- the only change

Then converge: ansible-playbook playbooks/site.yml --limit caddy.

Default is off for every vhost, so no other site logs a single request.

What gets recorded — and what deliberately does not. The filter encoder rewrites the line before it is written, so the discarded fields exist nowhere — not in the journal, not in Loki, not in a backup:

Recorded	Deleted in the encoder
ts, request.method, request.host, request.uri	request.headers (all of it: User-Agent, Referer, Cookie)
status, size, bytes_read, duration	request.tls (SNI, cipher, version)
request.remote_ip / client_ip masked to /24 (v6 /48)	resp_headers, request.remote_port

The result is a tally, not a profile: no UA string, no referrer, no cookie, and an address that can't identify a household. The real client IP survives the edge hop (the edges prepend PROXY-protocol, runbook 17) and is masked here, at the origin.

⚠️ Only opt in a site whose per-user state rides the URL fragment. A fragment (#...) is never sent to a server; a query string is, and would land verbatim in uri. Check before flipping the flag on a site that shares links.

Where the counting rules live: the dashboard, not the terminator. Caddy logs the raw request; the Grafana dashboard Site visits (server-side) (site-visits.json, runbook 18) decides what counts — a page view is GET of / or *.html with 2xx/304 (so js/icons/webmanifest fall out), artifacts (.uf2/.zip/.py/.mpy) count as downloads, and "visitors" is distinct masked /24s. Retuning that never touches the public TLS terminator. The dashboard is generic — it groups by request_host, so any newly opted-in vhost appears on it with no edit.

Read the numbers honestly (the panel says this too): a shared network undercounts visitors and a roaming phone overcounts; an installed PWA with a service worker serves repeat visits from cache and never reaches the server, so counts skew to first visits; and bots are included (no UA = no bot filtering).

Two horizons — the log expires, the count does not. Raw lines are request records, so they get the short retention: Loki drops them at 30 days (obs_loki_retention). To answer "how many people came" over years without hoarding requests, the count is promoted to a metric — alloy_access_log_metrics: true (set in host_vars/caddy.yml) makes that host's Alloy parse its own Caddy access lines and export

site_http_requests_total{vhost="…", class="page|download|other"}

remote-written to VictoriaMetrics and kept for obs_vm_retention (13 months). The durable artifact of a visit is then an aggregate, not a request record — better for privacy and for the long-range question. Notes worth keeping:

• Labels are a closed set (vhost + class). The path is never a label — bots probe arbitrary URLs, so a path label is unbounded cardinality. Per-path detail stays in Loki, where it expires.
• Distinct visitors cannot be a counter (that needs a distinct-count), so that figure stays 30-day/log-only. The metric answers how many visits, not how many people.
• Alloy exports a stage.metrics counter as loki_process_custom_<name>; the role keeps only that series, renames it, and drops alloy's plumbing labels — none of Alloy's internal metrics reach VictoriaMetrics.
• Classification lives in one Go-template line in the Alloy config (page/download/other) and is the same rule the dashboard uses on the logs, so the two horizons agree.

---

Validation

(Phase 1 — internal, before any edge exists. Test from a MGMT host, or add a temporary Unbound host-override so cloud.${domain} → 10.20.30.10 internally.)

• [ ] https://cloud.${domain} (internal) serves a valid Let's Encrypt cert (auto-issued by Caddy) → backend; HTTP→HTTPS redirect; HTTP/3 (h3).
• [ ] Multiple sites route correctly by hostname.
• [ ] caddy validate passes; renewal works (Caddy logs show cert management).
• [ ] Cert issued via DNS-01/RFC2136 (Caddy writes the TXT to the primary; no inbound :80 needed); auto-renew OK.
• [ ] Bind (primary) answers your zone internally (dig @10.20.30.40 ${domain} SOA); recursion off (dig @10.20.30.40 google.com → REFUSED); AXFR allowed only with the TSIG key (anonymous AXFR refused).
• [ ] No WAN forwards exist: Firewall → NAT → Destination NAT has no 80/443/53/22 rules; an external scan of the Comcast IP shows only UDP 51820 on .69.
• [ ] Origin Caddy/Bind can reach their backends; cannot reach MGMT/clients (DMZ isolation from runbook 06 holds).
• [ ] Visit counting (§9), only on vhosts that opted in: fetch a page publicly, then journalctl -u caddy-proxy | grep http.log.access on the origin — the line must show the client IP ending in .0 (masked) and contain no headers, tls, or resp_headers object. A vhost without access_log: true must produce no access line at all. The same line then appears in Loki ({job="journal"} |= "http.log.access") and on the Site visits dashboard.

(Phase 2 end-to-end — through a real edge — is validated in runbook 17.)

Then → runbook 10 (FreeIPA). Edge build is runbook 17 (after the origin works).