Invidious) + Home Assistant¶

Stands up the three media services as AlmaLinux 9 + podman quadlets (the locked clean-build path, same machinery as Forgejo/SearXNG/FMD), plus the Home Assistant OS appliance VM on the SERVERS VLAN (HAOS is an appliance, like OPNsense — not a quadlet-managed EL box).

The media stack¶

Host	What it is	Home	How
`jellyfin`	Jellyfin media server	EL9 quadlet (`9123`, SERVERS `.30`)	clean EL9 quadlet; library via NFS from the fileserver
`tvheadend`	DVR/tuner backend, USB tuner `2040:826d` (Hauppauge)	Debian VM (`9124`, SERVERS `.31`)	EL9 lacks the `lgdt3306a` demod, so Debian + LinuxServer quadlet; claims the USB tuner + serves Jellyfin Live TV
`invidious`	privacy YouTube frontend	EL9 quadlets (`9125`, DMZ `.70`)	app + pg + companion quadlets
`haos`	Home Assistant OS + Zigbee USB `10c4:ea60`	HAOS appliance VM (`105`, SERVERS `.45`)	appliance VM on the SERVERS VLAN; Zigbee via USB coordinator

Tuner split is the point. The DVR backend (tvheadend) is separated from the player (Jellyfin): tvheadend owns the USB tuner and feeds Jellyfin's Live TV over HTSP.

Architecture¶

Jellyfin — SERVERS 10.20.20.30:8096. Config/metadata in /var/lib/jellyfin; the movie/TV library is an NFS automount from the fileserver (nfs_mounts in host_vars/jellyfin.yml), not local disk. CPU transcode (pass /dev/dri later for VAAPI/QSV). Reached by Roku/IOT (8096, rb06), WG users (rb07), and jellyfin.${domain} via the apps-proxy (Tier 2, internal split-horizon).

Auth = FreeIPA LDAP, not Authelia SSO. Jellyfin's clients are native apps (Roku/mobile) that can't do a forward-auth/OIDC browser redirect, so Jellyfin uses its LDAP plugin bound to FreeIPA for accounts — same UID/identity, no SSO.
tvheadend — SERVERS 10.20.20.31. Debian VM (template 9000) — RHEL/EL9 stripped the DVB stack and ELRepo's kmod-media lacks the lgdt3306a demod this Hauppauge WinTV-dualHD needs; Debian ships it in-kernel so /dev/dvb (dual adapters) works OOTB. Runs the LinuxServer tvheadend quadlet with /dev/dvb passed in (the backends role is OS-agnostic for podman/NFS; common/alloy are EL-only and skipped here). Web/HTSP 9981, HTSP streaming 9982. Recordings → fileserver NFS. Jellyfin's Live TV points at 10.20.20.31:9981 (intra-SERVERS, no fw rule).
Invidious — DMZ 10.20.30.70:3000, quadlets on a private podman net (invidious + invidious-db postgres + invidious-companion — the YouTube innertube/PoToken/signature backend). Tier-2 private, served via the apps-proxy at https://invidious.${domain} (like Jellyfin) — a split-horizon Unbound override → apps-proxy reverse-proxies to this backend :3000 (needs the SERVERS→DMZ allow, rb06). Not behind the origin Caddy / Authelia and not delegated through an edge; also reachable directly at http://10.20.30.70:3000 on the LAN. FreeTube can use it as its Invidious backend.
Home Assistant — SERVERS 10.20.20.45:8123 (appliance VM, not EL9/quadlet). Zigbee/Z-Wave devices ride the USB coordinator (radio, not network), so HA doesn't need L2-adjacency to the IoT VLAN; the IP IoT devices are reached via an explicit HOMEASSISTANT → IOT allow + mDNS reflection. Reached by clients at :8123 (rb06 §WS), ha.${domain} via the apps-proxy (Tier 2), and the HA companion app over WG when away.

Configure HA from the CLI — `scripts/ha-ws.py` (no GUI, no add-on)¶

Most HA "configuration" is not in configuration.yaml — areas, device/entity names, entity ids, room-group membership and helper options live in UI-managed registries behind the frontend's WebSocket API, and a long-lived access token (the same vault_ha_token the rb34 update probe uses) reaches all of it. scripts/ha-ws.py is a stdlib-only client: each JSON command on the argv gets sent after auth and its result prints as one JSON line.

cd ansible
export HA_TOKEN=$(ansible localhost -m debug -a "msg={{ vault_ha_token }}" \
  -e @group_vars/all/ha-check.vault.yml 2>/dev/null | grep -oP '"msg": "\K[^"]+')
../scripts/ha-ws.py '{"type":"config/area_registry/list"}'

Worked example (2026-08-16, the moved kitchen bulb): a physically relocated smart bulb was re-homed entirely from ctrl — search/related first (nothing referenced the old entity id — always check refs before renaming, a renamed entity id silently breaks any automation/scene that used it), then config/device_registry/update (area + name), config/entity_registry/update (new_entity_id), and the room-group membership via the REST options flow (flows are REST, not WS — same Bearer token):

curl -s -X POST -H "Authorization: Bearer $HA_TOKEN" -H "Content-Type: application/json" \
  -d '{"handler":"<group config_entry_id>","show_advanced_options":false}' \
  http://10.20.20.45:8123/api/config/config_entries/options/flow          # -> flow_id
curl -s -X POST ... /api/config/config_entries/options/flow/<flow_id> \
  -d '{"entities":[...full member list...],"hide_members":false,"all":false}'

Finish any change with a physical or state-level verification (a service call through the changed path — e.g. light/turn_on on the room group — beats trusting the registry write). Boundaries: configuration.yaml and everything Supervisor-side (add-ons, backups) are reachable by neither API — /api/hassio/* rejects even owner tokens (only a few read paths like core/logs are proxied); file edits need the SSH or File-editor add-on, deliberately not standing infrastructure here.

Egress note — the YouTube consumers (single Proton exit each, NOT the LB group)¶

Invidious hammers YouTube/Google, which frequently blocks ProtonVPN exit IPs — and, sharper than that: YouTube PO tokens are bound to the session's exit IP, so the companion's session breaks whenever the load-balanced Proton group re-homes the host onto a different tunnel (sticky expiry), even to a non-blocked exit. Observed 2026-07-10→12: days of flapping Failed to validate PO token in the companion journal, then hard-down when stickiness parked the host on a Google-rejected exit → every watch page "Video unavailable" while search still worked (browse API isn't PO-token-gated).

Fix (2026-07-12): the invidious host's internet egress is pinned to a single Proton tunnel via an automation-filter rule — INVIDIOUS → !RFC1918, gateway PROTON_GW2 (provision/opnsense/fw-automation-rules.json, applied by scripts/opnsense-fw.py). Still all-via-Proton, still fail-closed; the rule only removes the per-connection exit hopping for this one host. If that exit's reputation goes bad (token failures return): rotate gateway to another PROTON_GW* in the JSON, apply, then restart companion with a cleared session cache: systemctl stop invidious invidious-companion && rm -rf /var/lib/invidious/companion-cache/* && systemctl start invidious-companion invidious. Also try bumping the image digests first (see the caveat in ansible/roles/backends/defaults/main.yml — upstream ships cat-and-mouse fixes often).

Detection is codified (2026-07-12): the Grafana rule fleet-invidious-po-token (runbook 27) watches the companion's journal in Loki for PO-token failure lines and pages ntfy lab-alerts within ~10–30 min of a recurrence — the 2026-07-10→12 incident ran ~2 days undetected because the companion's unit stays green while playback dies. An ntfy page from that rule is the trigger for the rotation procedure above.

Last resort only (operator decision — exposes the home IP to Google): add Google's ranges to the VPN_BYPASS alias (runbook 05 §6 / runbook 06 §IOT) so only invidious→Google exits the real WAN. Don't pre-open it.

The fileserver gets the same pin (2026-08-15) — seq 590, deliberately a DIFFERENT gateway (PROTON_GW1). media-fetch/yt-dlp is the lab's second YouTube consumer and hit the same wall on its first phone-dispatched fetch (429 → "confirm you're not a bot", gone on retry — classic LB-exit weather). Same rule shape (10.20.20.10 → !RFC1918, single gateway); a different exit than invidious's GW2 so the two consumers don't share one IP's rate-limit budget or one exit's outage. The gateway was chosen empirically (a yt-dlp --simulate probe through each candidate): GW3's exit was already Google-flagged at pin time — deterministic bot-check — while GW1's extracts cleanly; a pin is an assertion about a specific exit, so test it, don't assume it. Rotations are routine weather (GW1 rotted within hours of the pin — rotated to GW4 the same evening; the JSON _comment on seq 590 carries the current choice and its history, this runbook only the procedure). Same rotation procedure when it goes bad — rotate gateway in the JSON, delete the live rule (the apply script only adds; it never modifies), re-apply, then re-probe with yt-dlp --simulate <any watch URL> (no session cache to clear; yt-dlp is stateless per fetch). Detection: every failed fetch is relayed to the operator by the Downloader bot (the human IS the first alarm), and the Grafana rule fleet-media-fetch-dispatch pages lab-alerts on >2 failed dispatches in 6 h — repeated identical failures are the "pinned exit rotted" signature, and the alert text names this section. The pin and its detect story landed in the same change, by policy: a deliberate constraint without detection silently degrades into a forgotten one.

Triage first: which of the THREE failure modes is it? (added 2026-07-30, third mode 2026-08-17)¶

These three are all instance-side ("the video will not play"). If the complaint is instead "the Kodi box keeps hanging", skip this table — that is the client-side AV1 teardown deadlock, diagnosed further down under A FOURTH mode.

The egress/PO-token story above is only one of the ways this instance dies, and they need different fixes. Hit /api/v1/search first — it is the discriminator, because search/trending/comments are innertube calls invidious makes itself; only the player/PO-token path goes through companion.

	(a) exit-IP / PO-token	(b) stale image vs a YouTube change	(c) storyboards String-vs-Hash crash
`/api/v1/search`	works (200)	broken (400/500)	works (200)
playback / `/watch`	broken	broken	per-video: some play, some hang
companion journal	`Failed to validate PO token`	`Successfully validated PO token` (clean!)	clean
error text	"Video unavailable"	`Youtube API returned status code 400`	invidious log: `Expected Hash for #[]?(key : String), not String` in `'storyboards'`; `/api/v1/videos` = 200 with a 0-byte body
both units	green	green	green
fix	rotate the pinned Proton gateway (above)	bump the image digests	the lab fork's patch (below); if it recurs, rebase the fork

Incident 2026-07-30 — mode (b). Every innertube-backed endpoint 400'd (/api/v1/videos + /watch → 500; search/trending → 400/500) while both units were green and the companion validated PO tokens cleanly every 5 min — so the fleet-invidious-po-token rule correctly stayed silent, and nothing paged. Root cause: the pinned images were from Jul 12; YouTube changed something and upstream shipped v2.20260723.0 (Jul 23) with two "fix YouTube change" hotfixes. Fix = bump both digests in ansible/roles/backends/defaults/main.yml (c8b3946→228015f9, companion 53950bb→0602f33b) + restart with a cleared companion cache. Verified after: metadata 10/10, media stream 6/6 (206, full range), search + trending 200.

Surfaced by a client, not by monitoring. FreeTube (which drives the /api/v1/* API) erroring was the first signal — fleet-invidious-po-token only matches unit="invidious-companion.service" + PO-token text, so mode (b) was invisible to it.

Both halves are covered now: - Cause, proactively — the weekly image staleness check (rb34) flags the stale pin whether or not anyone is using invidious. - Symptom, reactively — Grafana rule fleet-invidious-innertube-400 watches unit="invidious.service" for Youtube API returned status code (>3 in 30m). Validated by replaying that query over the real 2026-07-30 window: 4 (would have fired) then, empty (silent) now.

Keep both, because the rule is usage-dependent: invidious only logs an error when someone actually requests something, so a broken-but-idle instance stays quiet. That is precisely how this ran 18 days unnoticed — and it is the staleness check, not the rule, that closes it.

Incident 2026-08-17 — mode (c), and the retraction of a "benign edge case". An earlier revision of this runbook wrote off jNQXAC9IVRw ("Me at the zoo") returning HTTP 200 with an empty body as an isolated old-video quirk. It was not: it is an upstream invidious bug, latent in every published image (checked through master, 2026-08-16). Video#storyboards falls back to JSON::Any.new("{}") — which wraps the two-character String "{}", not an empty hash — so any video whose player response carries no storyboards field at all raises Expected Hash for #[]?(key : String), not String mid-JSON-stream and truncates the /api/v1/videos response to nothing; the Kodi add-on spins forever on the empty body. On 2026-08-16 YouTube stopped sending storyboards for many videos on the TV client and the "quirk" became "most videos hang".

Because no upstream fix exists, the fix is the lab's first forked app image: provision/invidious/build.sh rebuilds upstream at the exact previously-pinned revision plus the one-line provision/invidious/storyboard-empty-hash.patch, pushed to zot as apps/invidious and digest-pinned in ansible/roles/backends/defaults/main.yml like any other pin. Un-fork condition (in build.sh): when upstream fixes the storyboards fallback in src/invidious/videos.cr, drop provision/invidious/ and re-pin to quay. Verify per-video state in the DB: podman exec invidious-db psql -U kemal invidious -c "select jsonb_typeof(info::jsonb->'storyboards') as t, count(*) from videos group by 1" (the info column needs the ::jsonb cast; String-typed or absent → mode (c) trigger).

Check for a newer image without pulling (read-only, from any host):

TOKEN=$(curl -s "https://quay.io/v2/auth?service=quay.io&scope=repository:invidious/invidious:pull" \
  | sed -E 's/.*"token":"([^"]+)".*/\1/')
curl -sI -H "Authorization: Bearer $TOKEN" \
  -H "Accept: application/vnd.oci.image.index.v1+json" \
  "https://quay.io/v2/invidious/invidious/manifests/latest" | grep -i docker-content-digest

A FOURTH mode, and it is not the instance: the Kodi box deadlocks on AV1 teardown (added 2026-08-21)¶

Modes (a)-(c) are all "the video will not play." This one presents as "the Kodi box keeps hanging" — a different complaint, a different box, and the instance is entirely innocent. Do not start on invidious.${domain} when the report is the appliance freezing.

Signature. Picture frozen on the TV; a remote press still makes a sound; the box answers ping and SSH; only a power cycle appears to fix it. That combination is the tell: the Kodi process is alive and its non-GUI threads are running, while the application/render thread is deadlocked.

Root cause. The KODI_BOX is a Raspberry Pi 4, which has hardware decoders for H.264 and HEVC only — there is no AV1 decoder. The stock plugin.video.invidious add-on requests a DASH manifest, and DASH is where YouTube serves AV1, so inputstream.adaptive happily selects a 1080p AV1 representation and Kodi falls back to dav1d software decode. Playback itself merely stutters; the failure is on stop — the video thread never exits and the main thread blocks forever waiting for it, taking the whole GUI with it. Captured 2026-08-21: the box had been wedged 4h45m, with the AV1 session's dav1d-worker threads still resident.

Diagnose it in one command — the GUI-lock probe. Kodi's JSON-RPC (TCP :9090, no auth) is the discriminator, because its methods split cleanly on whether they take the graphics lock:

# on the box (or via ssh); a hung Kodi answers the first two INSTANTLY and times out on the third
python3 - <<'EOF'
import socket, json
def call(m, p=None, t=8):
    s = socket.create_connection(("127.0.0.1", 9090), timeout=t); s.settimeout(t)
    r = {"jsonrpc": "2.0", "id": 1, "method": m}
    if p: r["params"] = p
    s.sendall(json.dumps(r).encode())
    try:    return s.recv(65536).decode("utf-8", "replace")[:200]
    except Exception as e: return "TIMEOUT %r" % e
    finally: s.close()
print("lock-free :", call("JSONRPC.Ping"))                                   # pong even when hung
print("lock-free :", call("Player.GetActivePlayers"))                        # answers even when hung
print("NEEDS LOCK:", call("XBMC.GetInfoBooleans", {"booleans": ["Player.Playing"]}))
EOF

XBMC.GetInfoBooleans / GetInfoLabels / GUI.ShowNotification timing out while JSONRPC.Ping returns pong is the deadlock. Corroborate with the log — the deadlock leaves teardown truncated at exactly this line, with no thread end: video_thread after it:

CVideoPlayer::CloseFile()
waiting for video thread to exit      <-- last line ever written

A healthy stop always continues thread end: video_thread -> VideoPlayer: finished waiting.

Two red herrings, both checked and cleared. OutputPicture - timeout waiting for buffer is normal at teardown (the video thread's final wait aborts) — it appears on clean stops too, so it is not a hang indicator. And frozen vc4/v3d interrupt counters in /proc/interrupts are not sufficient either: a static Kodi GUI legitimately stops flipping, so idle looks identical to wedged. Only force a render and then observe no interrupts — or better, use the lock probe above, which needs no baseline.

Recovery is systemctl restart kodi, not a power cycle. SSH works throughout; pulling power only destroys the evidence.

Fix — remove the AV1 path (one stock add-on setting).

# Settings > Add-ons > Invidious > Configure > "Disable DASH"  == disable_dash
F=/storage/.kodi/userdata/addon_data/plugin.video.invidious/settings.xml
systemctl stop kodi && cp "$F" "$F.bak"
# handles BOTH forms: <setting id=... >false</setting> and the self-closing <setting id=... />
python3 -c 'import re,sys; f=sys.argv[1]; s=open(f).read(); \
s=re.sub(r"<setting id=\"disable_dash\"[^>]*>.*?</setting>|<setting id=\"disable_dash\"[^/]*/>", \
         "<setting id=\"disable_dash\">true</setting>", s); open(f,"w").write(s)' "$F"
systemctl start kodi

With DASH off the add-on uses YouTube's progressive formats, which are H.264/AAC only — the Pi's hardware decoder handles them and the teardown path that deadlocks is never entered. The cost is a 720p cap (that is the best progressive format YouTube publishes); on a Pi 4 that is a better picture than 1080p AV1, which software-decodes into a stutter regardless. Verified end-to-end on the exact video that caused the 2026-08-21 deadlock: Creating video codec with codec id: 27 / V4L2 mem2mem H.264 decoder wrapper, and a stop that runs to VideoPlayer: finished waiting with zero leftover decoder threads.

Make the next one diagnosable — persistent journald. LibreELEC keeps its journal in tmpfs, and Kodi keeps only kodi.log + kodi.old.log, so every power cycle destroyed the evidence — which is why this went undiagnosed for so long. LibreELEC ships the switch; it is a flag file, not a hack (var-log.mount has a ConditionPathExists on it and bind-mounts /var/log onto /storage/.cache/log):

mkdir -p /storage/.cache/log/journal /storage/.cache/journald.conf.d
printf '[Journal]\nStorage=persistent\n' > /storage/.cache/journald.conf.d/00_settings.conf
chgrp systemd-journal /var/log/journal && chmod 2755 /var/log/journal   # first time only
systemctl start storage-log.service var-log.mount && journalctl --flush

Verify it survived a reboot with journalctl --list-boots — two boots listed, and journalctl -b -1 readable, means the next hang's kernel log outlives the power cycle. Enabled on KODI_BOX 2026-08-21; forensics from that incident are kept at /storage/hang-forensics/2026-08-21_deadlock/.

The box is deliberately unmanaged (untrusted IOT — no Ansible, no IPA, no fleet SSH-CA; see rb06 §IOT), so these are host-local changes recorded here rather than codified in a role. Root login is password-only, credential in lab.kdbx ("Kodi (LibreELEC) root").

KODI_BOX streaming stack: why scrapers rot, and what replaced them (added 2026-08-22)¶

The measurement, before the opinion. The official Kodi (Omega) repo carries 177 plugin.video.* add-ons. Only 19 were updated in the last 6 months; 131 (74%) have not been touched in over 18 months. Of the 19 live ones, most are European public broadcasters and are geo-locked. Reproduce it any time:

curl -s https://mirrors.kodi.tv/addons/omega/ | tr '\n' ' ' \
  | grep -oE 'href="plugin\.video\.[^/"]+/"[^>]*>[^<]*</a></td><td>[^<]*</td><td>[0-9]{4}-[A-Za-z]{3}-[0-9]{2}'

So "the add-on stopped working" is the base rate, not a misconfiguration. Before diagnosing one, diff installed vs published versions (addons.xml.gz) — if they match, there is no fix to install and the add-on is simply abandoned. On KODI_BOX in 2026-08 all twelve installed add-ons were already at the newest published version, and seven of them shared the same 2023-02-20 repo date.

The rule this yields: add-ons that SCRAPE a website rot; add-ons that talk to a service you control, or a real API, do not. The lab's durable sources are therefore jellyfin, tvheadend (OTA), and invidious — all three are ours. Everything else is someone else's HTML.

What replaced them: FAST providers merged into the PVR. The maintained free ad-supported providers are not in the official Kodi repo; they live in the SlyGuy repo (https://k.slyguy.xyz/repository.slyguy.zip, datadir https://slyguy.uk/.repo/repository.slyguy/). Deployed on KODI_BOX: slyguy.pluto.tv.provider, slyguy.pbs.live, slyguy.plex.live, slyguy.roku, and plugin.program.iptv.merge, which merges them into one M3U/EPG that PVR IPTV Simple serves alongside tvheadend — 421 + 144 + 687 + 292 streaming + 39 OTA = 1583 channels in one guide. Verified playing: H.264 on the V4L2 hardware decoder, so these streams cannot trigger the AV1 teardown deadlock above.

Headless install — the order matters, and four steps fail SILENTLY. All of this is doable over SSH via JSON-RPC (:9090); none of it needs the TV.

Extract each add-on zip into /storage/.kodi/addons/ with Kodi stopped. Side-loaded add-ons register disabled — enable them in dependency order via Addons.SetAddonEnabled, and set addons.unknownsources=true (Settings.SetSettingValue).
plugin.program.iptv.merge needs pvr.iptvsimple, a binary add-on — it must come from the LibreELEC repo (https://addons.libreelec.tv/12.2.0/ARMv8/aarch64), not mirrors.kodi.tv, because it is compiled against the LibreELEC build. It in turn wants inputstream.ffmpegdirect and inputstream.rtmp from the same place.
pvr.iptvsimple with an empty M3U path ABORTS Kodi — an uncaught C++ exception in CPVRClient::Destroy() (SIGABRT, stack inside pvr.iptvsimple.so). Either configure it or remove the stray instance-settings-N.xml; leaving an unconfigured instance crashes Kodi on every reload.
Run IPTV Merge's own wiring, Addons.ExecuteAddon → {'_': 'setup'}. It creates instance-settings-N.xml named IPTV Merge pointing at http://127.0.0.1:8097/playlist.m3u8 — and leaves pvr.iptvsimple DISABLED. Re-enable it, or you get a perfectly configured stack serving zero channels with no error anywhere.
Register each provider with {'_': 'setup_addon', 'addon_id': <id>} ONE AT A TIME. Each opens a modal Select dialog that blocks subsequent Addons.ExecuteAddon calls — a batch of four returns OK four times and registers only the first. Dismiss with Input.Back between calls.
Region choice is not in settings.xml — it is sqlite at addon_data/script.module.slyguy/settings.db, table settings(addon_id, key, value), key userdata_merge_regions, value JSON. An empty region list merges zero channels, silently. Set it before registering:

python3 - <<'EOF'
import sqlite3, json
c = sqlite3.connect('/storage/.kodi/userdata/addon_data/script.module.slyguy/settings.db')
for addon, regions in [('slyguy.pluto.tv.provider', ['us']),
                       ('slyguy.plex.live', ['us']),
                       ('slyguy.roku', ['all'])]:
    c.execute('insert or replace into settings (addon_id,key,value) values (?,?,?)',
              (addon, 'userdata_merge_regions', json.dumps(regions)))
c.commit()
EOF

slyguy.roku (v0.0.2) errors out of its configure_merge if the dialog is cancelled and may need a second run_merge pass before it populates. Confirm every provider actually contributed — per-source counts live in addon_data/plugin.program.iptv.merge/data.db, table playlist, column results; a source showing [] merged nothing.

Do not sweep add-ons with Files.GetDirectory in a loop. Kodi serialises plugin directory execution, so one hung add-on (here: plugin.video.cbc, 40 s) queues every call behind it and the whole run reports TIMEOUT — including add-ons known to work. The results look like a catastrophic failure and mean nothing. Test one add-on at a time, or restart Kodi between them.

MeTV is free over the air. Before installing any add-on for a broadcast network, check the tuner: the lineup already carries RetroTV, Movies!, Bounce, Laff, Buzzr, CHARGE, TBD, Quest, Confess and DABL. A rescan (rb21 §ATSC scan) beats a scraper that will rot again.

Build (PVE + control node)¶

# 1. provision the three media VMs (PVE) — clones + cloud-init from vms.conf
cd /root/lab-provision
./lab-provision.sh up jellyfin     # 9123, SERVERS .30  (EL9)
./lab-provision.sh up tvheadend    # 9124, SERVERS .31  (Debian, template 9000)
./lab-provision.sh up invidious    # 9125, DMZ .70      (EL9)

# 1a. give tvheadend the tuner (lab-provision.sh can't express USB passthrough)
qm set 9124 --usb0 host=2040:826d
#   then confirm the host exposes it as a DVB adapter (driver + firmware):
#   the Hauppauge needs linux-firmware; after a tuner-VM reboot, `ls /dev/dvb` inside
#   the guest must list adapter0. If empty: install linux-firmware on the VM and check
#   `dmesg | grep -i dvb`.

# 2. Invidious secrets -> vault (first build only), then configure the fleet
cd ansible   # from the repo root (~/src/myos)
ANSIBLE_VAULT_PASSWORD_FILE=~/.config/lab-vault-pass ansible-vault edit group_vars/all/vault.yml
#   add:  vault_invidious_hmac_key: "<openssl rand -hex 20>"
#         vault_invidious_db_password: "<openssl rand -hex 24>"

# 3. configure — common baseline + the app quadlets (+ NFS media mounts + Alloy)
ANSIBLE_VAULT_PASSWORD_FILE=~/.config/lab-vault-pass \
  ansible-playbook -i inventory.ini playbooks/site.yml -l jellyfin,tvheadend,invidious

# 4. publish the jellyfin/ha vhosts on the apps-proxy
ansible-playbook -i inventory.ini playbooks/site.yml -l apps-proxy

Jellyfin LDAP auth¶

Install the LDAP-Auth plugin in Jellyfin (Dashboard → Plugins) and point it at FreeIPA:

LDAP server 10.20.20.2:636 (LDAPS), base DN cn=users,cn=accounts,${basedn}
bind with a least-privilege IPA service/bind account; user filter on uid
map the Jellyfin admin to your IPA admin group.

This gives Jellyfin FreeIPA-backed accounts (same identity as the rest of the fleet) without needing a browser SSO redirect the native apps can't do.

Initial state¶

The quadlets are config-only — services start empty:

Jellyfin: add libraries pointing at /media (the NFS mount). Watch state/users live in /var/lib/jellyfin.
tvheadend: build the channel lineup — see "ATSC scan + channel map" below.
Invidious: accounts/subscriptions are created in the app; data persists in the invidious-db postgres volume — since 2026-08-14 a postgres 18 cluster at /var/lib/invidious/db/18/docker (the 18+ datadir convention; the pre-migration v14 cluster is kept aside as db-pg14 until two bizon replication cycles pass). Major upgrades = dump/restore per rb16 §"Postgres major upgrades". Since 2026-08-14 that DB also gets the nightly logical dump tier (db_backup_kind=invidious in the [db_servers] inventory group → pg_dump to /var/backups/db, rb16 §2) — until then accounts/subscriptions had only the crash-consistent guest replication under them.

ATSC scan + channel map (tvheadend)¶

Building the lineup. The Hauppauge is an ATSC (US over-the-air) tuner — its frontend shows as LG Electronics LGDT3306A VSB/QAM. Confirm /dev/dvb/adapter0 is in the container (podman exec tvheadend ls /dev/dvb); if the VM has it but the container doesn't, the quadlet needs AddDevice=/dev/dvb + a container restart. tvheadend UI = http://10.20.20.31:9981.

Access control (per the auth policy: local admin + no anonymous-admin). tvheadend ships with a single anonymous * entry that has full admin. tvheadend has no LDAP, so it's local-only — give it a named local admin and de-privilege anonymous: - Configuration → Users → Passwords → add tvadmin + a password (record in KeePass; also vault_tvheadend_admin_password). Auth is digest. - Configuration → Users → Access Entries → add tvadmin: Admin + Web interface + Streaming + DVR. Then edit the * (anonymous) entry: uncheck Admin (keep Streaming — incl. HTSP — and Web interface) so Jellyfin/Kodi HTSP + channel icons keep working but configuration now requires the tvadmin login.

GUI path (Configuration → DVB Inputs): 1. Networks → Add → ATSC-T Network → name ATSC OTA, Pre-defined muxes = United States: us-ATSC-center-frequencies-8VSB-072020 → Save. (Pre-loads ~35 muxes.) 2. Adapters → expand the LGDT3306A → tick the ATSC-T frontend → set its Networks = ATSC OTA, Enabled on → Save. The tuner starts scanning the muxes. 3. Watch Muxes: locked muxes go scan_result = OK; live signal under the adapter should be SNR ≳ 20 dB, BER/UNC 0 (a good antenna here read 24 dB / −43 dBm). 4. Services tab → "Map all services to channels" with merge same name = on, and map-encrypted / fuzzy / tidy-name = off (US OTA is free; fuzzy over-merges subchannels). Channels appear with their ATSC virtual numbers (e.g. 25.1, 38.3).

Gotchas (cost real time — for scripting via the JSON API): - The service-mapper endpoint is POST /api/service/mapper/save (not mapall), and its services field is a JSON array of service UUIDs — passing a comma-string returns HTTP 400 / maps 0. Options: merge_same_name, encrypted, merge_same_name_fuzzy, tidy_channel_name, check_availability, *_tags. - A mux can show num_svc > 0 (PAT read) before its service objects exist; force a per-mux scan (set scan_state = 1) so the PMTs are read and services materialize, then map. Services aren't persisted to disk (rediscovered on startup scan, same UUIDs) but channels are — so the lineup survives a restart. - Query the service grid with a form-POST (--data-urlencode "limit=400"), not a GET ?limit=1 query string — the latter can report total: 0 even when services exist.

Fetching media into the library (`media_fetch`)¶

Pinned yt-dlp + a media-fetch wrapper on the fileserver, writing straight into the Jellyfin library datasets. Role: ansible/roles/media_fetch (applied by the Storage play).

Why the fileserver. The media datasets are local there, so there is no NFS write path (the media export is sec=sys + root_squash) and no zone crossing. Jellyfin's own media mount is deliberately read-only, so Jellyfin can never be the downloader.

# run as almalinux on the fileserver (media tree is almalinux:almalinux 0775)
media-fetch <url> [url...]             # audio -> music library  (default)
media-fetch -t video <url>             # video -> movies library
media-fetch -t video -d tvshows <url>  # video -> a named library

Produces music/<uploader>/<playlist|Singles>/NN - <title>.<ext> and movies/<title> [<id>].<ext>, mirroring the conventions already in the library so Jellyfin's scanners read artist/album correctly.

Visibility: the wrapper triggers the Jellyfin scan itself (added 2026-08-15). Jellyfin mounts the media tree read-only over NFS, and NFS delivers no file-change events to the client — so Jellyfin's "real-time monitoring" can never see a server-side write, and a finished fetch stayed invisible until the daily scheduled scan ("the phone said downloaded, the library shows nothing"). The wrapper now POSTs /Library/Refresh after every successful run with a scoped API key minted in Jellyfin as app media-fetch (Dashboard → API Keys; escrowed as vault_jellyfin_api_key in lab-secrets jellyfin.vault.yml + KeePass entry "Jellyfin API key (media-fetch)" — the first Jellyfin automation credential; jfadmin remains human/break-glass-only per the custody model). Best-effort by design: a refresh failure warns and never fails the fetch — the file is on disk and the daily scan is the fallback, so the dispatch bot still reports the download itself truthfully.

This replaced a hand-rolled /supernas/media/download/fetch.sh, which had two failures worth remembering — both of the "still runs, still exits 0, no longer does what it claims" class:

Its output never reached Jellyfin. It wrote to download/, which is not a Jellyfin library (the libraries are movies, music, photos, tvmovies, tv, tvshows). The files were fine; the destination was invisible. The wrapper now refuses to write to a directory that does not already exist, so a typo'd -d exits 3 instead of silently repeating this.
Its extractor pin had rotted. It passed --extractor-args youtube:player_client=android. Measured 2026-08-15: with that flag YouTube warns "Some android client https formats have been skipped as they are missing a URL" under its SABR rollout — i.e. a silently reduced format set. Without it the default client resolves cleanly. The wrapper passes no --extractor-args; if a future breakage needs one, add it with a dated comment saying why.

A JS runtime is required, and cookies are NOT the fix. Modern yt-dlp needs a JavaScript runtime to run YouTube's player logic and derive stream URLs. Without one it falls back to a degraded path that misses formats and trips bot checks — measured 2026-08-15, a fetch died with "Sign in to confirm you're not a bot" while a plain title probe seconds later succeeded, and the log carried No supported JavaScript runtime could be found. The role therefore pins deno (media_fetch_deno_version, staleness-covered). After installing it the same video extracted cleanly with no warning and no bot check.

The tempting fix — a cookies.txt export — is the wrong one: it expires, cannot be automated, ties a Google account to the host (accounts used this way get flagged), and is unmaintainable. Cookies remain supported as a fallback for genuinely age-gated items, never as the answer to "extraction broke".

TRAP — --cookies FILE REWRITES FILE. yt-dlp saves the updated jar back over the file it was given. Pointed at the canonical cookies.txt, it truncated a 1153-byte hand-made export to 354 bytes on its first real run (2026-08-15; recovered from the hourly ZFS snapshot, which is why that snapshot exists). A cookie jar is a browser export that cannot be regenerated on the server, so the wrapper now copies it to a throwaway temp file per run and lets yt-dlp rewrite that. Related: the script must not exec yt-dlp, or the cleanup trap never fires and the copy leaks into /tmp every run.

Staleness. media_fetch_version is declared in scripts/image-staleness-check.py BINARY_SOURCES, so the pin is covered by the weekly check (runbook 34) from its first converge. This pin matters more than most — it tracks a deliberately hostile, fast-moving target, and when it goes stale downloads degrade rather than error.

Audio format. Default best = keep the source codec, no re-encode: YouTube audio is already lossy and Jellyfin plays m4a/opus natively. A deliberate choice, not a limitation — the mirror's ffmpeg-free 5.1.9 does carry libmp3lame, so media_fetch_audio_format: mp3 works if a device needs it (the predecessor transcoded to mp3, hence the existing library is mp3). Either way the role asserts the encoder exists at converge, rather than letting a transcode fail per-file with the unit green.

Dependencies. ffmpeg-free from our own pinned mirror. RPM Fusion is deliberately not added — an unpinned third-party repo on the storage host is the supply-chain exposure the mirror exists to avoid. Re-running a URL is a no-op (archive ledger at <media_root>/download/.fetch-archive.txt), which is what makes the wrapper safe to drive from an automated dispatcher.

Home Assistant (HAOS appliance VM)¶

HAOS runs as an appliance VM (105) on the SERVERS VLAN with the Zigbee USB coordinator passed through. Provision/import the HAOS image as VM 105, attach its NIC to the SERVERS VLAN, and pass through the Zigbee stick:

# On PVE — HAOS NIC on the SERVERS VLAN (tag 20), virtio + firewall on.
qm set 105 --net0 virtio,bridge=vmbr1,firewall=1,tag=20
# Pass the USB Zigbee coordinator (10c4:ea60) through to the VM.
qm set 105 --usb0 host=10c4:ea60
# Group it with the lab.
qm set 105 --tags lab

Then in Home Assistant (Settings → System → Network, or the console network update): set a static address on the SERVERS VLAN —

IP: 10.20.20.45/24 · Gateway: 10.20.20.1 · DNS: 10.20.20.1 (Unbound)

The ha.${domain} vhost on the apps-proxy points there. Add an internal DNS record (Unbound host override ha.${domain} → 10.20.20.45) if you reach it by name on the LAN.

Why SERVERS, not IOT. HA reaches everything, so it belongs in a trusted zone, not the locked IoT zone. Its Zigbee/Z-Wave devices are radio (USB coordinator), so no VLAN adjacency is needed for them; the IP IoT devices are reached via the explicit HOMEASSISTANT → IOT allow (rb06 §SERVERS 1a) + mDNS reflection.

Firewall (added in runbook 06 — apply/verify there)¶

SERVERS §1a: HOMEASSISTANT (10.20.20.45) → IOT net (default gw) — HA controls IP IoT. Above the SERVERS→RFC1918 block.
WORKSTATIONS §2a: clients → HOMEASSISTANT:8123 + JELLYFIN:8096 (default gw) for fast local access.
IOT §2 (already present): IOT → JELLYFIN:8096 (Roku). No change.
mDNS (runbook 08): extend the reflector to SERVERS⇄IOT so HA discovers IP IoT (and so casting to the Roku keeps working from HA).
WG (runbook 07, optional): to use the HA companion app while away, add the user peers → HOMEASSISTANT:8123 to their per-peer rules (they already reach Jellyfin).

New host aliases: HOMEASSISTANT = 10.20.20.45 (add in Firewall → Aliases; JELLYFIN already exists).

AI-assisted configuration (ha-mcp, optional)¶

Claude Code can author HA config (automations, scripts, dashboards, entity registry) through the community homeassistant-ai/ha-mcp MCP server (REST + WebSocket API). The official HA mcp_server integration is not used here — it only exposes the Assist API (control/query of exposed entities), which can't edit config.

Runs on the operator's laptop, not the fleet: uvx ha-mcp@<pinned-version> (pin, not @latest — it's driven by a crown-jewel token), registered in that laptop's Claude Code at user scope (claude mcp add ha-mcp -s user -e HOMEASSISTANT_URL=https://ha.${domain} -e HOMEASSISTANT_TOKEN=<KeePass> -- uvx ha-mcp@<ver>). Never at project scope — the token must not enter the tracked tree.
No firewall change. ha-mcp targets https://ha.${domain}, which over the laptop WG tier rides the existing sanctioned path: WG_LAPTOPS → SERVERS_NET:IPA_PORTS includes 443 (rb07 §5.3, rb03 IPA_PORTS), Unbound (10.20.80.1) resolves the split-horizon name to the apps-proxy, and apps-proxy reverse-proxies HA :8123 incl. its WebSocket (rb06 §112,144). Verified end-to-end: REST 200, WS 101, valid *.${domain} cert. HA stays Tier-2 private — no new hole, no public exposure.
Credential. A dedicated HA admin user claude-mcp (config writes need admin — HA has no finer RBAC) with a long-lived access token in KeePass lab.kdbx (T3 custody). This is a known HA credential, revocable independently of the fleet SSH-CA/PAW model (delete the token / disable the user). Distinct access path from T1 automation and the T2 agent PAW on ctrl — an app-level grant to a crown jewel, so treat it deliberately: start ha-mcp read-only, take an HA backup before the first write session, rely on its edit-backups.
The beta YAML/file tools need HA's ha_mcp_tools custom component; the registry/automation APIs cover most editing without it.

Validation¶

[ ] ls /dev/dvb/adapter0 inside the tvheadend VM lists the tuner; tvheadend UI (http://10.20.20.31:9981) shows the adapter + scans muxes.
[ ] Jellyfin UI (http://10.20.20.30:8096) loads; libraries resolve under /media (NFS mount up); Live TV added via the tvheadend HTSP/M3U backend plays.
[ ] Jellyfin login with a FreeIPA account works (LDAP plugin bound to 10.20.20.2).
[ ] From a Roku (IOT): Jellyfin connects on 8096 (rb06 IOT rule).
[ ] http://10.20.30.70:3000 (Invidious) loads → search + video plays (proves the companion/sig path).
[ ] Home Assistant reachable at https://ha.${domain} and http://10.20.20.45:8123 from a workstation; Zigbee devices paired; HA can control an IP IoT device (proves SERVERS→IOT allow + mDNS). > HA http: YAML is DEAD since Core 2026.8 — the UI owns reverse-proxy trust. > The http: block (use_x_forwarded_for, trusted_proxies = the apps-proxy) was > auto-migrated into the UI store (Settings ▸ System ▸ Network) and the YAML is now > ignored (Core logs a warning saying so). Verified intact post-migration > 2026-08-16: probes via the apps-proxy were attributed to the true client IP, not > 10.20.20.6. The trap: if the apps-proxy IP ever changes, editing > configuration.yaml is a silent no-op — change it in the UI. Removing the stale > http: block (File editor add-on) just silences the warning.
[ ] All four egress via Proton (check exit IP); invidious still plays — if Google blocks it, apply the VPN_BYPASS Google-ranges valve.
[ ] (if using ha-mcp) from the laptop on WG: claude mcp list shows ha-mcp connected; Claude can list automations (read) and create + delete a trivial test automation (write) — proving the claude-mcp admin token + apps-proxy WebSocket path.