Skip to content

21 — Media apps (Jellyfin / tvheadend / Invidious) + Home Assistant

Stands up the three media services as AlmaLinux 9 + podman quadlets (the locked clean-build path, same machinery as Forgejo/SearXNG/FMD), plus the Home Assistant OS appliance VM on the SERVERS VLAN (HAOS is an appliance, like OPNsense — not a quadlet-managed EL box).

The media stack

Host What it is Home How
jellyfin Jellyfin media server EL9 quadlet (9123, SERVERS .30) clean EL9 quadlet; library via NFS from the fileserver
tvheadend DVR/tuner backend, USB tuner 2040:826d (Hauppauge) Debian VM (9124, SERVERS .31) EL9 lacks the lgdt3306a demod, so Debian + LinuxServer quadlet; claims the USB tuner + serves Jellyfin Live TV
invidious privacy YouTube frontend EL9 quadlets (9125, DMZ .70) app + pg + companion quadlets
haos Home Assistant OS + Zigbee USB 10c4:ea60 HAOS appliance VM (105, SERVERS .45) appliance VM on the SERVERS VLAN; Zigbee via USB coordinator

Tuner split is the point. The DVR backend (tvheadend) is separated from the player (Jellyfin): tvheadend owns the USB tuner and feeds Jellyfin's Live TV over HTSP.

Architecture

  • Jellyfin — SERVERS 10.20.20.30:8096. Config/metadata in /var/lib/jellyfin; the movie/TV library is an NFS automount from the fileserver (nfs_mounts in host_vars/jellyfin.yml), not local disk. CPU transcode (pass /dev/dri later for VAAPI/QSV). Reached by Roku/IOT (8096, rb06), WG users (rb07), and jellyfin.${domain} via the apps-proxy (Tier 2, internal split-horizon).

    Auth = FreeIPA LDAP, not Authelia SSO. Jellyfin's clients are native apps (Roku/mobile) that can't do a forward-auth/OIDC browser redirect, so Jellyfin uses its LDAP plugin bound to FreeIPA for accounts — same UID/identity, no SSO.

  • tvheadend — SERVERS 10.20.20.31. Debian VM (template 9000) — RHEL/EL9 stripped the DVB stack and ELRepo's kmod-media lacks the lgdt3306a demod this Hauppauge WinTV-dualHD needs; Debian ships it in-kernel so /dev/dvb (dual adapters) works OOTB. Runs the LinuxServer tvheadend quadlet with /dev/dvb passed in (the backends role is OS-agnostic for podman/NFS; common/alloy are EL-only and skipped here). Web/HTSP 9981, HTSP streaming 9982. Recordings → fileserver NFS. Jellyfin's Live TV points at 10.20.20.31:9981 (intra-SERVERS, no fw rule).
  • Invidious — DMZ 10.20.30.70:3000, quadlets on a private podman net (invidious + invidious-db postgres + invidious-companion — the YouTube innertube/PoToken/signature backend). Tier-2 private, served via the apps-proxy at https://invidious.${domain} (like Jellyfin) — a split-horizon Unbound override → apps-proxy reverse-proxies to this backend :3000 (needs the SERVERS→DMZ allow, rb06). Not behind the origin Caddy / Authelia and not delegated through an edge; also reachable directly at http://10.20.30.70:3000 on the LAN. FreeTube can use it as its Invidious backend.
  • Home Assistant — SERVERS 10.20.20.45:8123 (appliance VM, not EL9/quadlet). Zigbee/Z-Wave devices ride the USB coordinator (radio, not network), so HA doesn't need L2-adjacency to the IoT VLAN; the IP IoT devices are reached via an explicit HOMEASSISTANT → IOT allow + mDNS reflection. Reached by clients at :8123 (rb06 §WS), ha.${domain} via the apps-proxy (Tier 2), and the HA companion app over WG when away.

Egress note — Invidious → YouTube (single Proton exit, NOT the LB group)

Invidious hammers YouTube/Google, which frequently blocks ProtonVPN exit IPs — and, sharper than that: YouTube PO tokens are bound to the session's exit IP, so the companion's session breaks whenever the load-balanced Proton group re-homes the host onto a different tunnel (sticky expiry), even to a non-blocked exit. Observed 2026-07-10→12: days of flapping Failed to validate PO token in the companion journal, then hard-down when stickiness parked the host on a Google-rejected exit → every watch page "Video unavailable" while search still worked (browse API isn't PO-token-gated).

Fix (2026-07-12): the invidious host's internet egress is pinned to a single Proton tunnel via an automation-filter rule — INVIDIOUS → !RFC1918, gateway PROTON_GW2 (provision/opnsense/fw-automation-rules.json, applied by scripts/opnsense-fw.py). Still all-via-Proton, still fail-closed; the rule only removes the per-connection exit hopping for this one host. If that exit's reputation goes bad (token failures return): rotate gateway to another PROTON_GW* in the JSON, apply, then restart companion with a cleared session cache: systemctl stop invidious invidious-companion && rm -rf /var/lib/invidious/companion-cache/* && systemctl start invidious-companion invidious. Also try bumping the image digests first (see the caveat in ansible/roles/backends/defaults/main.yml — upstream ships cat-and-mouse fixes often).

Detection is codified (2026-07-12): the Grafana rule fleet-invidious-po-token (runbook 27) watches the companion's journal in Loki for PO-token failure lines and pages ntfy lab-alerts within ~10–30 min of a recurrence — the 2026-07-10→12 incident ran ~2 days undetected because the companion's unit stays green while playback dies. An ntfy page from that rule is the trigger for the rotation procedure above.

Last resort only (operator decision — exposes the home IP to Google): add Google's ranges to the VPN_BYPASS alias (runbook 05 §6 / runbook 06 §IOT) so only invidious→Google exits the real WAN. Don't pre-open it.

Triage first: which of the TWO failure modes is it? (added 2026-07-30)

The egress/PO-token story above is only one of two ways this instance dies, and they need opposite fixes. Hit /api/v1/search first — it is the discriminator, because search/trending/comments are innertube calls invidious makes itself; only the player/PO-token path goes through companion.

(a) exit-IP / PO-token (b) stale image vs a YouTube change
/api/v1/search works (200) broken (400/500)
playback / /watch broken broken
companion journal Failed to validate PO token Successfully validated PO token (clean!)
error text "Video unavailable" Youtube API returned status code 400
both units green green
fix rotate the pinned Proton gateway (above) bump the image digests

Incident 2026-07-30 — mode (b). Every innertube-backed endpoint 400'd (/api/v1/videos + /watch → 500; search/trending → 400/500) while both units were green and the companion validated PO tokens cleanly every 5 min — so the fleet-invidious-po-token rule correctly stayed silent, and nothing paged. Root cause: the pinned images were from Jul 12; YouTube changed something and upstream shipped v2.20260723.0 (Jul 23) with two "fix YouTube change" hotfixes. Fix = bump both digests in ansible/roles/backends/defaults/main.yml (c8b3946→228015f9, companion 53950bb→0602f33b) + restart with a cleared companion cache. Verified after: metadata 10/10, media stream 6/6 (206, full range), search + trending 200.

Surfaced by a client, not by monitoring. FreeTube (which drives the /api/v1/* API) erroring was the first signal — fleet-invidious-po-token only matches unit="invidious-companion.service" + PO-token text, so mode (b) was invisible to it.

Both halves are covered now: - Cause, proactively — the weekly image staleness check (rb34) flags the stale pin whether or not anyone is using invidious. - Symptom, reactively — Grafana rule fleet-invidious-innertube-400 watches unit="invidious.service" for Youtube API returned status code (>3 in 30m). Validated by replaying that query over the real 2026-07-30 window: 4 (would have fired) then, empty (silent) now.

Keep both, because the rule is usage-dependent: invidious only logs an error when someone actually requests something, so a broken-but-idle instance stays quiet. That is precisely how this ran 18 days unnoticed — and it is the staleness check, not the rule, that closes it.

Known benign edge case: a few very old videos (e.g. jNQXAC9IVRw, "Me at the zoo", 2005) return HTTP 200 with an empty body and then cache that empty result (~0.7 ms replays). Isolated per-video quirk, not an instance-wide fault — don't chase it as one.

Check for a newer image without pulling (read-only, from any host):

TOKEN=$(curl -s "https://quay.io/v2/auth?service=quay.io&scope=repository:invidious/invidious:pull" \
  | sed -E 's/.*"token":"([^"]+)".*/\1/')
curl -sI -H "Authorization: Bearer $TOKEN" \
  -H "Accept: application/vnd.oci.image.index.v1+json" \
  "https://quay.io/v2/invidious/invidious/manifests/latest" | grep -i docker-content-digest

Build (PVE + control node)

# 1. provision the three media VMs (PVE) — clones + cloud-init from vms.conf
cd /root/lab-provision
./lab-provision.sh up jellyfin     # 9123, SERVERS .30  (EL9)
./lab-provision.sh up tvheadend    # 9124, SERVERS .31  (Debian, template 9000)
./lab-provision.sh up invidious    # 9125, DMZ .70      (EL9)

# 1a. give tvheadend the tuner (lab-provision.sh can't express USB passthrough)
qm set 9124 --usb0 host=2040:826d
#   then confirm the host exposes it as a DVB adapter (driver + firmware):
#   the Hauppauge needs linux-firmware; after a tuner-VM reboot, `ls /dev/dvb` inside
#   the guest must list adapter0. If empty: install linux-firmware on the VM and check
#   `dmesg | grep -i dvb`.

# 2. Invidious secrets -> vault (first build only), then configure the fleet
cd ansible   # from the repo root (~/src/myos)
ANSIBLE_VAULT_PASSWORD_FILE=~/.config/lab-vault-pass ansible-vault edit group_vars/all/vault.yml
#   add:  vault_invidious_hmac_key: "<openssl rand -hex 20>"
#         vault_invidious_db_password: "<openssl rand -hex 24>"

# 3. configure — common baseline + the app quadlets (+ NFS media mounts + Alloy)
ANSIBLE_VAULT_PASSWORD_FILE=~/.config/lab-vault-pass \
  ansible-playbook -i inventory.ini playbooks/site.yml -l jellyfin,tvheadend,invidious

# 4. publish the jellyfin/ha vhosts on the apps-proxy
ansible-playbook -i inventory.ini playbooks/site.yml -l apps-proxy

Jellyfin LDAP auth

Install the LDAP-Auth plugin in Jellyfin (Dashboard → Plugins) and point it at FreeIPA:

  • LDAP server 10.20.20.2:636 (LDAPS), base DN cn=users,cn=accounts,${basedn}
  • bind with a least-privilege IPA service/bind account; user filter on uid
  • map the Jellyfin admin to your IPA admin group.

This gives Jellyfin FreeIPA-backed accounts (same identity as the rest of the fleet) without needing a browser SSO redirect the native apps can't do.

Initial state

The quadlets are config-only — services start empty:

  • Jellyfin: add libraries pointing at /media (the NFS mount). Watch state/users live in /var/lib/jellyfin.
  • tvheadend: build the channel lineup — see "ATSC scan + channel map" below.
  • Invidious: accounts/subscriptions are created in the app; data persists in the invidious-db postgres volume.

ATSC scan + channel map (tvheadend)

Building the lineup. The Hauppauge is an ATSC (US over-the-air) tuner — its frontend shows as LG Electronics LGDT3306A VSB/QAM. Confirm /dev/dvb/adapter0 is in the container (podman exec tvheadend ls /dev/dvb); if the VM has it but the container doesn't, the quadlet needs AddDevice=/dev/dvb + a container restart. tvheadend UI = http://10.20.20.31:9981.

Access control (per the auth policy: local admin + no anonymous-admin). tvheadend ships with a single anonymous * entry that has full admin. tvheadend has no LDAP, so it's local-only — give it a named local admin and de-privilege anonymous: - Configuration → Users → Passwords → add tvadmin + a password (record in KeePass; also vault_tvheadend_admin_password). Auth is digest. - Configuration → Users → Access Entries → add tvadmin: Admin + Web interface + Streaming + DVR. Then edit the * (anonymous) entry: uncheck Admin (keep Streaming — incl. HTSP — and Web interface) so Jellyfin/Kodi HTSP + channel icons keep working but configuration now requires the tvadmin login.

GUI path (Configuration → DVB Inputs): 1. Networks → Add → ATSC-T Network → name ATSC OTA, Pre-defined muxes = United States: us-ATSC-center-frequencies-8VSB-072020 → Save. (Pre-loads ~35 muxes.) 2. Adapters → expand the LGDT3306A → tick the ATSC-T frontend → set its Networks = ATSC OTA, Enabled on → Save. The tuner starts scanning the muxes. 3. Watch Muxes: locked muxes go scan_result = OK; live signal under the adapter should be SNR ≳ 20 dB, BER/UNC 0 (a good antenna here read 24 dB / −43 dBm). 4. Services tab → "Map all services to channels" with merge same name = on, and map-encrypted / fuzzy / tidy-name = off (US OTA is free; fuzzy over-merges subchannels). Channels appear with their ATSC virtual numbers (e.g. 25.1, 38.3).

Gotchas (cost real time — for scripting via the JSON API): - The service-mapper endpoint is POST /api/service/mapper/save (not mapall), and its services field is a JSON array of service UUIDs — passing a comma-string returns HTTP 400 / maps 0. Options: merge_same_name, encrypted, merge_same_name_fuzzy, tidy_channel_name, check_availability, *_tags. - A mux can show num_svc > 0 (PAT read) before its service objects exist; force a per-mux scan (set scan_state = 1) so the PMTs are read and services materialize, then map. Services aren't persisted to disk (rediscovered on startup scan, same UUIDs) but channels are — so the lineup survives a restart. - Query the service grid with a form-POST (--data-urlencode "limit=400"), not a GET ?limit=1 query string — the latter can report total: 0 even when services exist.

Home Assistant (HAOS appliance VM)

HAOS runs as an appliance VM (105) on the SERVERS VLAN with the Zigbee USB coordinator passed through. Provision/import the HAOS image as VM 105, attach its NIC to the SERVERS VLAN, and pass through the Zigbee stick:

# On PVE — HAOS NIC on the SERVERS VLAN (tag 20), virtio + firewall on.
qm set 105 --net0 virtio,bridge=vmbr1,firewall=1,tag=20
# Pass the USB Zigbee coordinator (10c4:ea60) through to the VM.
qm set 105 --usb0 host=10c4:ea60
# Group it with the lab.
qm set 105 --tags lab

Then in Home Assistant (Settings → System → Network, or the console network update): set a static address on the SERVERS VLAN —

  • IP: 10.20.20.45/24 · Gateway: 10.20.20.1 · DNS: 10.20.20.1 (Unbound)

The ha.${domain} vhost on the apps-proxy points there. Add an internal DNS record (Unbound host override ha.${domain} → 10.20.20.45) if you reach it by name on the LAN.

Why SERVERS, not IOT. HA reaches everything, so it belongs in a trusted zone, not the locked IoT zone. Its Zigbee/Z-Wave devices are radio (USB coordinator), so no VLAN adjacency is needed for them; the IP IoT devices are reached via the explicit HOMEASSISTANT → IOT allow (rb06 §SERVERS 1a) + mDNS reflection.

Firewall (added in runbook 06 — apply/verify there)

  • SERVERS §1a: HOMEASSISTANT (10.20.20.45) → IOT net (default gw) — HA controls IP IoT. Above the SERVERS→RFC1918 block.
  • WORKSTATIONS §2a: clients → HOMEASSISTANT:8123 + JELLYFIN:8096 (default gw) for fast local access.
  • IOT §2 (already present): IOT → JELLYFIN:8096 (Roku). No change.
  • mDNS (runbook 08): extend the reflector to SERVERS⇄IOT so HA discovers IP IoT (and so casting to the Roku keeps working from HA).
  • WG (runbook 07, optional): to use the HA companion app while away, add the user peers → HOMEASSISTANT:8123 to their per-peer rules (they already reach Jellyfin).

New host aliases: HOMEASSISTANT = 10.20.20.45 (add in Firewall → Aliases; JELLYFIN already exists).

AI-assisted configuration (ha-mcp, optional)

Claude Code can author HA config (automations, scripts, dashboards, entity registry) through the community homeassistant-ai/ha-mcp MCP server (REST + WebSocket API). The official HA mcp_server integration is not used here — it only exposes the Assist API (control/query of exposed entities), which can't edit config.

  • Runs on the operator's laptop, not the fleet: uvx ha-mcp@<pinned-version> (pin, not @latest — it's driven by a crown-jewel token), registered in that laptop's Claude Code at user scope (claude mcp add ha-mcp -s user -e HOMEASSISTANT_URL=https://ha.${domain} -e HOMEASSISTANT_TOKEN=<KeePass> -- uvx ha-mcp@<ver>). Never at project scope — the token must not enter the tracked tree.
  • No firewall change. ha-mcp targets https://ha.${domain}, which over the laptop WG tier rides the existing sanctioned path: WG_LAPTOPS → SERVERS_NET:IPA_PORTS includes 443 (rb07 §5.3, rb03 IPA_PORTS), Unbound (10.20.80.1) resolves the split-horizon name to the apps-proxy, and apps-proxy reverse-proxies HA :8123 incl. its WebSocket (rb06 §112,144). Verified end-to-end: REST 200, WS 101, valid *.${domain} cert. HA stays Tier-2 private — no new hole, no public exposure.
  • Credential. A dedicated HA admin user claude-mcp (config writes need admin — HA has no finer RBAC) with a long-lived access token in KeePass lab.kdbx (T3 custody). This is a known HA credential, revocable independently of the fleet SSH-CA/PAW model (delete the token / disable the user). Distinct access path from T1 automation and the T2 agent PAW on ctrl — an app-level grant to a crown jewel, so treat it deliberately: start ha-mcp read-only, take an HA backup before the first write session, rely on its edit-backups.
  • The beta YAML/file tools need HA's ha_mcp_tools custom component; the registry/automation APIs cover most editing without it.

Validation

  • [ ] ls /dev/dvb/adapter0 inside the tvheadend VM lists the tuner; tvheadend UI (http://10.20.20.31:9981) shows the adapter + scans muxes.
  • [ ] Jellyfin UI (http://10.20.20.30:8096) loads; libraries resolve under /media (NFS mount up); Live TV added via the tvheadend HTSP/M3U backend plays.
  • [ ] Jellyfin login with a FreeIPA account works (LDAP plugin bound to 10.20.20.2).
  • [ ] From a Roku (IOT): Jellyfin connects on 8096 (rb06 IOT rule).
  • [ ] http://10.20.30.70:3000 (Invidious) loads → search + video plays (proves the companion/sig path).
  • [ ] Home Assistant reachable at https://ha.${domain} and http://10.20.20.45:8123 from a workstation; Zigbee devices paired; HA can control an IP IoT device (proves SERVERS→IOT allow + mDNS).
  • [ ] All four egress via Proton (check exit IP); invidious still plays — if Google blocks it, apply the VPN_BYPASS Google-ranges valve.
  • [ ] (if using ha-mcp) from the laptop on WG: claude mcp list shows ha-mcp connected; Claude can list automations (read) and create + delete a trivial test automation (write) — proving the claude-mcp admin token + apps-proxy WebSocket path.