Skip to content

Runbook 11 — AlmaLinux ZFS + NFSv4 storage

The ZFS pool is served from an AlmaLinux 9 VM with the data NVMe passed through. Rationale (see the "Identity + NFS" section of CLAUDE.md): a storage appliance gives us no better ZFS (TrueNAS SCALE is Linux OpenZFS — the same code), only a UI for the sharing; the box is a single NVMe (no array to manage); we need NFS only; and an IPA-enrolled EL box folds into the fleet's Ansible/identity/observability AND keeps the krb5 path open (an AD-first appliance structurally blocks it). Redundancy comes from zfs send replication to a second box, not local RAID.

⚠️ sec=sys, not sec=krb5 (for now). Identity still maps correctly because the server and clients read the same FreeIPA UID/GID (operator = 640400004 everywhere). Threat model = ISP/edge, not the firewalled internal LAN, so this is fully defensible. This EL box can do krb5 later with no appliance blockers — see §7.

Result: fileserver (10.20.20.10, SERVERS) serves the Jellyfin/tvheadend mounts (runbook 21) and the home automounts (runbook 12). Provisioned by provision/vms.conf (9126) + the Ansible nfs_server role.

Prereqs: runbook 10 (FreeIPA up).

1. Provision the storage VM (PVE)

cd /root/lab-provision
./lab-provision.sh up fileserver            # 9126, SERVERS .10, AlmaLinux 9

lab-provision.sh can't express PCI passthrough, so attach the NVMe by hand. It's 0000:10:00.0 (the Micron/Crucial P3 Plus), alone in IOMMU group 21 (verified — isolated, safe to pass through):

qm set 9126 --hostpci0 0000:10:00.0,pcie=1

Leave RAM at 8 GB to start (ZFS ARC ≈ half RAM). Bump if the working set grows.

2. Install OpenZFS (kABI kmod) — the licensing-safe way

The nfs_server role does this, but here's what it's doing and why (EL ships ZFS out-of-tree; the kmod, not dkms, is what keeps a kernel update from breaking the pool):

dnf install -y https://zfsonlinux.org/epel/zfs-release-2-3.el9.noarch.rpm
dnf config-manager --disable zfs --enable zfs-kmod     # kABI kmod, NOT dkms
dnf install -y kmod-zfs zfs nfs-utils python3-dnf-plugin-versionlock
dnf versionlock add kernel kernel-core zfs kmod-zfs libzfs5 zfs-dracut
modprobe zfs && zfs version
systemctl enable zfs-import-cache zfs-mount zfs.target

Update discipline (the whole reason this is safe): on an EL point-release jump, dnf update zfs kmod-zfs FIRST, reboot, confirm zpool status, THEN let the kernel advance. versionlock holds the pair together so an accidental dnf update can't outrun the module. This is exactly how the Proxmox host already runs Debian+ZFS without drama.

3. Create the pool + datasets

The NVMe is a single device, so the pool is a single-disk vdev (redundancy is the replica in §6, not local RAID). Create the pool, then the three datasets.

# identify the passed-through disk inside the VM (use the /dev/disk/by-id path so the
# pool name binding survives device renumbering):
ls -l /dev/disk/by-id/                         # find the nvme-* id for the data disk

zpool create -o ashift=12 -O compression=lz4 -O atime=off \
  -O mountpoint=/mnt/tank tank /dev/disk/by-id/nvme-<data-disk-id>

zfs create tank/home
zfs create tank/media
zfs create tank/recordings

zpool status tank                              # ONLINE
zfs list -o name,mountpoint,compression

Pool name tank is a convention — if you choose another, set storage_pool in host_vars/fileserver.yml and the media mounts (runbook 21) to match. Datasets mount at /mnt/tank/{home,media,recordings}.

Optional native encryption. If you want the data encrypted at rest, create the datasets with -o encryption=on -o keyformat=passphrase (or a keyfile) and zfs load-key + zfs mount -a after a reboot — wire the key unlock into the boot sequence if you want it automatic.

4. Enroll in FreeIPA + export NFS (Ansible)

The box is in [all_linux], so site.yml IPA-enrolls it (the common role) and runs the nfs_server role (exports /mnt/tank/{home,media,recordings} sec=sys to the SERVERS/WORKSTATIONS VLANs, plus a per-peer /32 in the WG-RW subnet for each roaming laptop that mounts over the road-warrior tunnel — see nfs_clients in host_vars/fileserver.yml):

cd /root/opnsense/ansible
ANSIBLE_VAULT_PASSWORD_FILE=~/.config/lab-vault-pass \
  ansible-playbook -i inventory.ini playbooks/site.yml -l fileserver

Confirm getent passwd operator on the fileserver returns the FreeIPA UID (proves enrollment → correct sec=sys ownership mapping).

5. Home-dir dataset layout

  • home parent dataset 0771 root:root; per-user subdirs /mnt/tank/home/<user> owned by that IPA user, mode 0700. oddjob-mkhomedir on the clients (runbook 12) auto-creates on first login, or pre-create per user.
  • Set the NFSv4 idmap domain to ${ipa_domain} on server + clients (/etc/idmapd.conf Domain =), else IDs map to nobody.

6. Redundancy — replicate, don't RAID

Single NVMe = no local redundancy, so protect the data by replicating snapshots to a second box. Use sanoid for snapshot policy + syncoid for the incremental zfs send:

# sanoid (on the fileserver): /etc/sanoid/sanoid.conf takes hourly/daily/monthly
# retention per dataset, run by the sanoid systemd timer. Sketch:
#   [tank/home]
#       use_template = production
#   [template_production]
#       hourly = 36
#       daily  = 30
#       monthly = 3
#       autosnap = yes
#       autoprune = yes

# syncoid (incremental send to the second ZFS host over the LAN/WG, e.g. nightly timer):
syncoid --recursive tank repltarget:backup/tank

Equivalent raw zfs send if you prefer not to use syncoid:

zfs snapshot -r tank@$(date +%F)
zfs send -RI tank@<prev> tank@<today> | ssh repltarget zfs recv -Fdu backup
  • Optional cheap bitrot insurance on a single disk: zfs set copies=2 tank/home (2× space on that dataset, lets ZFS self-heal checksum errors).
  • Backup note: this data lives on a passthrough NVMe, so Proxmox vzdump does not cover it. The replica above is the backup. The VM's OS disk rides the normal Proxmox job.

7. (Optional, later) harden to krb5

The EL kernel NFS server does krb5 natively, so promoting the home export is straightforward: ipa service-add nfs/fileserver.${ipa_domain}, ipa-getkeytab for it with an aes-sha1-restricted enctype (kernel RPCSEC_GSS limitation), set the export to sec=krb5, and add the client krb5.conf enctype drop-in. (FreeIPA mints aes-sha2 tickets while the kernel RPCSEC_GSS only supports aes-sha1 — hence the enctype-restricted keytab.)

Firewall

Runbook 06 already allows WS → SERVERS_NET on IPA_PORTS (incl. NFSv4 2049), on the default gateway (NFS stays LAN-speed, untunneled). Nothing new here.

Validation

  • [ ] zpool status tank ONLINE on the fileserver; zfs list shows the datasets mounted under /mnt/tank (and keystatus available if encrypted).
  • [ ] getent passwd operator on the fileserver returns the FreeIPA UID.
  • [ ] exportfs -v shows the three exports to the client VLANs.
  • [ ] From a WORKSTATIONS test client: mount -t nfs4 10.20.20.10:/mnt/tank/home /mnt/test succeeds; a file created there is owned by operator, not nobody (idmap domain matches).
  • [ ] Jellyfin/tvheadend (runbook 21) see their /media and /recordings mounts.
  • [ ] Replication: a snapshot lands on the second box (zfs list -t snapshot there).

Then → runbook 12 (client enrollment + automount).